-- Logs begin at Thu 2022-10-13 03:42:03 UTC, end at Thu 2022-10-13 04:42:13 UTC. -- Oct 13 04:07:33.209664 dv-cinder-860997-1-iscsi systemd[1]: Starting Devstack devstack@keystone.service... Oct 13 04:07:33.217838 dv-cinder-860997-1-iscsi devstack@keystone.service[81395]: [uWSGI] getting INI configuration from /etc/keystone/keystone-uwsgi-public.ini Oct 13 04:07:33.219847 dv-cinder-860997-1-iscsi devstack@keystone.service[81395]: *** Starting uWSGI 2.0.18-debian (64bit) on [Thu Oct 13 04:07:33 2022] *** Oct 13 04:07:33.219847 dv-cinder-860997-1-iscsi devstack@keystone.service[81395]: compiled with version: 10.0.1 20200405 (experimental) [master revision 0be9efad938:fcb98e4978a:705510a708d3642c9c962beb663c476167e4e8a4] on 11 April 2020 11:15:55 Oct 13 04:07:33.219847 dv-cinder-860997-1-iscsi devstack@keystone.service[81395]: os: Linux-5.4.0-72-generic #80-Ubuntu SMP Mon Apr 12 17:35:00 UTC 2021 Oct 13 04:07:33.219847 dv-cinder-860997-1-iscsi devstack@keystone.service[81395]: nodename: dv-cinder-860997-1-iscsi Oct 13 04:07:33.219847 dv-cinder-860997-1-iscsi devstack@keystone.service[81395]: machine: x86_64 Oct 13 04:07:33.219847 dv-cinder-860997-1-iscsi devstack@keystone.service[81395]: clock source: unix Oct 13 04:07:33.219847 dv-cinder-860997-1-iscsi devstack@keystone.service[81395]: pcre jit disabled Oct 13 04:07:33.220291 dv-cinder-860997-1-iscsi devstack@keystone.service[81395]: detected number of CPU cores: 4 Oct 13 04:07:33.220291 dv-cinder-860997-1-iscsi devstack@keystone.service[81395]: current working directory: / Oct 13 04:07:33.220291 dv-cinder-860997-1-iscsi devstack@keystone.service[81395]: detected binary path: /usr/bin/uwsgi-core Oct 13 04:07:33.220291 dv-cinder-860997-1-iscsi devstack@keystone.service[81395]: your processes number limit is 47680 Oct 13 04:07:33.220291 dv-cinder-860997-1-iscsi devstack@keystone.service[81395]: your memory page size is 4096 bytes Oct 13 04:07:33.220291 dv-cinder-860997-1-iscsi devstack@keystone.service[81395]: detected max file descriptor number: 2048 Oct 13 04:07:33.220291 dv-cinder-860997-1-iscsi devstack@keystone.service[81395]: lock engine: pthread robust mutexes Oct 13 04:07:33.220291 dv-cinder-860997-1-iscsi devstack@keystone.service[81395]: thunder lock: enabled Oct 13 04:07:33.220291 dv-cinder-860997-1-iscsi devstack@keystone.service[81395]: uwsgi socket 0 bound to UNIX address /var/run/uwsgi/keystone-wsgi-public.socket fd 4 Oct 13 04:07:33.220291 dv-cinder-860997-1-iscsi devstack@keystone.service[81395]: Python version: 3.8.10 (default, Jun 22 2022, 20:18:18) [GCC 9.4.0] Oct 13 04:07:33.264841 dv-cinder-860997-1-iscsi devstack@keystone.service[81395]: Python main interpreter initialized at 0x557fce1b69c0 Oct 13 04:07:33.264841 dv-cinder-860997-1-iscsi devstack@keystone.service[81395]: python threads support enabled Oct 13 04:07:33.264841 dv-cinder-860997-1-iscsi devstack@keystone.service[81395]: your server socket listen backlog is limited to 100 connections Oct 13 04:07:33.264841 dv-cinder-860997-1-iscsi devstack@keystone.service[81395]: your mercy for graceful operations on workers is 90 seconds Oct 13 04:07:33.265203 dv-cinder-860997-1-iscsi devstack@keystone.service[81395]: mapped 403077 bytes (393 KB) for 2 cores Oct 13 04:07:33.265290 dv-cinder-860997-1-iscsi devstack@keystone.service[81395]: *** Operational MODE: preforking *** Oct 13 04:07:33.265564 dv-cinder-860997-1-iscsi devstack@keystone.service[81395]: *** uWSGI is running in multiple interpreter mode *** Oct 13 04:07:33.265564 dv-cinder-860997-1-iscsi devstack@keystone.service[81395]: spawned uWSGI master process (pid: 81395) Oct 13 04:07:33.265763 dv-cinder-860997-1-iscsi systemd[1]: Started Devstack devstack@keystone.service. Oct 13 04:07:33.266854 dv-cinder-860997-1-iscsi devstack@keystone.service[81395]: spawned uWSGI worker 1 (pid: 81408, cores: 1) Oct 13 04:07:33.266854 dv-cinder-860997-1-iscsi devstack@keystone.service[81395]: spawned uWSGI worker 2 (pid: 81409, cores: 1) Oct 13 04:07:33.266854 dv-cinder-860997-1-iscsi devstack@keystone.service[81395]: running "unix_signal:15 gracefully_kill_them_all" (master-start)... Oct 13 04:07:37.563048 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: /usr/lib/python3/dist-packages/secretstorage/dhcrypto.py:15: CryptographyDeprecationWarning: int_from_bytes is deprecated, use int.from_bytes instead Oct 13 04:07:37.563048 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: from cryptography.utils import int_from_bytes Oct 13 04:07:37.563841 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: /usr/lib/python3/dist-packages/secretstorage/dhcrypto.py:15: CryptographyDeprecationWarning: int_from_bytes is deprecated, use int.from_bytes instead Oct 13 04:07:37.563841 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: from cryptography.utils import int_from_bytes Oct 13 04:07:37.564286 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: /usr/lib/python3/dist-packages/secretstorage/util.py:19: CryptographyDeprecationWarning: int_from_bytes is deprecated, use int.from_bytes instead Oct 13 04:07:37.564286 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: from cryptography.utils import int_from_bytes Oct 13 04:07:37.564395 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: /usr/lib/python3/dist-packages/secretstorage/util.py:19: CryptographyDeprecationWarning: int_from_bytes is deprecated, use int.from_bytes instead Oct 13 04:07:37.564395 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: from cryptography.utils import int_from_bytes Oct 13 04:07:37.614165 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG uwsgi [-] ******************************************************************************** {{(pid=81408) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2589}} Oct 13 04:07:37.614386 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG uwsgi [-] Configuration options gathered from: {{(pid=81408) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2590}} Oct 13 04:07:37.614485 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG uwsgi [-] ******************************************************************************** {{(pid=81409) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2589}} Oct 13 04:07:37.614572 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG uwsgi [-] command line args: [] {{(pid=81408) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2591}} Oct 13 04:07:37.614687 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG uwsgi [-] Configuration options gathered from: {{(pid=81409) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2590}} Oct 13 04:07:37.614837 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG uwsgi [-] config files: ['/etc/keystone/keystone.conf'] {{(pid=81408) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2592}} Oct 13 04:07:37.614920 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG uwsgi [-] command line args: [] {{(pid=81409) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2591}} Oct 13 04:07:37.615001 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG uwsgi [-] ================================================================================ {{(pid=81408) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2594}} Oct 13 04:07:37.615144 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG uwsgi [-] config files: ['/etc/keystone/keystone.conf'] {{(pid=81409) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2592}} Oct 13 04:07:37.615257 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG uwsgi [-] admin_token = **** {{(pid=81408) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 13 04:07:37.615340 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG uwsgi [-] ================================================================================ {{(pid=81409) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2594}} Oct 13 04:07:37.615508 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG uwsgi [-] config_dir = [] {{(pid=81408) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 13 04:07:37.615593 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG uwsgi [-] admin_token = **** {{(pid=81409) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 13 04:07:37.615694 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG uwsgi [-] config_file = ['/etc/keystone/keystone.conf'] {{(pid=81408) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 13 04:07:37.615808 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG uwsgi [-] config_dir = [] {{(pid=81409) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 13 04:07:37.615890 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG uwsgi [-] config_source = [] {{(pid=81408) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 13 04:07:37.615999 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG uwsgi [-] config_file = ['/etc/keystone/keystone.conf'] {{(pid=81409) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 13 04:07:37.616074 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG uwsgi [-] debug = True {{(pid=81408) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 13 04:07:37.616184 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG uwsgi [-] config_source = [] {{(pid=81409) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 13 04:07:37.616260 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG uwsgi [-] default_log_levels = ['amqp=WARN', 'amqplib=WARN', 'boto=WARN', 'qpid=WARN', 'sqlalchemy=WARN', 'suds=INFO', 'oslo.messaging=INFO', 'oslo_messaging=INFO', 'iso8601=WARN', 'requests.packages.urllib3.connectionpool=WARN', 'urllib3.connectionpool=WARN', 'websocket=WARN', 'requests.packages.urllib3.util.retry=WARN', 'urllib3.util.retry=WARN', 'keystonemiddleware=WARN', 'routes.middleware=WARN', 'stevedore=WARN', 'taskflow=WARN', 'keystoneauth=WARN', 'oslo.cache=INFO', 'oslo_policy=INFO', 'dogpile.core.dogpile=INFO', 'dogpile=INFO', 'routes=INFO'] {{(pid=81408) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 13 04:07:37.616479 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG uwsgi [-] debug = True {{(pid=81409) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 13 04:07:37.616549 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG uwsgi [-] default_log_levels = ['amqp=WARN', 'amqplib=WARN', 'boto=WARN', 'qpid=WARN', 'sqlalchemy=WARN', 'suds=INFO', 'oslo.messaging=INFO', 'oslo_messaging=INFO', 'iso8601=WARN', 'requests.packages.urllib3.connectionpool=WARN', 'urllib3.connectionpool=WARN', 'websocket=WARN', 'requests.packages.urllib3.util.retry=WARN', 'urllib3.util.retry=WARN', 'keystonemiddleware=WARN', 'routes.middleware=WARN', 'stevedore=WARN', 'taskflow=WARN', 'keystoneauth=WARN', 'oslo.cache=INFO', 'oslo_policy=INFO', 'dogpile.core.dogpile=INFO', 'dogpile=INFO', 'routes=INFO'] {{(pid=81409) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 13 04:07:37.616635 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG uwsgi [-] default_publisher_id = None {{(pid=81408) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 13 04:07:37.616780 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG uwsgi [-] insecure_debug = False {{(pid=81408) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 13 04:07:37.616879 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG uwsgi [-] default_publisher_id = None {{(pid=81409) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 13 04:07:37.616971 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG uwsgi [-] insecure_debug = False {{(pid=81409) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 13 04:07:37.617068 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG uwsgi [-] instance_format = [instance: %(uuid)s] {{(pid=81408) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 13 04:07:37.617259 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG uwsgi [-] instance_format = [instance: %(uuid)s] {{(pid=81409) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 13 04:07:37.617454 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG uwsgi [-] instance_uuid_format = [instance: %(uuid)s] {{(pid=81408) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 13 04:07:37.617634 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG uwsgi [-] instance_uuid_format = [instance: %(uuid)s] {{(pid=81409) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 13 04:07:37.617808 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG uwsgi [-] list_limit = None {{(pid=81408) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 13 04:07:37.617942 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG uwsgi [-] list_limit = None {{(pid=81409) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 13 04:07:37.618047 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG uwsgi [-] log_config_append = None {{(pid=81408) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 13 04:07:37.618146 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG uwsgi [-] log_config_append = None {{(pid=81409) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 13 04:07:37.618291 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG uwsgi [-] log_date_format = %Y-%m-%d %H:%M:%S {{(pid=81408) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 13 04:07:37.618392 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG uwsgi [-] log_date_format = %Y-%m-%d %H:%M:%S {{(pid=81409) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 13 04:07:37.618491 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG uwsgi [-] log_dir = None {{(pid=81408) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 13 04:07:37.618583 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG uwsgi [-] log_dir = None {{(pid=81409) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 13 04:07:37.618699 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG uwsgi [-] log_file = None {{(pid=81408) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 13 04:07:37.618810 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG uwsgi [-] log_file = None {{(pid=81409) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 13 04:07:37.618921 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG uwsgi [-] log_rotate_interval = 1 {{(pid=81408) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 13 04:07:37.619038 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG uwsgi [-] log_rotate_interval = 1 {{(pid=81409) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 13 04:07:37.619153 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG uwsgi [-] log_rotate_interval_type = days {{(pid=81408) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 13 04:07:37.619262 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG uwsgi [-] log_rotate_interval_type = days {{(pid=81409) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 13 04:07:37.619378 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG uwsgi [-] log_rotation_type = none {{(pid=81408) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 13 04:07:37.619493 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG uwsgi [-] log_rotation_type = none {{(pid=81409) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 13 04:07:37.619611 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG uwsgi [-] logging_context_format_string = %(color)s%(levelname)s %(name)s [%(global_request_id)s %(request_id)s %(project_name)s %(user_name)s%(color)s] %(instance)s%(color)s%(message)s {{(pid=81408) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 13 04:07:37.619732 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG uwsgi [-] logging_context_format_string = %(color)s%(levelname)s %(name)s [%(global_request_id)s %(request_id)s %(project_name)s %(user_name)s%(color)s] %(instance)s%(color)s%(message)s {{(pid=81409) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 13 04:07:37.619846 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG uwsgi [-] logging_debug_format_suffix = {{(pid=%(process)d) %(funcName)s %(pathname)s:%(lineno)d}} {{(pid=81408) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 13 04:07:37.619956 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG uwsgi [-] logging_debug_format_suffix = {{(pid=%(process)d) %(funcName)s %(pathname)s:%(lineno)d}} {{(pid=81409) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 13 04:07:37.620073 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG uwsgi [-] logging_default_format_string = %(color)s%(levelname)s %(name)s [-%(color)s] %(instance)s%(color)s%(message)s {{(pid=81408) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 13 04:07:37.620196 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG uwsgi [-] logging_default_format_string = %(color)s%(levelname)s %(name)s [-%(color)s] %(instance)s%(color)s%(message)s {{(pid=81409) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 13 04:07:37.620315 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG uwsgi [-] logging_exception_prefix = ERROR %(name)s %(instance)s {{(pid=81408) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 13 04:07:37.620426 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG uwsgi [-] logging_exception_prefix = ERROR %(name)s %(instance)s {{(pid=81409) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 13 04:07:37.620536 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG uwsgi [-] logging_user_identity_format = %(user)s %(project)s %(domain)s %(system_scope)s %(user_domain)s %(project_domain)s {{(pid=81408) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 13 04:07:37.620646 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG uwsgi [-] logging_user_identity_format = %(user)s %(project)s %(domain)s %(system_scope)s %(user_domain)s %(project_domain)s {{(pid=81409) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 13 04:07:37.620770 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG uwsgi [-] max_logfile_count = 30 {{(pid=81408) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 13 04:07:37.620888 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG uwsgi [-] max_logfile_count = 30 {{(pid=81409) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 13 04:07:37.620999 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG uwsgi [-] max_logfile_size_mb = 200 {{(pid=81408) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 13 04:07:37.621114 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG uwsgi [-] max_logfile_size_mb = 200 {{(pid=81409) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 13 04:07:37.621230 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG uwsgi [-] max_param_size = 64 {{(pid=81408) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 13 04:07:37.621345 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG uwsgi [-] max_param_size = 64 {{(pid=81409) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 13 04:07:37.621462 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG uwsgi [-] max_project_tree_depth = 5 {{(pid=81408) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 13 04:07:37.621579 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG uwsgi [-] max_project_tree_depth = 5 {{(pid=81409) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 13 04:07:37.621695 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG uwsgi [-] max_token_size = 16384 {{(pid=81408) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 13 04:07:37.621815 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG uwsgi [-] max_token_size = 16384 {{(pid=81409) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 13 04:07:37.621933 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG uwsgi [-] notification_format = cadf {{(pid=81408) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 13 04:07:37.622048 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG uwsgi [-] notification_format = cadf {{(pid=81409) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 13 04:07:37.622164 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG uwsgi [-] notification_opt_out = ['identity.authenticate.success', 'identity.authenticate.pending', 'identity.authenticate.failed'] {{(pid=81408) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 13 04:07:37.622322 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG uwsgi [-] notification_opt_out = ['identity.authenticate.success', 'identity.authenticate.pending', 'identity.authenticate.failed'] {{(pid=81409) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 13 04:07:37.622442 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG uwsgi [-] public_endpoint = http://10.222.0.27/identity {{(pid=81408) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 13 04:07:37.622554 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG uwsgi [-] public_endpoint = http://10.222.0.27/identity {{(pid=81409) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 13 04:07:37.622673 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG uwsgi [-] publish_errors = False {{(pid=81408) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 13 04:07:37.622783 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG uwsgi [-] publish_errors = False {{(pid=81409) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 13 04:07:37.622900 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG uwsgi [-] pydev_debug_host = None {{(pid=81408) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 13 04:07:37.623009 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG uwsgi [-] pydev_debug_host = None {{(pid=81409) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 13 04:07:37.623124 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG uwsgi [-] pydev_debug_port = None {{(pid=81408) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 13 04:07:37.623243 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG uwsgi [-] pydev_debug_port = None {{(pid=81409) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 13 04:07:37.623354 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG uwsgi [-] rate_limit_burst = 0 {{(pid=81408) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 13 04:07:37.623468 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG uwsgi [-] rate_limit_burst = 0 {{(pid=81409) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 13 04:07:37.623579 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG uwsgi [-] rate_limit_except_level = CRITICAL {{(pid=81408) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 13 04:07:37.623705 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG uwsgi [-] rate_limit_except_level = CRITICAL {{(pid=81409) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 13 04:07:37.623824 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG uwsgi [-] rate_limit_interval = 0 {{(pid=81408) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 13 04:07:37.623942 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG uwsgi [-] rate_limit_interval = 0 {{(pid=81409) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 13 04:07:37.624059 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG uwsgi [-] standard_threads = False {{(pid=81408) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 13 04:07:37.624169 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG uwsgi [-] standard_threads = False {{(pid=81409) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 13 04:07:37.624285 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG uwsgi [-] strict_password_check = False {{(pid=81408) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 13 04:07:37.624406 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG uwsgi [-] strict_password_check = False {{(pid=81409) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 13 04:07:37.624524 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG uwsgi [-] syslog_log_facility = LOG_USER {{(pid=81408) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 13 04:07:37.624634 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG uwsgi [-] syslog_log_facility = LOG_USER {{(pid=81409) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 13 04:07:37.624744 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG uwsgi [-] use_eventlog = False {{(pid=81408) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 13 04:07:37.624853 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG uwsgi [-] use_eventlog = False {{(pid=81409) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 13 04:07:37.624962 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG uwsgi [-] use_journal = False {{(pid=81408) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 13 04:07:37.625121 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG uwsgi [-] use_journal = False {{(pid=81409) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 13 04:07:37.625240 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG uwsgi [-] use_json = False {{(pid=81408) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 13 04:07:37.625351 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG uwsgi [-] use_json = False {{(pid=81409) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 13 04:07:37.625461 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG uwsgi [-] use_stderr = False {{(pid=81408) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 13 04:07:37.625570 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG uwsgi [-] use_stderr = False {{(pid=81409) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 13 04:07:37.625679 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG uwsgi [-] use_syslog = False {{(pid=81408) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 13 04:07:37.625796 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG uwsgi [-] use_syslog = False {{(pid=81409) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 13 04:07:37.625908 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG uwsgi [-] watch_log_file = False {{(pid=81408) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 13 04:07:37.626024 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG uwsgi [-] watch_log_file = False {{(pid=81409) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 13 04:07:37.626135 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG uwsgi [-] oslo_concurrency.disable_process_locking = False {{(pid=81408) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 13 04:07:37.626268 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG uwsgi [-] oslo_concurrency.disable_process_locking = False {{(pid=81409) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 13 04:07:37.626389 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG uwsgi [-] oslo_concurrency.lock_path = None {{(pid=81408) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 13 04:07:37.626512 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG uwsgi [-] oslo_concurrency.lock_path = None {{(pid=81409) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 13 04:07:37.626623 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG uwsgi [-] profiler.connection_string = messaging:// {{(pid=81408) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 13 04:07:37.626732 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG uwsgi [-] profiler.connection_string = messaging:// {{(pid=81409) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 13 04:07:37.626849 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG uwsgi [-] profiler.enabled = False {{(pid=81408) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 13 04:07:37.626959 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG uwsgi [-] profiler.enabled = False {{(pid=81409) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 13 04:07:37.627069 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG uwsgi [-] profiler.es_doc_type = notification {{(pid=81408) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 13 04:07:37.627190 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG uwsgi [-] profiler.es_doc_type = notification {{(pid=81409) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 13 04:07:37.627301 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG uwsgi [-] profiler.es_scroll_size = 10000 {{(pid=81408) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 13 04:07:37.627411 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG uwsgi [-] profiler.es_scroll_size = 10000 {{(pid=81409) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 13 04:07:37.627527 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG uwsgi [-] profiler.es_scroll_time = 2m {{(pid=81408) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 13 04:07:37.627637 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG uwsgi [-] profiler.es_scroll_time = 2m {{(pid=81409) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 13 04:07:37.627746 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG uwsgi [-] profiler.filter_error_trace = False {{(pid=81408) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 13 04:07:37.627863 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG uwsgi [-] profiler.filter_error_trace = False {{(pid=81409) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 13 04:07:37.628198 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG uwsgi [-] profiler.hmac_keys = SECRET_KEY {{(pid=81408) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 13 04:07:37.628308 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG uwsgi [-] profiler.hmac_keys = SECRET_KEY {{(pid=81409) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 13 04:07:37.628417 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG uwsgi [-] profiler.sentinel_service_name = mymaster {{(pid=81408) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 13 04:07:37.628526 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG uwsgi [-] profiler.sentinel_service_name = mymaster {{(pid=81409) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 13 04:07:37.628642 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG uwsgi [-] profiler.socket_timeout = 0.1 {{(pid=81408) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 13 04:07:37.628760 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG uwsgi [-] profiler.socket_timeout = 0.1 {{(pid=81409) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 13 04:07:37.628871 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG uwsgi [-] profiler.trace_sqlalchemy = False {{(pid=81408) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 13 04:07:37.628980 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG uwsgi [-] profiler.trace_sqlalchemy = False {{(pid=81409) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 13 04:07:37.629098 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG uwsgi [-] oslo_messaging_metrics.metrics_buffer_size = 1000 {{(pid=81408) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 13 04:07:37.629209 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG uwsgi [-] oslo_messaging_metrics.metrics_buffer_size = 1000 {{(pid=81409) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 13 04:07:37.629325 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG uwsgi [-] oslo_messaging_metrics.metrics_enabled = False {{(pid=81408) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 13 04:07:37.629451 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG uwsgi [-] oslo_messaging_metrics.metrics_enabled = False {{(pid=81409) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 13 04:07:37.629562 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG uwsgi [-] oslo_messaging_metrics.metrics_process_name = {{(pid=81408) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 13 04:07:37.629672 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG uwsgi [-] oslo_messaging_metrics.metrics_process_name = {{(pid=81409) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 13 04:07:37.629781 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG uwsgi [-] oslo_messaging_metrics.metrics_socket_file = /var/tmp/metrics_collector.sock {{(pid=81408) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 13 04:07:37.629891 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG uwsgi [-] oslo_messaging_metrics.metrics_socket_file = /var/tmp/metrics_collector.sock {{(pid=81409) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 13 04:07:37.630001 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG uwsgi [-] oslo_messaging_metrics.metrics_thread_stop_timeout = 10 {{(pid=81408) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 13 04:07:37.630118 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG uwsgi [-] oslo_messaging_metrics.metrics_thread_stop_timeout = 10 {{(pid=81409) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 13 04:07:37.630257 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG uwsgi [-] oslo_middleware.http_basic_auth_user_file = /etc/htpasswd {{(pid=81408) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 13 04:07:37.630371 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG uwsgi [-] oslo_middleware.http_basic_auth_user_file = /etc/htpasswd {{(pid=81409) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 13 04:07:37.630481 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG uwsgi [-] audit.namespace = openstack {{(pid=81408) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 13 04:07:37.630591 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG uwsgi [-] audit.namespace = openstack {{(pid=81409) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 13 04:07:37.630706 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG uwsgi [-] oslo_policy.enforce_new_defaults = False {{(pid=81408) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 13 04:07:37.630896 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG uwsgi [-] oslo_policy.enforce_new_defaults = False {{(pid=81409) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 13 04:07:37.631016 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG uwsgi [-] oslo_policy.enforce_scope = False {{(pid=81408) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 13 04:07:37.631132 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG uwsgi [-] oslo_policy.enforce_scope = False {{(pid=81409) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 13 04:07:37.631243 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG uwsgi [-] oslo_policy.policy_default_rule = default {{(pid=81408) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 13 04:07:37.631353 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG uwsgi [-] oslo_policy.policy_default_rule = default {{(pid=81409) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 13 04:07:37.631474 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG uwsgi [-] oslo_policy.policy_dirs = ['policy.d'] {{(pid=81408) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 13 04:07:37.631591 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG uwsgi [-] oslo_policy.policy_dirs = ['policy.d'] {{(pid=81409) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 13 04:07:37.631701 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG uwsgi [-] oslo_policy.policy_file = policy.yaml {{(pid=81408) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 13 04:07:37.631810 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG uwsgi [-] oslo_policy.policy_file = policy.yaml {{(pid=81409) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 13 04:07:37.631919 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG uwsgi [-] oslo_policy.remote_content_type = application/x-www-form-urlencoded {{(pid=81408) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 13 04:07:37.632029 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG uwsgi [-] oslo_policy.remote_content_type = application/x-www-form-urlencoded {{(pid=81409) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 13 04:07:37.632149 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG uwsgi [-] oslo_policy.remote_ssl_ca_crt_file = None {{(pid=81408) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 13 04:07:37.632267 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG uwsgi [-] oslo_policy.remote_ssl_ca_crt_file = None {{(pid=81409) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 13 04:07:37.632385 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG uwsgi [-] oslo_policy.remote_ssl_client_crt_file = None {{(pid=81408) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 13 04:07:37.632495 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG uwsgi [-] oslo_policy.remote_ssl_client_crt_file = None {{(pid=81409) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 13 04:07:37.632604 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG uwsgi [-] oslo_policy.remote_ssl_client_key_file = None {{(pid=81408) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 13 04:07:37.632720 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG uwsgi [-] oslo_policy.remote_ssl_client_key_file = None {{(pid=81409) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 13 04:07:37.632838 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG uwsgi [-] oslo_policy.remote_ssl_verify_server_crt = False {{(pid=81408) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 13 04:07:37.632973 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG uwsgi [-] oslo_policy.remote_ssl_verify_server_crt = False {{(pid=81409) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 13 04:07:37.633157 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG uwsgi [-] keystone_authtoken.admin_password = **** {{(pid=81408) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 13 04:07:37.633325 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG uwsgi [-] keystone_authtoken.admin_password = **** {{(pid=81409) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 13 04:07:37.633512 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG uwsgi [-] keystone_authtoken.admin_tenant_name = admin {{(pid=81408) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 13 04:07:37.633690 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG uwsgi [-] keystone_authtoken.admin_tenant_name = admin {{(pid=81409) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 13 04:07:37.633823 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG uwsgi [-] keystone_authtoken.admin_token = **** {{(pid=81408) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 13 04:07:37.633936 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG uwsgi [-] keystone_authtoken.admin_token = **** {{(pid=81409) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 13 04:07:37.634053 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG uwsgi [-] keystone_authtoken.admin_user = None {{(pid=81408) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 13 04:07:37.634164 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG uwsgi [-] keystone_authtoken.admin_user = None {{(pid=81409) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 13 04:07:37.634164 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG uwsgi [-] keystone_authtoken.auth_admin_prefix = {{(pid=81409) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 13 04:07:37.634419 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG uwsgi [-] keystone_authtoken.auth_admin_prefix = {{(pid=81408) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 13 04:07:37.634545 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG uwsgi [-] keystone_authtoken.auth_host = 127.0.0.1 {{(pid=81409) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 13 04:07:37.634658 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG uwsgi [-] keystone_authtoken.auth_host = 127.0.0.1 {{(pid=81408) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 13 04:07:37.634767 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG uwsgi [-] keystone_authtoken.auth_port = 35357 {{(pid=81409) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 13 04:07:37.634876 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG uwsgi [-] keystone_authtoken.auth_port = 35357 {{(pid=81408) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 13 04:07:37.634986 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG uwsgi [-] keystone_authtoken.auth_protocol = https {{(pid=81409) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 13 04:07:37.635095 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG uwsgi [-] keystone_authtoken.auth_protocol = https {{(pid=81408) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 13 04:07:37.635217 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG uwsgi [-] keystone_authtoken.auth_section = None {{(pid=81409) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 13 04:07:37.635329 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG uwsgi [-] keystone_authtoken.auth_section = None {{(pid=81408) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 13 04:07:37.635439 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG uwsgi [-] keystone_authtoken.auth_type = None {{(pid=81409) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 13 04:07:37.635555 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG uwsgi [-] keystone_authtoken.auth_type = None {{(pid=81408) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 13 04:07:37.635672 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG uwsgi [-] keystone_authtoken.auth_uri = None {{(pid=81409) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 13 04:07:37.635783 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG uwsgi [-] keystone_authtoken.auth_uri = None {{(pid=81408) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 13 04:07:37.635900 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG uwsgi [-] keystone_authtoken.auth_version = None {{(pid=81409) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 13 04:07:37.636016 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG uwsgi [-] keystone_authtoken.auth_version = None {{(pid=81408) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 13 04:07:37.636133 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG uwsgi [-] keystone_authtoken.cache = None {{(pid=81409) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 13 04:07:37.636243 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG uwsgi [-] keystone_authtoken.cache = None {{(pid=81408) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 13 04:07:37.636352 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG uwsgi [-] keystone_authtoken.cafile = None {{(pid=81409) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 13 04:07:37.636461 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG uwsgi [-] keystone_authtoken.cafile = None {{(pid=81408) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 13 04:07:37.636648 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG uwsgi [-] keystone_authtoken.certfile = None {{(pid=81409) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 13 04:07:37.636761 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG uwsgi [-] keystone_authtoken.certfile = None {{(pid=81408) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 13 04:07:37.636878 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG uwsgi [-] keystone_authtoken.delay_auth_decision = False {{(pid=81409) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 13 04:07:37.636989 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG uwsgi [-] keystone_authtoken.delay_auth_decision = False {{(pid=81408) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 13 04:07:37.637104 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG uwsgi [-] keystone_authtoken.enforce_token_bind = permissive {{(pid=81409) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 13 04:07:37.637221 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG uwsgi [-] keystone_authtoken.enforce_token_bind = permissive {{(pid=81408) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 13 04:07:37.637339 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG uwsgi [-] keystone_authtoken.http_connect_timeout = None {{(pid=81409) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 13 04:07:37.637456 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG uwsgi [-] keystone_authtoken.http_connect_timeout = None {{(pid=81408) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 13 04:07:37.637567 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG uwsgi [-] keystone_authtoken.http_request_max_retries = 3 {{(pid=81409) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 13 04:07:37.637677 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG uwsgi [-] keystone_authtoken.http_request_max_retries = 3 {{(pid=81408) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 13 04:07:37.637785 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG uwsgi [-] keystone_authtoken.identity_uri = None {{(pid=81409) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 13 04:07:37.637894 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG uwsgi [-] keystone_authtoken.identity_uri = None {{(pid=81408) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 13 04:07:37.638012 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG uwsgi [-] keystone_authtoken.include_service_catalog = True {{(pid=81409) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 13 04:07:37.638128 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG uwsgi [-] keystone_authtoken.include_service_catalog = True {{(pid=81408) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 13 04:07:37.638261 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG uwsgi [-] keystone_authtoken.insecure = False {{(pid=81409) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 13 04:07:37.638375 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG uwsgi [-] keystone_authtoken.insecure = False {{(pid=81408) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 13 04:07:37.638491 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG uwsgi [-] keystone_authtoken.interface = internal {{(pid=81409) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 13 04:07:37.638587 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG uwsgi [-] keystone_authtoken.interface = internal {{(pid=81408) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 13 04:07:37.638689 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG uwsgi [-] keystone_authtoken.keyfile = None {{(pid=81409) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 13 04:07:37.638784 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG uwsgi [-] keystone_authtoken.keyfile = None {{(pid=81408) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 13 04:07:37.638877 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG uwsgi [-] keystone_authtoken.memcache_pool_conn_get_timeout = 10 {{(pid=81409) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 13 04:07:37.638969 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG uwsgi [-] keystone_authtoken.memcache_pool_conn_get_timeout = 10 {{(pid=81408) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 13 04:07:37.639061 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG uwsgi [-] keystone_authtoken.memcache_pool_dead_retry = 300 {{(pid=81409) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 13 04:07:37.639153 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG uwsgi [-] keystone_authtoken.memcache_pool_dead_retry = 300 {{(pid=81408) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 13 04:07:37.639253 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG uwsgi [-] keystone_authtoken.memcache_pool_maxsize = 10 {{(pid=81409) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 13 04:07:37.639353 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG uwsgi [-] keystone_authtoken.memcache_pool_maxsize = 10 {{(pid=81408) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 13 04:07:37.639460 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG uwsgi [-] keystone_authtoken.memcache_pool_socket_timeout = 3 {{(pid=81409) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 13 04:07:37.639606 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG uwsgi [-] keystone_authtoken.memcache_pool_socket_timeout = 3 {{(pid=81408) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 13 04:07:37.639728 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG uwsgi [-] keystone_authtoken.memcache_pool_unused_timeout = 60 {{(pid=81409) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 13 04:07:37.639848 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG uwsgi [-] keystone_authtoken.memcache_pool_unused_timeout = 60 {{(pid=81408) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 13 04:07:37.639959 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG uwsgi [-] keystone_authtoken.memcache_secret_key = **** {{(pid=81409) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 13 04:07:37.640069 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG uwsgi [-] keystone_authtoken.memcache_secret_key = **** {{(pid=81408) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 13 04:07:37.640179 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG uwsgi [-] keystone_authtoken.memcache_security_strategy = None {{(pid=81409) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 13 04:07:37.640291 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG uwsgi [-] keystone_authtoken.memcache_security_strategy = None {{(pid=81408) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 13 04:07:37.640408 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG uwsgi [-] keystone_authtoken.memcache_use_advanced_pool = True {{(pid=81409) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 13 04:07:37.640527 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG uwsgi [-] keystone_authtoken.memcache_use_advanced_pool = True {{(pid=81408) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 13 04:07:37.640639 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG uwsgi [-] keystone_authtoken.memcached_servers = None {{(pid=81409) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 13 04:07:37.640750 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG uwsgi [-] keystone_authtoken.memcached_servers = None {{(pid=81408) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 13 04:07:37.640860 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG uwsgi [-] keystone_authtoken.region_name = None {{(pid=81409) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 13 04:07:37.640984 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG uwsgi [-] keystone_authtoken.region_name = None {{(pid=81408) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 13 04:07:37.641105 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG uwsgi [-] keystone_authtoken.service_token_roles = ['service'] {{(pid=81409) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 13 04:07:37.641216 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG uwsgi [-] keystone_authtoken.service_token_roles = ['service'] {{(pid=81408) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 13 04:07:37.641327 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG uwsgi [-] keystone_authtoken.service_token_roles_required = False {{(pid=81409) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 13 04:07:37.641449 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG uwsgi [-] keystone_authtoken.service_token_roles_required = False {{(pid=81408) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 13 04:07:37.641554 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG uwsgi [-] keystone_authtoken.service_type = None {{(pid=81409) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 13 04:07:37.641662 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG uwsgi [-] keystone_authtoken.service_type = None {{(pid=81408) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 13 04:07:37.641780 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG uwsgi [-] keystone_authtoken.token_cache_time = 300 {{(pid=81409) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 13 04:07:37.641892 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG uwsgi [-] keystone_authtoken.token_cache_time = 300 {{(pid=81408) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 13 04:07:37.642002 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG uwsgi [-] keystone_authtoken.www_authenticate_uri = None {{(pid=81409) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 13 04:07:37.642120 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG uwsgi [-] keystone_authtoken.www_authenticate_uri = None {{(pid=81408) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 13 04:07:37.642246 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG uwsgi [-] cache.backend = dogpile.cache.memcached {{(pid=81409) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 13 04:07:37.642359 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG uwsgi [-] cache.backend = dogpile.cache.memcached {{(pid=81408) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 13 04:07:37.642470 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG uwsgi [-] cache.backend_argument = **** {{(pid=81409) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 13 04:07:37.642571 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG uwsgi [-] cache.backend_argument = **** {{(pid=81408) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 13 04:07:37.642680 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG uwsgi [-] cache.config_prefix = cache.oslo {{(pid=81409) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 13 04:07:37.642802 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG uwsgi [-] cache.config_prefix = cache.oslo {{(pid=81408) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 13 04:07:37.642916 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG uwsgi [-] cache.dead_timeout = 60.0 {{(pid=81409) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 13 04:07:37.643026 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG uwsgi [-] cache.dead_timeout = 60.0 {{(pid=81408) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 13 04:07:37.643144 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG uwsgi [-] cache.debug_cache_backend = False {{(pid=81409) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 13 04:07:37.643263 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG uwsgi [-] cache.debug_cache_backend = False {{(pid=81408) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 13 04:07:37.643375 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG uwsgi [-] cache.enable_retry_client = False {{(pid=81409) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 13 04:07:37.643493 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG uwsgi [-] cache.enable_retry_client = False {{(pid=81408) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 13 04:07:37.643597 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG uwsgi [-] cache.enable_socket_keepalive = False {{(pid=81409) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 13 04:07:37.643706 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG uwsgi [-] cache.enable_socket_keepalive = False {{(pid=81408) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 13 04:07:37.643816 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG uwsgi [-] cache.enabled = True {{(pid=81409) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 13 04:07:37.643927 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG uwsgi [-] cache.enabled = True {{(pid=81408) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 13 04:07:37.644037 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG uwsgi [-] cache.expiration_time = 600 {{(pid=81409) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 13 04:07:37.644155 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG uwsgi [-] cache.expiration_time = 600 {{(pid=81408) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 13 04:07:37.644266 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG uwsgi [-] cache.hashclient_retry_attempts = 2 {{(pid=81409) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 13 04:07:37.644383 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG uwsgi [-] cache.hashclient_retry_attempts = 2 {{(pid=81408) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 13 04:07:37.644502 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG uwsgi [-] cache.hashclient_retry_delay = 1.0 {{(pid=81409) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 13 04:07:37.644613 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG uwsgi [-] cache.hashclient_retry_delay = 1.0 {{(pid=81408) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 13 04:07:37.644723 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG uwsgi [-] cache.memcache_dead_retry = 300 {{(pid=81409) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 13 04:07:37.644849 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG uwsgi [-] cache.memcache_dead_retry = 300 {{(pid=81408) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 13 04:07:37.644969 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG uwsgi [-] cache.memcache_password = {{(pid=81409) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 13 04:07:37.645087 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG uwsgi [-] cache.memcache_password = {{(pid=81408) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 13 04:07:37.645198 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG uwsgi [-] cache.memcache_pool_connection_get_timeout = 10 {{(pid=81409) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 13 04:07:37.645309 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG uwsgi [-] cache.memcache_pool_connection_get_timeout = 10 {{(pid=81408) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 13 04:07:37.645419 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG uwsgi [-] cache.memcache_pool_flush_on_reconnect = False {{(pid=81409) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 13 04:07:37.645537 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG uwsgi [-] cache.memcache_pool_flush_on_reconnect = False {{(pid=81408) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 13 04:07:37.645649 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG uwsgi [-] cache.memcache_pool_maxsize = 10 {{(pid=81409) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 13 04:07:37.645759 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG uwsgi [-] cache.memcache_pool_maxsize = 10 {{(pid=81408) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 13 04:07:37.645869 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG uwsgi [-] cache.memcache_pool_unused_timeout = 60 {{(pid=81409) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 13 04:07:37.645981 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG uwsgi [-] cache.memcache_pool_unused_timeout = 60 {{(pid=81408) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 13 04:07:37.646097 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG uwsgi [-] cache.memcache_sasl_enabled = False {{(pid=81409) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 13 04:07:37.646240 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG uwsgi [-] cache.memcache_sasl_enabled = False {{(pid=81408) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 13 04:07:37.646355 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG uwsgi [-] cache.memcache_servers = ['localhost:11211'] {{(pid=81409) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 13 04:07:37.646466 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG uwsgi [-] cache.memcache_servers = ['localhost:11211'] {{(pid=81408) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 13 04:07:37.646581 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG uwsgi [-] cache.memcache_socket_timeout = 1.0 {{(pid=81409) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 13 04:07:37.646692 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG uwsgi [-] cache.memcache_socket_timeout = 1.0 {{(pid=81408) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 13 04:07:37.646802 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG uwsgi [-] cache.memcache_username = {{(pid=81409) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 13 04:07:37.646924 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG uwsgi [-] cache.memcache_username = {{(pid=81408) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 13 04:07:37.647039 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG uwsgi [-] cache.proxies = [] {{(pid=81409) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 13 04:07:37.647156 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG uwsgi [-] cache.proxies = [] {{(pid=81408) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 13 04:07:37.647268 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG uwsgi [-] cache.retry_attempts = 2 {{(pid=81409) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 13 04:07:37.647378 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG uwsgi [-] cache.retry_attempts = 2 {{(pid=81408) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 13 04:07:37.647489 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG uwsgi [-] cache.retry_delay = 0.0 {{(pid=81409) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 13 04:07:37.647675 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG uwsgi [-] cache.retry_delay = 0.0 {{(pid=81408) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 13 04:07:37.647788 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG uwsgi [-] cache.socket_keepalive_count = 1 {{(pid=81409) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 13 04:07:37.647899 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG uwsgi [-] cache.socket_keepalive_count = 1 {{(pid=81408) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 13 04:07:37.648009 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG uwsgi [-] cache.socket_keepalive_idle = 1 {{(pid=81409) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 13 04:07:37.648121 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG uwsgi [-] cache.socket_keepalive_idle = 1 {{(pid=81408) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 13 04:07:37.648239 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG uwsgi [-] cache.socket_keepalive_interval = 1 {{(pid=81409) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 13 04:07:37.648359 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG uwsgi [-] cache.socket_keepalive_interval = 1 {{(pid=81408) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 13 04:07:37.649011 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG uwsgi [-] cache.tls_allowed_ciphers = None {{(pid=81409) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 13 04:07:37.649200 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG uwsgi [-] cache.tls_allowed_ciphers = None {{(pid=81408) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 13 04:07:37.649383 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG uwsgi [-] cache.tls_cafile = None {{(pid=81409) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 13 04:07:37.649540 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG uwsgi [-] cache.tls_cafile = None {{(pid=81408) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 13 04:07:37.649655 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG uwsgi [-] cache.tls_certfile = None {{(pid=81409) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 13 04:07:37.649774 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG uwsgi [-] cache.tls_certfile = None {{(pid=81408) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 13 04:07:37.649896 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG uwsgi [-] cache.tls_enabled = False {{(pid=81409) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 13 04:07:37.650009 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG uwsgi [-] cache.tls_enabled = False {{(pid=81408) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 13 04:07:37.650121 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG uwsgi [-] cache.tls_keyfile = None {{(pid=81409) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 13 04:07:37.650259 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG uwsgi [-] cache.tls_keyfile = None {{(pid=81408) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 13 04:07:37.650375 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG uwsgi [-] application_credential.cache_time = None {{(pid=81409) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 13 04:07:37.650487 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG uwsgi [-] application_credential.cache_time = None {{(pid=81408) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 13 04:07:37.650607 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG uwsgi [-] application_credential.caching = True {{(pid=81409) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 13 04:07:37.650720 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG uwsgi [-] application_credential.caching = True {{(pid=81408) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 13 04:07:37.650831 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG uwsgi [-] application_credential.driver = sql {{(pid=81409) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 13 04:07:37.650942 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG uwsgi [-] application_credential.driver = sql {{(pid=81408) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 13 04:07:37.651053 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG uwsgi [-] application_credential.user_limit = -1 {{(pid=81409) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 13 04:07:37.651170 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG uwsgi [-] application_credential.user_limit = -1 {{(pid=81408) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 13 04:07:37.651294 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG uwsgi [-] assignment.driver = sql {{(pid=81409) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 13 04:07:37.651408 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG uwsgi [-] assignment.driver = sql {{(pid=81408) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 13 04:07:37.651519 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG uwsgi [-] assignment.prohibited_implied_role = ['admin'] {{(pid=81409) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 13 04:07:37.651631 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG uwsgi [-] assignment.prohibited_implied_role = ['admin'] {{(pid=81408) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 13 04:07:37.651754 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG uwsgi [-] auth.application_credential = None {{(pid=81409) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 13 04:07:37.651897 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG uwsgi [-] auth.application_credential = None {{(pid=81408) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 13 04:07:37.652019 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG uwsgi [-] auth.external = None {{(pid=81409) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 13 04:07:37.652133 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG uwsgi [-] auth.external = None {{(pid=81408) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 13 04:07:37.652254 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG uwsgi [-] auth.mapped = None {{(pid=81409) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 13 04:07:37.652368 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG uwsgi [-] auth.mapped = None {{(pid=81408) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 13 04:07:37.652480 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG uwsgi [-] auth.methods = ['external', 'password', 'token', 'oauth1', 'mapped', 'application_credential'] {{(pid=81409) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 13 04:07:37.652593 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG uwsgi [-] auth.methods = ['external', 'password', 'token', 'oauth1', 'mapped', 'application_credential'] {{(pid=81408) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 13 04:07:37.652711 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG uwsgi [-] auth.oauth1 = None {{(pid=81409) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 13 04:07:37.652831 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG uwsgi [-] auth.oauth1 = None {{(pid=81408) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 13 04:07:37.652952 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG uwsgi [-] auth.password = None {{(pid=81409) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 13 04:07:37.653067 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG uwsgi [-] auth.password = None {{(pid=81408) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 13 04:07:37.653178 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG uwsgi [-] auth.token = None {{(pid=81409) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 13 04:07:37.653291 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG uwsgi [-] auth.token = None {{(pid=81408) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 13 04:07:37.653402 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG uwsgi [-] catalog.cache_time = None {{(pid=81409) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 13 04:07:37.653514 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG uwsgi [-] catalog.cache_time = None {{(pid=81408) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 13 04:07:37.653633 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG uwsgi [-] catalog.caching = True {{(pid=81409) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 13 04:07:37.653746 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG uwsgi [-] catalog.caching = True {{(pid=81408) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 13 04:07:37.653858 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG uwsgi [-] catalog.driver = sql {{(pid=81409) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 13 04:07:37.653976 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG uwsgi [-] catalog.driver = sql {{(pid=81408) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 13 04:07:37.654089 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG uwsgi [-] catalog.list_limit = None {{(pid=81409) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 13 04:07:37.654238 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG uwsgi [-] catalog.list_limit = None {{(pid=81408) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 13 04:07:37.654367 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG uwsgi [-] catalog.template_file = default_catalog.templates {{(pid=81409) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 13 04:07:37.654482 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG uwsgi [-] catalog.template_file = default_catalog.templates {{(pid=81408) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 13 04:07:37.654595 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG uwsgi [-] credential.auth_ttl = 15 {{(pid=81409) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 13 04:07:37.654706 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG uwsgi [-] credential.auth_ttl = 15 {{(pid=81408) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 13 04:07:37.654818 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG uwsgi [-] credential.cache_time = None {{(pid=81409) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 13 04:07:37.654929 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG uwsgi [-] credential.cache_time = None {{(pid=81408) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 13 04:07:37.655049 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG uwsgi [-] credential.caching = True {{(pid=81409) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 13 04:07:37.655163 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG uwsgi [-] credential.caching = True {{(pid=81408) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 13 04:07:37.655274 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG uwsgi [-] credential.driver = sql {{(pid=81409) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 13 04:07:37.655386 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG uwsgi [-] credential.driver = sql {{(pid=81408) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 13 04:07:37.655498 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG uwsgi [-] credential.key_repository = /etc/keystone/credential-keys/ {{(pid=81409) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 13 04:07:37.655610 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG uwsgi [-] credential.key_repository = /etc/keystone/credential-keys/ {{(pid=81408) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 13 04:07:37.655734 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG uwsgi [-] credential.provider = fernet {{(pid=81409) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 13 04:07:37.655848 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG uwsgi [-] credential.provider = fernet {{(pid=81408) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 13 04:07:37.655960 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG uwsgi [-] credential.user_limit = -1 {{(pid=81409) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 13 04:07:37.656071 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG uwsgi [-] credential.user_limit = -1 {{(pid=81408) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 13 04:07:37.656183 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG uwsgi [-] domain_config.cache_time = 300 {{(pid=81409) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 13 04:07:37.656294 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG uwsgi [-] domain_config.cache_time = 300 {{(pid=81408) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 13 04:07:37.656414 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG uwsgi [-] domain_config.caching = True {{(pid=81409) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 13 04:07:37.656532 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG uwsgi [-] domain_config.caching = True {{(pid=81408) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 13 04:07:37.656644 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG uwsgi [-] domain_config.driver = sql {{(pid=81409) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 13 04:07:37.656756 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG uwsgi [-] domain_config.driver = sql {{(pid=81408) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 13 04:07:37.656868 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG uwsgi [-] endpoint_filter.driver = sql {{(pid=81409) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 13 04:07:37.656979 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG uwsgi [-] endpoint_filter.driver = sql {{(pid=81408) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 13 04:07:37.657101 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG uwsgi [-] endpoint_filter.return_all_endpoints_if_no_filter = True {{(pid=81409) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 13 04:07:37.657215 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG uwsgi [-] endpoint_filter.return_all_endpoints_if_no_filter = True {{(pid=81408) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 13 04:07:37.657327 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG uwsgi [-] endpoint_policy.driver = sql {{(pid=81409) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 13 04:07:37.657438 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG uwsgi [-] endpoint_policy.driver = sql {{(pid=81408) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 13 04:07:37.657551 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG uwsgi [-] eventlet_server.admin_bind_host = 0.0.0.0 {{(pid=81409) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 13 04:07:37.657669 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG uwsgi [-] eventlet_server.admin_bind_host = 0.0.0.0 {{(pid=81408) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 13 04:07:37.657790 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG uwsgi [-] eventlet_server.admin_port = 35357 {{(pid=81409) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 13 04:07:37.657904 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG uwsgi [-] eventlet_server.admin_port = 35357 {{(pid=81408) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 13 04:07:37.658016 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG uwsgi [-] eventlet_server.public_bind_host = 0.0.0.0 {{(pid=81409) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 13 04:07:37.658126 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG uwsgi [-] eventlet_server.public_bind_host = 0.0.0.0 {{(pid=81408) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 13 04:07:37.658251 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG uwsgi [-] eventlet_server.public_port = 5000 {{(pid=81409) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 13 04:07:37.658365 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG uwsgi [-] eventlet_server.public_port = 5000 {{(pid=81408) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 13 04:07:37.658493 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG uwsgi [-] federation.assertion_prefix = {{(pid=81409) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 13 04:07:37.658609 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG uwsgi [-] federation.assertion_prefix = {{(pid=81408) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 13 04:07:37.658721 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG uwsgi [-] federation.caching = True {{(pid=81409) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 13 04:07:37.658833 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG uwsgi [-] federation.caching = True {{(pid=81408) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 13 04:07:37.658944 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG uwsgi [-] federation.default_authorization_ttl = 0 {{(pid=81409) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 13 04:07:37.659055 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG uwsgi [-] federation.default_authorization_ttl = 0 {{(pid=81408) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 13 04:07:37.659174 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG uwsgi [-] federation.driver = sql {{(pid=81409) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 13 04:07:37.659288 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG uwsgi [-] federation.driver = sql {{(pid=81408) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 13 04:07:37.659400 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG uwsgi [-] federation.federated_domain_name = Federated {{(pid=81409) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 13 04:07:37.659511 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG uwsgi [-] federation.federated_domain_name = Federated {{(pid=81408) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 13 04:07:37.659623 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG uwsgi [-] federation.remote_id_attribute = None {{(pid=81409) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 13 04:07:37.659735 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG uwsgi [-] federation.remote_id_attribute = None {{(pid=81408) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 13 04:07:37.659854 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG uwsgi [-] federation.sso_callback_template = /etc/keystone/sso_callback_template.html {{(pid=81409) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 13 04:07:37.659968 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG uwsgi [-] federation.sso_callback_template = /etc/keystone/sso_callback_template.html {{(pid=81408) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 13 04:07:37.660081 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG uwsgi [-] federation.trusted_dashboard = [] {{(pid=81409) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 13 04:07:37.660192 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG uwsgi [-] federation.trusted_dashboard = [] {{(pid=81408) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 13 04:07:37.660303 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG uwsgi [-] fernet_receipts.key_repository = /etc/keystone/fernet-keys/ {{(pid=81409) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 13 04:07:37.660422 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG uwsgi [-] fernet_receipts.key_repository = /etc/keystone/fernet-keys/ {{(pid=81408) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 13 04:07:37.660536 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG uwsgi [-] fernet_receipts.max_active_keys = 3 {{(pid=81409) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 13 04:07:37.660648 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG uwsgi [-] fernet_receipts.max_active_keys = 3 {{(pid=81408) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 13 04:07:37.660760 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG uwsgi [-] fernet_tokens.key_repository = /etc/keystone/fernet-keys/ {{(pid=81409) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 13 04:07:37.660871 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG uwsgi [-] fernet_tokens.key_repository = /etc/keystone/fernet-keys/ {{(pid=81408) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 13 04:07:37.660983 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG uwsgi [-] fernet_tokens.max_active_keys = 3 {{(pid=81409) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 13 04:07:37.661105 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG uwsgi [-] fernet_tokens.max_active_keys = 3 {{(pid=81408) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 13 04:07:37.661219 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG uwsgi [-] identity.cache_time = 600 {{(pid=81409) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 13 04:07:37.661331 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG uwsgi [-] identity.cache_time = 600 {{(pid=81408) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 13 04:07:37.661442 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG uwsgi [-] identity.caching = True {{(pid=81409) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 13 04:07:37.661567 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG uwsgi [-] identity.caching = True {{(pid=81408) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 13 04:07:37.661681 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG uwsgi [-] identity.default_domain_id = default {{(pid=81409) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 13 04:07:37.661864 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG uwsgi [-] identity.default_domain_id = default {{(pid=81408) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 13 04:07:37.661980 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG uwsgi [-] identity.domain_config_dir = /etc/keystone/domains {{(pid=81409) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 13 04:07:37.662092 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG uwsgi [-] identity.domain_config_dir = /etc/keystone/domains {{(pid=81408) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 13 04:07:37.662264 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG uwsgi [-] identity.domain_configurations_from_database = False {{(pid=81409) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 13 04:07:37.662381 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG uwsgi [-] identity.domain_configurations_from_database = False {{(pid=81408) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 13 04:07:37.662493 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG uwsgi [-] identity.domain_specific_drivers_enabled = False {{(pid=81409) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 13 04:07:37.662617 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG uwsgi [-] identity.domain_specific_drivers_enabled = False {{(pid=81408) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 13 04:07:37.662733 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG uwsgi [-] identity.driver = sql {{(pid=81409) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 13 04:07:37.662844 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG uwsgi [-] identity.driver = sql {{(pid=81408) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 13 04:07:37.662956 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG uwsgi [-] identity.list_limit = None {{(pid=81409) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 13 04:07:37.663067 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG uwsgi [-] identity.list_limit = None {{(pid=81408) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 13 04:07:37.663179 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG uwsgi [-] identity.max_password_length = 4096 {{(pid=81409) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 13 04:07:37.663297 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG uwsgi [-] identity.max_password_length = 4096 {{(pid=81408) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 13 04:07:37.663411 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG uwsgi [-] identity.password_hash_algorithm = bcrypt {{(pid=81409) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 13 04:07:37.663523 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG uwsgi [-] identity.password_hash_algorithm = bcrypt {{(pid=81408) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 13 04:07:37.663634 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG uwsgi [-] identity.password_hash_rounds = 4 {{(pid=81409) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 13 04:07:37.663745 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG uwsgi [-] identity.password_hash_rounds = 4 {{(pid=81408) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 13 04:07:37.663858 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG uwsgi [-] identity.salt_bytesize = None {{(pid=81409) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 13 04:07:37.663968 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG uwsgi [-] identity.salt_bytesize = None {{(pid=81408) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 13 04:07:37.664150 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG uwsgi [-] identity.scrypt_block_size = None {{(pid=81409) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 13 04:07:37.664852 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG uwsgi [-] identity.scrypt_block_size = None {{(pid=81408) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 13 04:07:37.665026 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG uwsgi [-] identity.scrypt_parallelism = None {{(pid=81409) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 13 04:07:37.665205 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG uwsgi [-] identity.scrypt_parallelism = None {{(pid=81408) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 13 04:07:37.665377 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG uwsgi [-] identity_mapping.backward_compatible_ids = True {{(pid=81409) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 13 04:07:37.665572 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG uwsgi [-] identity_mapping.backward_compatible_ids = True {{(pid=81408) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 13 04:07:37.665752 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG uwsgi [-] identity_mapping.driver = sql {{(pid=81409) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 13 04:07:37.665935 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG uwsgi [-] identity_mapping.driver = sql {{(pid=81408) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 13 04:07:37.666277 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG uwsgi [-] identity_mapping.generator = sha256 {{(pid=81409) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 13 04:07:37.666544 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG uwsgi [-] identity_mapping.generator = sha256 {{(pid=81408) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 13 04:07:37.666746 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG uwsgi [-] jwt_tokens.jws_private_key_repository = /etc/keystone/jws-keys/private {{(pid=81409) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 13 04:07:37.666888 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG uwsgi [-] jwt_tokens.jws_private_key_repository = /etc/keystone/jws-keys/private {{(pid=81408) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 13 04:07:37.667055 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG uwsgi [-] jwt_tokens.jws_public_key_repository = /etc/keystone/jws-keys/public {{(pid=81409) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 13 04:07:37.667178 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG uwsgi [-] jwt_tokens.jws_public_key_repository = /etc/keystone/jws-keys/public {{(pid=81408) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 13 04:07:37.667293 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG uwsgi [-] ldap.alias_dereferencing = default {{(pid=81409) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 13 04:07:37.667405 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG uwsgi [-] ldap.alias_dereferencing = default {{(pid=81408) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 13 04:07:37.667517 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG uwsgi [-] ldap.auth_pool_connection_lifetime = 60 {{(pid=81409) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 13 04:07:37.667638 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG uwsgi [-] ldap.auth_pool_connection_lifetime = 60 {{(pid=81408) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 13 04:07:37.667752 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG uwsgi [-] ldap.auth_pool_size = 100 {{(pid=81409) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 13 04:07:37.667864 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG uwsgi [-] ldap.auth_pool_size = 100 {{(pid=81408) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 13 04:07:37.667977 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG uwsgi [-] ldap.chase_referrals = None {{(pid=81409) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 13 04:07:37.668089 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG uwsgi [-] ldap.chase_referrals = None {{(pid=81408) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 13 04:07:37.668200 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG uwsgi [-] ldap.connection_timeout = -1 {{(pid=81409) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 13 04:07:37.668320 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG uwsgi [-] ldap.connection_timeout = -1 {{(pid=81408) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 13 04:07:37.668434 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG uwsgi [-] ldap.debug_level = None {{(pid=81409) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 13 04:07:37.668546 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG uwsgi [-] ldap.debug_level = None {{(pid=81408) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 13 04:07:37.668657 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG uwsgi [-] ldap.group_ad_nesting = False {{(pid=81409) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 13 04:07:37.668769 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG uwsgi [-] ldap.group_ad_nesting = False {{(pid=81408) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 13 04:07:37.668880 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG uwsgi [-] ldap.group_additional_attribute_mapping = [] {{(pid=81409) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 13 04:07:37.669001 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG uwsgi [-] ldap.group_additional_attribute_mapping = [] {{(pid=81408) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 13 04:07:37.669116 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG uwsgi [-] ldap.group_attribute_ignore = [] {{(pid=81409) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 13 04:07:37.669227 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG uwsgi [-] ldap.group_attribute_ignore = [] {{(pid=81408) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 13 04:07:37.669339 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG uwsgi [-] ldap.group_desc_attribute = description {{(pid=81409) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 13 04:07:37.669450 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG uwsgi [-] ldap.group_desc_attribute = description {{(pid=81408) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 13 04:07:37.669561 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG uwsgi [-] ldap.group_filter = None {{(pid=81409) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 13 04:07:37.669681 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG uwsgi [-] ldap.group_filter = None {{(pid=81408) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 13 04:07:37.669793 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG uwsgi [-] ldap.group_id_attribute = cn {{(pid=81409) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 13 04:07:37.669905 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG uwsgi [-] ldap.group_id_attribute = cn {{(pid=81408) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 13 04:07:37.670017 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG uwsgi [-] ldap.group_member_attribute = member {{(pid=81409) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 13 04:07:37.670129 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG uwsgi [-] ldap.group_member_attribute = member {{(pid=81408) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 13 04:07:37.670256 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG uwsgi [-] ldap.group_members_are_ids = False {{(pid=81409) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 13 04:07:37.670383 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG uwsgi [-] ldap.group_members_are_ids = False {{(pid=81408) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 13 04:07:37.670498 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG uwsgi [-] ldap.group_name_attribute = ou {{(pid=81409) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 13 04:07:37.670610 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG uwsgi [-] ldap.group_name_attribute = ou {{(pid=81408) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 13 04:07:37.670721 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG uwsgi [-] ldap.group_objectclass = groupOfNames {{(pid=81409) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 13 04:07:37.670833 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG uwsgi [-] ldap.group_objectclass = groupOfNames {{(pid=81408) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 13 04:07:37.670956 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG uwsgi [-] ldap.group_tree_dn = None {{(pid=81409) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 13 04:07:37.671069 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG uwsgi [-] ldap.group_tree_dn = None {{(pid=81408) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 13 04:07:37.671182 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG uwsgi [-] ldap.page_size = 0 {{(pid=81409) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 13 04:07:37.671294 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG uwsgi [-] ldap.page_size = 0 {{(pid=81408) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 13 04:07:37.671406 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG uwsgi [-] ldap.password = **** {{(pid=81409) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 13 04:07:37.671517 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG uwsgi [-] ldap.password = **** {{(pid=81408) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 13 04:07:37.671637 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG uwsgi [-] ldap.pool_connection_lifetime = 600 {{(pid=81409) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 13 04:07:37.671750 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG uwsgi [-] ldap.pool_connection_lifetime = 600 {{(pid=81408) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 13 04:07:37.671862 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG uwsgi [-] ldap.pool_connection_timeout = -1 {{(pid=81409) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 13 04:07:37.671974 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG uwsgi [-] ldap.pool_connection_timeout = -1 {{(pid=81408) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 13 04:07:37.672085 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG uwsgi [-] ldap.pool_retry_delay = 0.1 {{(pid=81409) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 13 04:07:37.672197 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG uwsgi [-] ldap.pool_retry_delay = 0.1 {{(pid=81408) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 13 04:07:37.672322 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG uwsgi [-] ldap.pool_retry_max = 3 {{(pid=81409) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 13 04:07:37.672436 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG uwsgi [-] ldap.pool_retry_max = 3 {{(pid=81408) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 13 04:07:37.672548 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG uwsgi [-] ldap.pool_size = 10 {{(pid=81409) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 13 04:07:37.672661 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG uwsgi [-] ldap.pool_size = 10 {{(pid=81408) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 13 04:07:37.672772 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG uwsgi [-] ldap.query_scope = one {{(pid=81409) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 13 04:07:37.672884 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG uwsgi [-] ldap.query_scope = one {{(pid=81408) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 13 04:07:37.673003 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG uwsgi [-] ldap.suffix = cn=example,cn=com {{(pid=81409) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 13 04:07:37.673118 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG uwsgi [-] ldap.suffix = cn=example,cn=com {{(pid=81408) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 13 04:07:37.673231 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG uwsgi [-] ldap.tls_cacertdir = None {{(pid=81409) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 13 04:07:37.673343 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG uwsgi [-] ldap.tls_cacertdir = None {{(pid=81408) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 13 04:07:37.673454 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG uwsgi [-] ldap.tls_cacertfile = None {{(pid=81409) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 13 04:07:37.673567 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG uwsgi [-] ldap.tls_cacertfile = None {{(pid=81408) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 13 04:07:37.673686 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG uwsgi [-] ldap.tls_req_cert = demand {{(pid=81409) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 13 04:07:37.673800 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG uwsgi [-] ldap.tls_req_cert = demand {{(pid=81408) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 13 04:07:37.673912 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG uwsgi [-] ldap.url = ldap://localhost {{(pid=81409) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 13 04:07:37.674024 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG uwsgi [-] ldap.url = ldap://localhost {{(pid=81408) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 13 04:07:37.674181 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG uwsgi [-] ldap.use_auth_pool = True {{(pid=81409) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 13 04:07:37.674313 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG uwsgi [-] ldap.use_auth_pool = True {{(pid=81408) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 13 04:07:37.674437 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG uwsgi [-] ldap.use_pool = True {{(pid=81409) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 13 04:07:37.674552 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG uwsgi [-] ldap.use_pool = True {{(pid=81408) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 13 04:07:37.674671 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG uwsgi [-] ldap.use_tls = False {{(pid=81409) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 13 04:07:37.674785 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG uwsgi [-] ldap.use_tls = False {{(pid=81408) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 13 04:07:37.674897 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG uwsgi [-] ldap.user = None {{(pid=81409) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 13 04:07:37.674992 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG uwsgi [-] ldap.user = None {{(pid=81408) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 13 04:07:37.675095 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG uwsgi [-] ldap.user_additional_attribute_mapping = [] {{(pid=81409) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 13 04:07:37.675208 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG uwsgi [-] ldap.user_additional_attribute_mapping = [] {{(pid=81408) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 13 04:07:37.675322 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG uwsgi [-] ldap.user_attribute_ignore = ['default_project_id'] {{(pid=81409) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 13 04:07:37.675434 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG uwsgi [-] ldap.user_attribute_ignore = ['default_project_id'] {{(pid=81408) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 13 04:07:37.675546 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG uwsgi [-] ldap.user_default_project_id_attribute = None {{(pid=81409) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 13 04:07:37.675658 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG uwsgi [-] ldap.user_default_project_id_attribute = None {{(pid=81408) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 13 04:07:37.675777 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG uwsgi [-] ldap.user_description_attribute = description {{(pid=81409) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 13 04:07:37.675891 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG uwsgi [-] ldap.user_description_attribute = description {{(pid=81408) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 13 04:07:37.676002 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG uwsgi [-] ldap.user_enabled_attribute = enabled {{(pid=81409) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 13 04:07:37.676114 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG uwsgi [-] ldap.user_enabled_attribute = enabled {{(pid=81408) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 13 04:07:37.676225 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG uwsgi [-] ldap.user_enabled_default = True {{(pid=81409) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 13 04:07:37.676336 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG uwsgi [-] ldap.user_enabled_default = True {{(pid=81408) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 13 04:07:37.676457 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG uwsgi [-] ldap.user_enabled_emulation = False {{(pid=81409) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 13 04:07:37.676570 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG uwsgi [-] ldap.user_enabled_emulation = False {{(pid=81408) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 13 04:07:37.676682 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG uwsgi [-] ldap.user_enabled_emulation_dn = None {{(pid=81409) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 13 04:07:37.676793 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG uwsgi [-] ldap.user_enabled_emulation_dn = None {{(pid=81408) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 13 04:07:37.676904 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG uwsgi [-] ldap.user_enabled_emulation_use_group_config = False {{(pid=81409) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 13 04:07:37.677020 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG uwsgi [-] ldap.user_enabled_emulation_use_group_config = False {{(pid=81408) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 13 04:07:37.677140 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG uwsgi [-] ldap.user_enabled_invert = False {{(pid=81409) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 13 04:07:37.677254 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG uwsgi [-] ldap.user_enabled_invert = False {{(pid=81408) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 13 04:07:37.677366 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG uwsgi [-] ldap.user_enabled_mask = 0 {{(pid=81409) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 13 04:07:37.677479 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG uwsgi [-] ldap.user_enabled_mask = 0 {{(pid=81408) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 13 04:07:37.677591 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG uwsgi [-] ldap.user_filter = None {{(pid=81409) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 13 04:07:37.677702 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG uwsgi [-] ldap.user_filter = None {{(pid=81408) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 13 04:07:37.677702 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG uwsgi [-] ldap.user_id_attribute = cn {{(pid=81408) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 13 04:07:37.677922 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG uwsgi [-] ldap.user_id_attribute = cn {{(pid=81409) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 13 04:07:37.678034 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG uwsgi [-] ldap.user_mail_attribute = mail {{(pid=81408) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 13 04:07:37.678145 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG uwsgi [-] ldap.user_mail_attribute = mail {{(pid=81409) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 13 04:07:37.678282 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG uwsgi [-] ldap.user_name_attribute = sn {{(pid=81408) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 13 04:07:37.678396 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG uwsgi [-] ldap.user_name_attribute = sn {{(pid=81409) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 13 04:07:37.678521 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG uwsgi [-] ldap.user_objectclass = inetOrgPerson {{(pid=81408) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 13 04:07:37.678636 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG uwsgi [-] ldap.user_objectclass = inetOrgPerson {{(pid=81409) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 13 04:07:37.678748 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG uwsgi [-] ldap.user_pass_attribute = userPassword {{(pid=81408) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 13 04:07:37.678859 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG uwsgi [-] ldap.user_pass_attribute = userPassword {{(pid=81409) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 13 04:07:37.678971 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG uwsgi [-] ldap.user_tree_dn = None {{(pid=81408) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 13 04:07:37.679082 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG uwsgi [-] ldap.user_tree_dn = None {{(pid=81409) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 13 04:07:37.679202 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG uwsgi [-] memcache.dead_retry = 300 {{(pid=81408) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 13 04:07:37.679315 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG uwsgi [-] memcache.dead_retry = 300 {{(pid=81409) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 13 04:07:37.679427 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG uwsgi [-] memcache.pool_connection_get_timeout = 10 {{(pid=81408) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 13 04:07:37.679540 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG uwsgi [-] memcache.pool_connection_get_timeout = 10 {{(pid=81409) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 13 04:07:37.679703 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG uwsgi [-] memcache.pool_maxsize = 10 {{(pid=81408) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 13 04:07:37.679831 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG uwsgi [-] memcache.pool_maxsize = 10 {{(pid=81409) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 13 04:07:37.680027 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG uwsgi [-] memcache.pool_unused_timeout = 60 {{(pid=81408) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 13 04:07:37.680221 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG uwsgi [-] memcache.pool_unused_timeout = 60 {{(pid=81409) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 13 04:07:37.680416 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG uwsgi [-] memcache.socket_timeout = 3 {{(pid=81408) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 13 04:07:37.680541 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG uwsgi [-] memcache.socket_timeout = 3 {{(pid=81409) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 13 04:07:37.680655 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG uwsgi [-] oauth1.access_token_duration = 86400 {{(pid=81408) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 13 04:07:37.680767 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG uwsgi [-] oauth1.access_token_duration = 86400 {{(pid=81409) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 13 04:07:37.680890 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG uwsgi [-] oauth1.driver = sql {{(pid=81408) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 13 04:07:37.681005 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG uwsgi [-] oauth1.driver = sql {{(pid=81409) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 13 04:07:37.681117 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG uwsgi [-] oauth1.request_token_duration = 28800 {{(pid=81408) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 13 04:07:37.681230 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG uwsgi [-] oauth1.request_token_duration = 28800 {{(pid=81409) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 13 04:07:37.681342 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG uwsgi [-] policy.driver = sql {{(pid=81408) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 13 04:07:37.681454 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG uwsgi [-] policy.driver = sql {{(pid=81409) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 13 04:07:37.681575 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG uwsgi [-] policy.list_limit = None {{(pid=81408) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 13 04:07:37.681690 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG uwsgi [-] policy.list_limit = None {{(pid=81409) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 13 04:07:37.681803 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG uwsgi [-] receipt.cache_on_issue = True {{(pid=81408) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 13 04:07:37.681914 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG uwsgi [-] receipt.cache_on_issue = True {{(pid=81409) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 13 04:07:37.682026 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG uwsgi [-] receipt.cache_time = 300 {{(pid=81408) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 13 04:07:37.682138 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG uwsgi [-] receipt.cache_time = 300 {{(pid=81409) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 13 04:07:37.682287 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG uwsgi [-] receipt.caching = True {{(pid=81408) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 13 04:07:37.682404 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG uwsgi [-] receipt.caching = True {{(pid=81409) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 13 04:07:37.682522 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG uwsgi [-] receipt.expiration = 300 {{(pid=81408) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 13 04:07:37.682636 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG uwsgi [-] receipt.expiration = 300 {{(pid=81409) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 13 04:07:37.682749 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG uwsgi [-] receipt.provider = fernet {{(pid=81408) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 13 04:07:37.682867 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG uwsgi [-] receipt.provider = fernet {{(pid=81409) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 13 04:07:37.682988 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG uwsgi [-] resource.admin_project_domain_name = None {{(pid=81408) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 13 04:07:37.683103 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG uwsgi [-] resource.admin_project_domain_name = None {{(pid=81409) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 13 04:07:37.683215 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG uwsgi [-] resource.admin_project_name = None {{(pid=81408) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 13 04:07:37.683327 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG uwsgi [-] resource.admin_project_name = None {{(pid=81409) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 13 04:07:37.683438 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG uwsgi [-] resource.cache_time = None {{(pid=81408) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 13 04:07:37.683550 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG uwsgi [-] resource.cache_time = None {{(pid=81409) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 13 04:07:37.683670 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG uwsgi [-] resource.caching = True {{(pid=81408) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 13 04:07:37.683784 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG uwsgi [-] resource.caching = True {{(pid=81409) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 13 04:07:37.683897 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG uwsgi [-] resource.domain_name_url_safe = off {{(pid=81408) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 13 04:07:37.684008 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG uwsgi [-] resource.domain_name_url_safe = off {{(pid=81409) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 13 04:07:37.684120 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG uwsgi [-] resource.driver = sql {{(pid=81408) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 13 04:07:37.684231 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG uwsgi [-] resource.driver = sql {{(pid=81409) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 13 04:07:37.684351 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG uwsgi [-] resource.list_limit = None {{(pid=81408) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 13 04:07:37.684465 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG uwsgi [-] resource.list_limit = None {{(pid=81409) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 13 04:07:37.684577 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG uwsgi [-] resource.project_name_url_safe = off {{(pid=81408) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 13 04:07:37.684688 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG uwsgi [-] resource.project_name_url_safe = off {{(pid=81409) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 13 04:07:37.684801 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG uwsgi [-] revoke.cache_time = 3600 {{(pid=81408) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 13 04:07:37.684913 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG uwsgi [-] revoke.cache_time = 3600 {{(pid=81409) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 13 04:07:37.685033 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG uwsgi [-] revoke.caching = True {{(pid=81408) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 13 04:07:37.685147 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG uwsgi [-] revoke.caching = True {{(pid=81409) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 13 04:07:37.685258 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG uwsgi [-] revoke.driver = sql {{(pid=81408) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 13 04:07:37.685378 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG uwsgi [-] revoke.driver = sql {{(pid=81409) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 13 04:07:37.685491 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG uwsgi [-] revoke.expiration_buffer = 1800 {{(pid=81408) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 13 04:07:37.685676 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG uwsgi [-] revoke.expiration_buffer = 1800 {{(pid=81409) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 13 04:07:37.685797 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG uwsgi [-] role.cache_time = None {{(pid=81408) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 13 04:07:37.685910 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG uwsgi [-] role.cache_time = None {{(pid=81409) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 13 04:07:37.686022 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG uwsgi [-] role.caching = True {{(pid=81408) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 13 04:07:37.686133 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG uwsgi [-] role.caching = True {{(pid=81409) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 13 04:07:37.686284 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG uwsgi [-] role.driver = sql {{(pid=81408) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 13 04:07:37.686412 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG uwsgi [-] role.driver = sql {{(pid=81409) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 13 04:07:37.686526 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG uwsgi [-] role.list_limit = None {{(pid=81408) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 13 04:07:37.686639 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG uwsgi [-] role.list_limit = None {{(pid=81409) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 13 04:07:37.686750 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG uwsgi [-] saml.assertion_expiration_time = 3600 {{(pid=81408) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 13 04:07:37.686863 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG uwsgi [-] saml.assertion_expiration_time = 3600 {{(pid=81409) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 13 04:07:37.686975 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG uwsgi [-] saml.certfile = /etc/keystone/ssl/certs/signing_cert.pem {{(pid=81408) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 13 04:07:37.687099 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG uwsgi [-] saml.certfile = /etc/keystone/ssl/certs/signing_cert.pem {{(pid=81409) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 13 04:07:37.687214 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG uwsgi [-] saml.idp_contact_company = Example, Inc. {{(pid=81408) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 13 04:07:37.687326 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG uwsgi [-] saml.idp_contact_company = Example, Inc. {{(pid=81409) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 13 04:07:37.687438 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG uwsgi [-] saml.idp_contact_email = support@example.com {{(pid=81408) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 13 04:07:37.687549 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG uwsgi [-] saml.idp_contact_email = support@example.com {{(pid=81409) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 13 04:07:37.687661 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG uwsgi [-] saml.idp_contact_name = SAML Identity Provider Support {{(pid=81408) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 13 04:07:37.687781 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG uwsgi [-] saml.idp_contact_name = SAML Identity Provider Support {{(pid=81409) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 13 04:07:37.687896 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG uwsgi [-] saml.idp_contact_surname = Support {{(pid=81408) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 13 04:07:37.688008 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG uwsgi [-] saml.idp_contact_surname = Support {{(pid=81409) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 13 04:07:37.688119 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG uwsgi [-] saml.idp_contact_telephone = +1 800 555 0100 {{(pid=81408) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 13 04:07:37.688231 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG uwsgi [-] saml.idp_contact_telephone = +1 800 555 0100 {{(pid=81409) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 13 04:07:37.688342 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG uwsgi [-] saml.idp_contact_type = other {{(pid=81408) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 13 04:07:37.688462 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG uwsgi [-] saml.idp_contact_type = other {{(pid=81409) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 13 04:07:37.688575 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG uwsgi [-] saml.idp_entity_id = None {{(pid=81408) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 13 04:07:37.688687 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG uwsgi [-] saml.idp_entity_id = None {{(pid=81409) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 13 04:07:37.688798 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG uwsgi [-] saml.idp_lang = en {{(pid=81408) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 13 04:07:37.688910 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG uwsgi [-] saml.idp_lang = en {{(pid=81409) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 13 04:07:37.689029 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG uwsgi [-] saml.idp_metadata_path = /etc/keystone/saml2_idp_metadata.xml {{(pid=81408) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 13 04:07:37.689151 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG uwsgi [-] saml.idp_metadata_path = /etc/keystone/saml2_idp_metadata.xml {{(pid=81409) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 13 04:07:37.689265 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG uwsgi [-] saml.idp_organization_display_name = OpenStack SAML Identity Provider {{(pid=81408) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 13 04:07:37.689378 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG uwsgi [-] saml.idp_organization_display_name = OpenStack SAML Identity Provider {{(pid=81409) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 13 04:07:37.689490 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG uwsgi [-] saml.idp_organization_name = SAML Identity Provider {{(pid=81408) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 13 04:07:37.689601 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG uwsgi [-] saml.idp_organization_name = SAML Identity Provider {{(pid=81409) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 13 04:07:37.689713 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG uwsgi [-] saml.idp_organization_url = https://example.com/ {{(pid=81408) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 13 04:07:37.689833 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG uwsgi [-] saml.idp_organization_url = https://example.com/ {{(pid=81409) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 13 04:07:37.689946 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG uwsgi [-] saml.idp_sso_endpoint = None {{(pid=81408) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 13 04:07:37.690060 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG uwsgi [-] saml.idp_sso_endpoint = None {{(pid=81409) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 13 04:07:37.690172 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG uwsgi [-] saml.keyfile = /etc/keystone/ssl/private/signing_key.pem {{(pid=81408) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 13 04:07:37.690295 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG uwsgi [-] saml.keyfile = /etc/keystone/ssl/private/signing_key.pem {{(pid=81409) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 13 04:07:37.690407 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG uwsgi [-] saml.relay_state_prefix = ss:mem: {{(pid=81408) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 13 04:07:37.690531 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG uwsgi [-] saml.relay_state_prefix = ss:mem: {{(pid=81409) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 13 04:07:37.690645 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG uwsgi [-] saml.xmlsec1_binary = xmlsec1 {{(pid=81408) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 13 04:07:37.690757 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG uwsgi [-] saml.xmlsec1_binary = xmlsec1 {{(pid=81409) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 13 04:07:37.690869 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG uwsgi [-] security_compliance.change_password_upon_first_use = False {{(pid=81408) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 13 04:07:37.690981 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG uwsgi [-] security_compliance.change_password_upon_first_use = False {{(pid=81409) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 13 04:07:37.691102 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG uwsgi [-] security_compliance.disable_user_account_days_inactive = None {{(pid=81408) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 13 04:07:37.691216 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG uwsgi [-] security_compliance.disable_user_account_days_inactive = None {{(pid=81409) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 13 04:07:37.691328 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG uwsgi [-] security_compliance.lockout_duration = 10 {{(pid=81408) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 13 04:07:37.691439 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG uwsgi [-] security_compliance.lockout_duration = 10 {{(pid=81409) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 13 04:07:37.691551 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG uwsgi [-] security_compliance.lockout_failure_attempts = 2 {{(pid=81408) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 13 04:07:37.691662 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG uwsgi [-] security_compliance.lockout_failure_attempts = 2 {{(pid=81409) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 13 04:07:37.691782 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG uwsgi [-] security_compliance.minimum_password_age = 0 {{(pid=81408) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 13 04:07:37.691895 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG uwsgi [-] security_compliance.minimum_password_age = 0 {{(pid=81409) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 13 04:07:37.692007 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG uwsgi [-] security_compliance.password_expires_days = None {{(pid=81408) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 13 04:07:37.692120 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG uwsgi [-] security_compliance.password_expires_days = None {{(pid=81409) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 13 04:07:37.692231 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG uwsgi [-] security_compliance.password_regex = None {{(pid=81408) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 13 04:07:37.692342 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG uwsgi [-] security_compliance.password_regex = None {{(pid=81409) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 13 04:07:37.692461 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG uwsgi [-] security_compliance.password_regex_description = None {{(pid=81408) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 13 04:07:37.692574 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG uwsgi [-] security_compliance.password_regex_description = None {{(pid=81409) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 13 04:07:37.692686 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG uwsgi [-] security_compliance.unique_last_password_count = 2 {{(pid=81408) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 13 04:07:37.692798 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG uwsgi [-] security_compliance.unique_last_password_count = 2 {{(pid=81409) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 13 04:07:37.692910 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG uwsgi [-] shadow_users.driver = sql {{(pid=81408) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 13 04:07:37.693021 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG uwsgi [-] shadow_users.driver = sql {{(pid=81409) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 13 04:07:37.693141 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG uwsgi [-] token.allow_expired_window = 172800 {{(pid=81408) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 13 04:07:37.693255 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG uwsgi [-] token.allow_expired_window = 172800 {{(pid=81409) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 13 04:07:37.693366 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG uwsgi [-] token.allow_rescope_scoped_token = True {{(pid=81408) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 13 04:07:37.693477 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG uwsgi [-] token.allow_rescope_scoped_token = True {{(pid=81409) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 13 04:07:37.693588 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG uwsgi [-] token.cache_on_issue = True {{(pid=81408) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 13 04:07:37.693700 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG uwsgi [-] token.cache_on_issue = True {{(pid=81409) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 13 04:07:37.693818 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG uwsgi [-] token.cache_time = None {{(pid=81408) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 13 04:07:37.693932 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG uwsgi [-] token.cache_time = None {{(pid=81409) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 13 04:07:37.694043 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG uwsgi [-] token.caching = True {{(pid=81408) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 13 04:07:37.694155 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG uwsgi [-] token.caching = True {{(pid=81409) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 13 04:07:37.694334 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG uwsgi [-] token.expiration = 3600 {{(pid=81408) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 13 04:07:37.694447 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG uwsgi [-] token.expiration = 3600 {{(pid=81409) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 13 04:07:37.694570 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG uwsgi [-] token.provider = fernet {{(pid=81408) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 13 04:07:37.694685 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG uwsgi [-] token.provider = fernet {{(pid=81409) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 13 04:07:37.694796 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG uwsgi [-] token.revoke_by_id = True {{(pid=81408) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 13 04:07:37.694908 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG uwsgi [-] token.revoke_by_id = True {{(pid=81409) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 13 04:07:37.695019 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG uwsgi [-] tokenless_auth.issuer_attribute = SSL_CLIENT_I_DN {{(pid=81408) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 13 04:07:37.695130 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG uwsgi [-] tokenless_auth.issuer_attribute = SSL_CLIENT_I_DN {{(pid=81409) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 13 04:07:37.695268 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG uwsgi [-] tokenless_auth.protocol = x509 {{(pid=81408) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 13 04:07:37.695411 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG uwsgi [-] tokenless_auth.protocol = x509 {{(pid=81409) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 13 04:07:37.695603 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG uwsgi [-] tokenless_auth.trusted_issuer = [] {{(pid=81408) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 13 04:07:37.695799 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG uwsgi [-] tokenless_auth.trusted_issuer = [] {{(pid=81409) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 13 04:07:37.695949 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG uwsgi [-] totp.included_previous_windows = 1 {{(pid=81408) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 13 04:07:37.696063 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG uwsgi [-] totp.included_previous_windows = 1 {{(pid=81409) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 13 04:07:37.696185 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG uwsgi [-] trust.allow_redelegation = False {{(pid=81408) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 13 04:07:37.696301 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG uwsgi [-] trust.allow_redelegation = False {{(pid=81409) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 13 04:07:37.696414 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG uwsgi [-] trust.driver = sql {{(pid=81408) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 13 04:07:37.696525 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG uwsgi [-] trust.driver = sql {{(pid=81409) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 13 04:07:37.696637 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG uwsgi [-] trust.max_redelegation_count = 3 {{(pid=81408) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 13 04:07:37.696749 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG uwsgi [-] trust.max_redelegation_count = 3 {{(pid=81409) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 13 04:07:37.696869 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG uwsgi [-] unified_limit.cache_time = None {{(pid=81408) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 13 04:07:37.696982 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG uwsgi [-] unified_limit.cache_time = None {{(pid=81409) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 13 04:07:37.697098 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG uwsgi [-] unified_limit.caching = True {{(pid=81408) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 13 04:07:37.697210 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG uwsgi [-] unified_limit.caching = True {{(pid=81409) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 13 04:07:37.697322 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG uwsgi [-] unified_limit.driver = sql {{(pid=81408) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 13 04:07:37.697434 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG uwsgi [-] unified_limit.driver = sql {{(pid=81409) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 13 04:07:37.697569 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG uwsgi [-] unified_limit.enforcement_model = flat {{(pid=81408) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 13 04:07:37.697742 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG uwsgi [-] unified_limit.enforcement_model = flat {{(pid=81409) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 13 04:07:37.697859 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG uwsgi [-] unified_limit.list_limit = None {{(pid=81408) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 13 04:07:37.697981 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG uwsgi [-] unified_limit.list_limit = None {{(pid=81409) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 13 04:07:37.698112 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG uwsgi [-] wsgi.debug_middleware = False {{(pid=81408) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 13 04:07:37.698257 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG uwsgi [-] wsgi.debug_middleware = False {{(pid=81409) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 13 04:07:37.698375 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG uwsgi [-] database.backend = sqlalchemy {{(pid=81408) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 13 04:07:37.698487 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG uwsgi [-] database.backend = sqlalchemy {{(pid=81409) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 13 04:07:37.698599 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG uwsgi [-] database.connection = **** {{(pid=81408) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 13 04:07:37.698719 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG uwsgi [-] database.connection = **** {{(pid=81409) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 13 04:07:37.698833 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG uwsgi [-] database.connection_debug = 0 {{(pid=81408) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 13 04:07:37.698945 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG uwsgi [-] database.connection_debug = 0 {{(pid=81409) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 13 04:07:37.699056 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG uwsgi [-] database.connection_parameters = {{(pid=81408) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 13 04:07:37.699176 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG uwsgi [-] database.connection_parameters = {{(pid=81409) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 13 04:07:37.699290 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG uwsgi [-] database.connection_recycle_time = 3600 {{(pid=81408) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 13 04:07:37.699410 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG uwsgi [-] database.connection_recycle_time = 3600 {{(pid=81409) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 13 04:07:37.699525 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG uwsgi [-] database.connection_trace = False {{(pid=81408) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 13 04:07:37.699636 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG uwsgi [-] database.connection_trace = False {{(pid=81409) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 13 04:07:37.699748 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG uwsgi [-] database.db_inc_retry_interval = True {{(pid=81408) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 13 04:07:37.699860 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG uwsgi [-] database.db_inc_retry_interval = True {{(pid=81409) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 13 04:07:37.699971 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG uwsgi [-] database.db_max_retries = 20 {{(pid=81408) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 13 04:07:37.700092 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG uwsgi [-] database.db_max_retries = 20 {{(pid=81409) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 13 04:07:37.700205 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG uwsgi [-] database.db_max_retry_interval = 10 {{(pid=81408) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 13 04:07:37.700317 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG uwsgi [-] database.db_max_retry_interval = 10 {{(pid=81409) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 13 04:07:37.700428 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG uwsgi [-] database.db_retry_interval = 1 {{(pid=81408) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 13 04:07:37.700542 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG uwsgi [-] database.db_retry_interval = 1 {{(pid=81409) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 13 04:07:37.700654 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG uwsgi [-] database.max_overflow = 50 {{(pid=81408) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 13 04:07:37.700774 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG uwsgi [-] database.max_overflow = 50 {{(pid=81409) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 13 04:07:37.700888 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG uwsgi [-] database.max_pool_size = 5 {{(pid=81408) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 13 04:07:37.701000 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG uwsgi [-] database.max_pool_size = 5 {{(pid=81409) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 13 04:07:37.701112 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG uwsgi [-] database.max_retries = 10 {{(pid=81408) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 13 04:07:37.701223 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG uwsgi [-] database.max_retries = 10 {{(pid=81409) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 13 04:07:37.701334 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG uwsgi [-] database.mysql_enable_ndb = False {{(pid=81408) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 13 04:07:37.701454 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG uwsgi [-] database.mysql_enable_ndb = False {{(pid=81409) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 13 04:07:37.701570 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG uwsgi [-] database.mysql_sql_mode = TRADITIONAL {{(pid=81408) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 13 04:07:37.701681 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG uwsgi [-] database.mysql_sql_mode = TRADITIONAL {{(pid=81409) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 13 04:07:37.701793 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG uwsgi [-] database.pool_timeout = None {{(pid=81408) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 13 04:07:37.701904 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG uwsgi [-] database.pool_timeout = None {{(pid=81409) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 13 04:07:37.702015 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG uwsgi [-] database.retry_interval = 10 {{(pid=81408) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 13 04:07:37.702135 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG uwsgi [-] database.retry_interval = 10 {{(pid=81409) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 13 04:07:37.702273 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG uwsgi [-] database.slave_connection = **** {{(pid=81408) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 13 04:07:37.702388 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG uwsgi [-] database.slave_connection = **** {{(pid=81409) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 13 04:07:37.702500 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG uwsgi [-] database.sqlite_synchronous = True {{(pid=81408) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 13 04:07:37.702613 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG uwsgi [-] database.sqlite_synchronous = True {{(pid=81409) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 13 04:07:37.702725 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG uwsgi [-] database.use_db_reconnect = False {{(pid=81408) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 13 04:07:37.702849 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG uwsgi [-] database.use_db_reconnect = False {{(pid=81409) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 13 04:07:37.702963 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG uwsgi [-] ******************************************************************************** {{(pid=81408) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2613}} Oct 13 04:07:37.703076 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG uwsgi [-] ******************************************************************************** {{(pid=81409) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2613}} Oct 13 04:07:37.731332 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.notifications [-] Callback: `keystone.application_credential.core.Manager._delete_app_creds_on_user_delete_callback` subscribed to event `identity.user.deleted`. {{(pid=81409) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 13 04:07:37.731485 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.notifications [-] Callback: `keystone.application_credential.core.Manager._delete_app_creds_on_user_delete_callback` subscribed to event `identity.user.deleted`. {{(pid=81408) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 13 04:07:37.731590 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.notifications [-] Callback: `keystone.application_credential.core.Manager._delete_app_creds_on_user_delete_callback` subscribed to event `identity.user.disabled`. {{(pid=81409) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 13 04:07:37.731670 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.notifications [-] Callback: `keystone.application_credential.core.Manager._delete_app_creds_on_user_delete_callback` subscribed to event `identity.user.disabled`. {{(pid=81408) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 13 04:07:37.731773 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.notifications [-] Callback: `keystone.application_credential.core.Manager._delete_app_creds_on_assignment_removal` subscribed to event `identity.remove_application_credentials_for_user.internal`. {{(pid=81409) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 13 04:07:37.731858 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.notifications [-] Callback: `keystone.application_credential.core.Manager._delete_app_creds_on_assignment_removal` subscribed to event `identity.remove_application_credentials_for_user.internal`. {{(pid=81408) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 13 04:07:37.740109 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.notifications [-] Callback: `keystone.assignment.core.Manager._delete_domain_assignments` subscribed to event `identity.domain.deleted`. {{(pid=81408) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 13 04:07:37.740278 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.notifications [-] Callback: `keystone.assignment.core.Manager._delete_domain_assignments` subscribed to event `identity.domain.deleted`. {{(pid=81409) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 13 04:07:37.767951 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.notifications [-] Callback: `keystone.catalog.core.Manager._on_project_or_endpoint_delete` subscribed to event `identity.project.deleted`. {{(pid=81409) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 13 04:07:37.768074 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.notifications [-] Callback: `keystone.catalog.core.Manager._on_project_or_endpoint_delete` subscribed to event `identity.project.deleted`. {{(pid=81408) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 13 04:07:37.768205 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.notifications [-] Callback: `keystone.catalog.core.Manager._on_project_or_endpoint_delete` subscribed to event `identity.endpoint.deleted`. {{(pid=81409) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 13 04:07:37.768283 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.notifications [-] Callback: `keystone.catalog.core.Manager._on_project_or_endpoint_delete` subscribed to event `identity.endpoint.deleted`. {{(pid=81408) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 13 04:07:37.821640 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.notifications [-] Callback: `keystone.federation.core.Manager._cleanup_identity_provider` subscribed to event `identity.domain_deleted.internal`. {{(pid=81408) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 13 04:07:37.822050 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.notifications [-] Callback: `keystone.federation.core.Manager._cleanup_identity_provider` subscribed to event `identity.domain_deleted.internal`. {{(pid=81409) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 13 04:07:37.870515 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.notifications [-] Callback: `keystone.identity.core.Manager._domain_deleted` subscribed to event `identity.domain_deleted.internal`. {{(pid=81408) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 13 04:07:37.870742 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.notifications [-] Callback: `keystone.identity.core.Manager._unset_default_project` subscribed to event `identity.project.deleted`. {{(pid=81408) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 13 04:07:37.871296 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.notifications [-] Callback: `keystone.identity.core.Manager._domain_deleted` subscribed to event `identity.domain_deleted.internal`. {{(pid=81409) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 13 04:07:37.871535 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.notifications [-] Callback: `keystone.identity.core.Manager._unset_default_project` subscribed to event `identity.project.deleted`. {{(pid=81409) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 13 04:07:38.025759 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.notifications [-] Callback: `keystone.revoke.core.Manager._trust_callback` subscribed to event `identity.OS-TRUST:trust.deleted`. {{(pid=81408) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 13 04:07:38.025989 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.notifications [-] Callback: `keystone.revoke.core.Manager._consumer_callback` subscribed to event `identity.OS-OAUTH1:consumer.deleted`. {{(pid=81408) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 13 04:07:38.026076 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.notifications [-] Callback: `keystone.revoke.core.Manager._trust_callback` subscribed to event `identity.OS-TRUST:trust.deleted`. {{(pid=81409) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 13 04:07:38.026229 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.notifications [-] Callback: `keystone.revoke.core.Manager._user_callback` subscribed to event `identity.user.deleted`. {{(pid=81408) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 13 04:07:38.026314 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.notifications [-] Callback: `keystone.revoke.core.Manager._consumer_callback` subscribed to event `identity.OS-OAUTH1:consumer.deleted`. {{(pid=81409) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 13 04:07:38.026458 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.notifications [-] Callback: `keystone.revoke.core.Manager._user_callback` subscribed to event `identity.user.deleted`. {{(pid=81409) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 13 04:07:38.026538 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.notifications [-] Callback: `keystone.revoke.core.Manager._project_callback` subscribed to event `identity.project.deleted`. {{(pid=81408) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 13 04:07:38.026674 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.notifications [-] Callback: `keystone.revoke.core.Manager._user_callback` subscribed to event `identity.user.disabled`. {{(pid=81408) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 13 04:07:38.026754 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.notifications [-] Callback: `keystone.revoke.core.Manager._project_callback` subscribed to event `identity.project.deleted`. {{(pid=81409) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 13 04:07:38.026891 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.notifications [-] Callback: `keystone.revoke.core.Manager._user_callback` subscribed to event `identity.user.disabled`. {{(pid=81409) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 13 04:07:38.026979 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.notifications [-] Callback: `keystone.revoke.core.Manager._user_callback` subscribed to event `identity.persist_revocation_event_for_user.internal`. {{(pid=81408) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 13 04:07:38.027110 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.notifications [-] Callback: `keystone.revoke.core.Manager._user_callback` subscribed to event `identity.persist_revocation_event_for_user.internal`. {{(pid=81409) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 13 04:07:38.050150 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.notifications [-] Callback: `keystone.receipt.provider.Manager._drop_receipt_cache` subscribed to event `identity.OS-TRUST:trust.deleted`. {{(pid=81408) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 13 04:07:38.050409 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.notifications [-] Callback: `keystone.receipt.provider.Manager._drop_receipt_cache` subscribed to event `identity.user.deleted`. {{(pid=81408) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 13 04:07:38.050501 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.notifications [-] Callback: `keystone.receipt.provider.Manager._drop_receipt_cache` subscribed to event `identity.OS-TRUST:trust.deleted`. {{(pid=81409) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 13 04:07:38.050637 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.notifications [-] Callback: `keystone.receipt.provider.Manager._drop_receipt_cache` subscribed to event `identity.domain.deleted`. {{(pid=81408) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 13 04:07:38.050742 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.notifications [-] Callback: `keystone.receipt.provider.Manager._drop_receipt_cache` subscribed to event `identity.user.deleted`. {{(pid=81409) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 13 04:07:38.050863 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.notifications [-] Callback: `keystone.receipt.provider.Manager._drop_receipt_cache` subscribed to event `identity.user.disabled`. {{(pid=81408) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 13 04:07:38.050946 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.notifications [-] Callback: `keystone.receipt.provider.Manager._drop_receipt_cache` subscribed to event `identity.domain.deleted`. {{(pid=81409) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 13 04:07:38.051085 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.notifications [-] Callback: `keystone.receipt.provider.Manager._drop_receipt_cache` subscribed to event `identity.domain.disabled`. {{(pid=81408) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 13 04:07:38.051168 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.notifications [-] Callback: `keystone.receipt.provider.Manager._drop_receipt_cache` subscribed to event `identity.user.disabled`. {{(pid=81409) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 13 04:07:38.051305 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.notifications [-] Callback: `keystone.receipt.provider.Manager._drop_receipt_cache` subscribed to event `identity.domain.disabled`. {{(pid=81409) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 13 04:07:38.051387 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.notifications [-] Callback: `keystone.receipt.provider.Manager._drop_receipt_cache` subscribed to event `identity.project.disabled`. {{(pid=81408) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 13 04:07:38.051516 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.notifications [-] Callback: `keystone.receipt.provider.Manager._drop_receipt_cache` subscribed to event `identity.invalidate_token_cache.internal`. {{(pid=81408) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 13 04:07:38.051597 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.notifications [-] Callback: `keystone.receipt.provider.Manager._drop_receipt_cache` subscribed to event `identity.project.disabled`. {{(pid=81409) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 13 04:07:38.051732 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.notifications [-] Callback: `keystone.receipt.provider.Manager._drop_receipt_cache` subscribed to event `identity.invalidate_token_cache.internal`. {{(pid=81409) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 13 04:07:38.062627 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.notifications [-] Callback: `keystone.trust.core.Manager._on_user_delete` subscribed to event `identity.user.deleted`. {{(pid=81408) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 13 04:07:38.062906 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.notifications [-] Callback: `keystone.trust.core.Manager._on_user_delete` subscribed to event `identity.user.deleted`. {{(pid=81409) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 13 04:07:38.075763 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.notifications [-] Callback: `keystone.token.provider.Manager._drop_token_cache` subscribed to event `identity.OS-TRUST:trust.deleted`. {{(pid=81408) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 13 04:07:38.076002 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.notifications [-] Callback: `keystone.token.provider.Manager._drop_token_cache` subscribed to event `identity.user.deleted`. {{(pid=81408) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 13 04:07:38.076138 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.notifications [-] Callback: `keystone.token.provider.Manager._drop_token_cache` subscribed to event `identity.OS-TRUST:trust.deleted`. {{(pid=81409) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 13 04:07:38.076243 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.notifications [-] Callback: `keystone.token.provider.Manager._drop_token_cache` subscribed to event `identity.domain.deleted`. {{(pid=81408) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 13 04:07:38.076365 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.notifications [-] Callback: `keystone.token.provider.Manager._drop_token_cache` subscribed to event `identity.user.deleted`. {{(pid=81409) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 13 04:07:38.076450 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.notifications [-] Callback: `keystone.token.provider.Manager._drop_token_cache` subscribed to event `identity.user.disabled`. {{(pid=81408) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 13 04:07:38.076593 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.notifications [-] Callback: `keystone.token.provider.Manager._drop_token_cache` subscribed to event `identity.domain.deleted`. {{(pid=81409) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 13 04:07:38.076676 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.notifications [-] Callback: `keystone.token.provider.Manager._drop_token_cache` subscribed to event `identity.domain.disabled`. {{(pid=81408) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 13 04:07:38.076812 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.notifications [-] Callback: `keystone.token.provider.Manager._drop_token_cache` subscribed to event `identity.user.disabled`. {{(pid=81409) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 13 04:07:38.076906 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.notifications [-] Callback: `keystone.token.provider.Manager._drop_token_cache` subscribed to event `identity.project.disabled`. {{(pid=81408) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 13 04:07:38.077053 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.notifications [-] Callback: `keystone.token.provider.Manager._drop_token_cache` subscribed to event `identity.domain.disabled`. {{(pid=81409) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 13 04:07:38.077142 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.notifications [-] Callback: `keystone.token.provider.Manager._drop_token_cache` subscribed to event `identity.invalidate_token_cache.internal`. {{(pid=81408) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 13 04:07:38.077242 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.notifications [-] Callback: `keystone.token.provider.Manager._drop_token_cache` subscribed to event `identity.project.disabled`. {{(pid=81409) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 13 04:07:38.077454 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.notifications [-] Callback: `keystone.token.provider.Manager._drop_token_cache` subscribed to event `identity.invalidate_token_cache.internal`. {{(pid=81409) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 13 04:07:38.093738 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth: ['/auth/projects' {}] {{(pid=81408) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 13 04:07:38.093887 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth: ['/auth/projects' {}] {{(pid=81409) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 13 04:07:38.093971 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.common [-] Adding additional resource route (alternate) to API auth: ['/OS-FEDERATION/projects' {}] {{(pid=81408) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:433}} Oct 13 04:07:38.094098 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.common [-] Adding additional resource route (alternate) to API auth: ['/OS-FEDERATION/projects' {}] {{(pid=81409) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:433}} Oct 13 04:07:38.094302 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth: ['/auth/domains' {}] {{(pid=81408) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 13 04:07:38.094475 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth: ['/auth/domains' {}] {{(pid=81409) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 13 04:07:38.094572 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.common [-] Adding additional resource route (alternate) to API auth: ['/OS-FEDERATION/domains' {}] {{(pid=81408) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:433}} Oct 13 04:07:38.094676 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.common [-] Adding additional resource route (alternate) to API auth: ['/OS-FEDERATION/domains' {}] {{(pid=81409) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:433}} Oct 13 04:07:38.094758 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth: ['/auth/system' {}] {{(pid=81408) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 13 04:07:38.094952 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth: ['/auth/system' {}] {{(pid=81409) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 13 04:07:38.095033 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth: ['/auth/catalog' {}] {{(pid=81408) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 13 04:07:38.095219 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth: ['/auth/catalog' {}] {{(pid=81409) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 13 04:07:38.095301 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth: ['/auth/tokens/OS-PKI/revoked' {}] {{(pid=81408) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 13 04:07:38.095484 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth: ['/auth/tokens/OS-PKI/revoked' {}] {{(pid=81409) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 13 04:07:38.095563 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth: ['/auth/tokens' {}] {{(pid=81408) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 13 04:07:38.095745 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth: ['/auth/tokens' {}] {{(pid=81409) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 13 04:07:38.106017 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth/OS-FEDERATION: ['/auth/OS-FEDERATION/saml2' {}] {{(pid=81408) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 13 04:07:38.106167 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth/OS-FEDERATION: ['/auth/OS-FEDERATION/saml2' {}] {{(pid=81409) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 13 04:07:38.106321 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth/OS-FEDERATION: ['/auth/OS-FEDERATION/saml2/ecp' {}] {{(pid=81408) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 13 04:07:38.106472 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth/OS-FEDERATION: ['/auth/OS-FEDERATION/saml2/ecp' {}] {{(pid=81409) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 13 04:07:38.106596 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth/OS-FEDERATION: ['/auth/OS-FEDERATION/websso/' {}] {{(pid=81408) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 13 04:07:38.106748 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth/OS-FEDERATION: ['/auth/OS-FEDERATION/websso/' {}] {{(pid=81409) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 13 04:07:38.106882 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth/OS-FEDERATION: ['/auth/OS-FEDERATION/identity_providers//protocols//websso' {}] {{(pid=81408) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 13 04:07:38.107039 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth/OS-FEDERATION: ['/auth/OS-FEDERATION/identity_providers//protocols//websso' {}] {{(pid=81409) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 13 04:07:38.113830 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.common [-] Adding standard routes to API credentials for `type` (API Prefix: ) [/credentials, /credentials/] {{(pid=81408) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Oct 13 04:07:38.114028 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.common [-] Adding standard routes to API credentials for `type` (API Prefix: ) [/credentials, /credentials/] {{(pid=81409) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Oct 13 04:07:38.116745 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.common [-] Adding standard routes to API domains for `type` (API Prefix: ) [/domains, /domains/] {{(pid=81408) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Oct 13 04:07:38.116980 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.common [-] Adding standard routes to API domains for `type` (API Prefix: ) [/domains, /domains/] {{(pid=81409) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Oct 13 04:07:38.117062 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains//config' {}] {{(pid=81408) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 13 04:07:38.117367 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains//config/' {}] {{(pid=81408) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 13 04:07:38.117454 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains//config' {}] {{(pid=81409) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 13 04:07:38.117640 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains//config//' {}] {{(pid=81408) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 13 04:07:38.117742 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains//config/' {}] {{(pid=81409) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 13 04:07:38.117928 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains/config/default' {}] {{(pid=81408) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 13 04:07:38.118016 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains//config//' {}] {{(pid=81409) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 13 04:07:38.118195 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains/config//default' {}] {{(pid=81408) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 13 04:07:38.118302 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains/config/default' {}] {{(pid=81409) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 13 04:07:38.118490 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains/config///default' {}] {{(pid=81408) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 13 04:07:38.118585 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains/config//default' {}] {{(pid=81409) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 13 04:07:38.118767 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains//users//roles' {}] {{(pid=81408) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 13 04:07:38.118847 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains/config///default' {}] {{(pid=81409) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 13 04:07:38.119040 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains//users//roles/' {}] {{(pid=81408) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 13 04:07:38.119120 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains//users//roles' {}] {{(pid=81409) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 13 04:07:38.119316 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains//groups//roles' {}] {{(pid=81408) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 13 04:07:38.119398 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains//users//roles/' {}] {{(pid=81409) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 13 04:07:38.119588 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains//groups//roles/' {}] {{(pid=81408) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 13 04:07:38.119668 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains//groups//roles' {}] {{(pid=81409) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 13 04:07:38.119860 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains//groups//roles/' {}] {{(pid=81409) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 13 04:07:38.142571 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.common [-] Adding resource routes to API ec2tokens: ['/ec2tokens' {}] {{(pid=81408) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 13 04:07:38.143042 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.common [-] Adding resource routes to API ec2tokens: ['/ec2tokens' {}] {{(pid=81409) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 13 04:07:38.144036 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.common [-] Adding standard routes to API endpoints for `type` (API Prefix: ) [/endpoints, /endpoints/] {{(pid=81408) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Oct 13 04:07:38.144323 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.common [-] Adding resource routes to API endpoints: ['/endpoints//OS-ENDPOINT-POLICY/policy' {}] {{(pid=81408) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 13 04:07:38.144518 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.common [-] Adding standard routes to API endpoints for `type` (API Prefix: ) [/endpoints, /endpoints/] {{(pid=81409) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Oct 13 04:07:38.144809 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.common [-] Adding resource routes to API endpoints: ['/endpoints//OS-ENDPOINT-POLICY/policy' {}] {{(pid=81409) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 13 04:07:38.149370 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.common [-] Adding standard routes to API groups for `type` (API Prefix: ) [/groups, /groups/] {{(pid=81408) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Oct 13 04:07:38.149588 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.common [-] Adding standard routes to API groups for `type` (API Prefix: ) [/groups, /groups/] {{(pid=81409) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Oct 13 04:07:38.149671 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.common [-] Adding resource routes to API groups: ['/groups//users' {}] {{(pid=81408) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 13 04:07:38.149875 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.common [-] Adding resource routes to API groups: ['/groups//users' {}] {{(pid=81409) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 13 04:07:38.149957 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.common [-] Adding resource routes to API groups: ['/groups//users/' {}] {{(pid=81408) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 13 04:07:38.150162 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.common [-] Adding resource routes to API groups: ['/groups//users/' {}] {{(pid=81409) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 13 04:07:38.156257 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.common [-] Adding standard routes to API limits for `type` (API Prefix: ) [/limits, /limits/] {{(pid=81408) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Oct 13 04:07:38.156534 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.common [-] Adding standard routes to API limits for `type` (API Prefix: ) [/limits, /limits/] {{(pid=81409) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Oct 13 04:07:38.156615 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.common [-] Adding resource routes to API limits: ['/limits/model' {}] {{(pid=81408) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 13 04:07:38.156827 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.common [-] Adding resource routes to API limits: ['/limits/model' {}] {{(pid=81409) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 13 04:07:38.162121 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.common [-] Adding standard routes to API OS-EP-FILTER for `type` (API Prefix: /OS-EP-FILTER) [/endpoint_groups, /endpoint_groups/] {{(pid=81408) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Oct 13 04:07:38.162446 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-EP-FILTER: ['/endpoints//projects' {}] {{(pid=81408) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 13 04:07:38.162530 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.common [-] Adding standard routes to API OS-EP-FILTER for `type` (API Prefix: /OS-EP-FILTER) [/endpoint_groups, /endpoint_groups/] {{(pid=81409) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Oct 13 04:07:38.162739 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-EP-FILTER: ['/projects//endpoints/' {}] {{(pid=81408) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 13 04:07:38.162822 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-EP-FILTER: ['/endpoints//projects' {}] {{(pid=81409) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 13 04:07:38.163031 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-EP-FILTER: ['/projects//endpoints' {}] {{(pid=81408) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 13 04:07:38.163125 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-EP-FILTER: ['/projects//endpoints/' {}] {{(pid=81409) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 13 04:07:38.163311 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-EP-FILTER: ['/projects//endpoint_groups' {}] {{(pid=81408) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 13 04:07:38.163391 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-EP-FILTER: ['/projects//endpoints' {}] {{(pid=81409) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 13 04:07:38.163589 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-EP-FILTER: ['/projects//endpoint_groups' {}] {{(pid=81409) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 13 04:07:38.163669 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-EP-FILTER: ['/endpoint_groups//endpoints' {}] {{(pid=81408) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 13 04:07:38.163860 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-EP-FILTER: ['/endpoint_groups//projects' {}] {{(pid=81408) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 13 04:07:38.163958 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-EP-FILTER: ['/endpoint_groups//endpoints' {}] {{(pid=81409) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 13 04:07:38.164654 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-EP-FILTER: ['/endpoint_groups//projects/' {}] {{(pid=81408) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 13 04:07:38.164746 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-EP-FILTER: ['/endpoint_groups//projects' {}] {{(pid=81409) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 13 04:07:38.165023 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-EP-FILTER: ['/endpoint_groups//projects/' {}] {{(pid=81409) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 13 04:07:38.180342 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-FEDERATION: ['/saml2/metadata' {}] {{(pid=81408) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 13 04:07:38.180624 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-FEDERATION: ['/identity_providers//protocols//auth' {}] {{(pid=81408) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 13 04:07:38.180708 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-FEDERATION: ['/saml2/metadata' {}] {{(pid=81409) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 13 04:07:38.180942 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-FEDERATION: ['/identity_providers//protocols//auth' {}] {{(pid=81409) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 13 04:07:38.184509 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.common [-] Adding standard routes to API identity_providers for `type` (API Prefix: /OS-FEDERATION) [/identity_providers, /identity_providers/] {{(pid=81408) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Oct 13 04:07:38.184772 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.common [-] Adding standard routes to API identity_providers for `type` (API Prefix: /OS-FEDERATION) [/identity_providers, /identity_providers/] {{(pid=81409) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Oct 13 04:07:38.187787 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.common [-] Adding resource routes to API protocols: ['/OS-FEDERATION/identity_providers//protocols/' {}] {{(pid=81408) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 13 04:07:38.188021 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.common [-] Adding resource routes to API protocols: ['/OS-FEDERATION/identity_providers//protocols/' {}] {{(pid=81409) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 13 04:07:38.188105 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.common [-] Adding resource routes to API protocols: ['/OS-FEDERATION/identity_providers//protocols' {}] {{(pid=81408) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 13 04:07:38.188326 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.common [-] Adding resource routes to API protocols: ['/OS-FEDERATION/identity_providers//protocols' {}] {{(pid=81409) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 13 04:07:38.192311 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.common [-] Adding standard routes to API mappings for `type` (API Prefix: /OS-FEDERATION) [/mappings, /mappings/] {{(pid=81408) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Oct 13 04:07:38.192498 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.common [-] Adding standard routes to API mappings for `type` (API Prefix: /OS-FEDERATION) [/mappings, /mappings/] {{(pid=81409) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Oct 13 04:07:38.196052 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.common [-] Adding standard routes to API service_providers for `type` (API Prefix: /OS-FEDERATION) [/service_providers, /service_providers/] {{(pid=81408) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Oct 13 04:07:38.196241 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.common [-] Adding standard routes to API service_providers for `type` (API Prefix: /OS-FEDERATION) [/service_providers, /service_providers/] {{(pid=81409) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Oct 13 04:07:38.199387 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-INHERIT: ['/domains//groups//roles//inherited_to_projects' {}] {{(pid=81408) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 13 04:07:38.199560 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-INHERIT: ['/domains//groups//roles//inherited_to_projects' {}] {{(pid=81409) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 13 04:07:38.199691 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-INHERIT: ['/domains//groups//roles/inherited_to_projects' {}] {{(pid=81408) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 13 04:07:38.199865 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-INHERIT: ['/domains//groups//roles/inherited_to_projects' {}] {{(pid=81409) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 13 04:07:38.199980 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-INHERIT: ['/domains//users//roles//inherited_to_projects' {}] {{(pid=81408) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 13 04:07:38.200156 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-INHERIT: ['/domains//users//roles//inherited_to_projects' {}] {{(pid=81409) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 13 04:07:38.200267 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-INHERIT: ['/domains//users//roles/inherited_to_projects' {}] {{(pid=81408) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 13 04:07:38.200450 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-INHERIT: ['/domains//users//roles/inherited_to_projects' {}] {{(pid=81409) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 13 04:07:38.200538 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-INHERIT: ['/projects//users//roles//inherited_to_projects' {}] {{(pid=81408) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 13 04:07:38.200736 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-INHERIT: ['/projects//users//roles//inherited_to_projects' {}] {{(pid=81409) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 13 04:07:38.200817 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-INHERIT: ['/projects//groups//roles//inherited_to_projects' {}] {{(pid=81408) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 13 04:07:38.201021 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-INHERIT: ['/projects//groups//roles//inherited_to_projects' {}] {{(pid=81409) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 13 04:07:38.216806 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.common [-] Adding standard routes to API OS-OAUTH1 for `type` (API Prefix: /OS-OAUTH1) [/consumers, /consumers/] {{(pid=81408) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Oct 13 04:07:38.216908 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.common [-] Adding standard routes to API OS-OAUTH1 for `type` (API Prefix: /OS-OAUTH1) [/consumers, /consumers/] {{(pid=81409) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Oct 13 04:07:38.217097 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-OAUTH1: ['/request_token' {}] {{(pid=81408) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 13 04:07:38.217179 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-OAUTH1: ['/request_token' {}] {{(pid=81409) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 13 04:07:38.217368 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-OAUTH1: ['/access_token' {}] {{(pid=81408) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 13 04:07:38.217450 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-OAUTH1: ['/access_token' {}] {{(pid=81409) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 13 04:07:38.217633 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-OAUTH1: ['/authorize/' {}] {{(pid=81408) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 13 04:07:38.217715 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-OAUTH1: ['/authorize/' {}] {{(pid=81409) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 13 04:07:38.224774 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-OAUTH2: ['/token' {}] {{(pid=81408) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 13 04:07:38.224866 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-OAUTH2: ['/token' {}] {{(pid=81409) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 13 04:07:38.226347 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.common [-] Adding resource routes to API events: ['/events' {}] {{(pid=81408) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 13 04:07:38.226480 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.common [-] Adding resource routes to API events: ['/events' {}] {{(pid=81409) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 13 04:07:38.228410 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-SIMPLE-CERT: ['/OS-SIMPLE-CERT/ca' {}] {{(pid=81409) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 13 04:07:38.228537 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-SIMPLE-CERT: ['/OS-SIMPLE-CERT/ca' {}] {{(pid=81408) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 13 04:07:38.228692 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-SIMPLE-CERT: ['/OS-SIMPLE-CERT/certificates' {}] {{(pid=81409) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 13 04:07:38.228817 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-SIMPLE-CERT: ['/OS-SIMPLE-CERT/certificates' {}] {{(pid=81408) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 13 04:07:38.231384 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.common [-] Adding standard routes to API policy for `type` (API Prefix: ) [/policies, /policies/] {{(pid=81409) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Oct 13 04:07:38.231673 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.common [-] Adding resource routes to API policy: ['/policies//OS-ENDPOINT-POLICY/endpoints' {}] {{(pid=81409) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 13 04:07:38.231753 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.common [-] Adding standard routes to API policy for `type` (API Prefix: ) [/policies, /policies/] {{(pid=81408) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Oct 13 04:07:38.231961 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.common [-] Adding resource routes to API policy: ['/policies//OS-ENDPOINT-POLICY/endpoints/' {}] {{(pid=81409) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 13 04:07:38.232042 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.common [-] Adding resource routes to API policy: ['/policies//OS-ENDPOINT-POLICY/endpoints' {}] {{(pid=81408) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 13 04:07:38.232247 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.common [-] Adding resource routes to API policy: ['/policies//OS-ENDPOINT-POLICY/endpoints/' {}] {{(pid=81408) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 13 04:07:38.232336 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.common [-] Adding resource routes to API policy: ['/policies//OS-ENDPOINT-POLICY/services/' {}] {{(pid=81409) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 13 04:07:38.232533 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.common [-] Adding resource routes to API policy: ['/policies//OS-ENDPOINT-POLICY/services/' {}] {{(pid=81408) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 13 04:07:38.232651 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.common [-] Adding resource routes to API policy: ['/policies//OS-ENDPOINT-POLICY/services//regions/' {}] {{(pid=81409) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 13 04:07:38.232928 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.common [-] Adding resource routes to API policy: ['/policies//OS-ENDPOINT-POLICY/services//regions/' {}] {{(pid=81408) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 13 04:07:38.243878 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.common [-] Adding standard routes to API projects for `type` (API Prefix: ) [/projects, /projects/] {{(pid=81409) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Oct 13 04:07:38.244169 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.common [-] Adding resource routes to API projects: ['/projects//tags' {}] {{(pid=81409) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 13 04:07:38.244375 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.common [-] Adding standard routes to API projects for `type` (API Prefix: ) [/projects, /projects/] {{(pid=81408) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Oct 13 04:07:38.244455 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.common [-] Adding resource routes to API projects: ['/projects//tags/' {}] {{(pid=81409) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 13 04:07:38.244666 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.common [-] Adding resource routes to API projects: ['/projects//tags' {}] {{(pid=81408) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 13 04:07:38.244755 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.common [-] Adding resource routes to API projects: ['/projects//users//roles/' {}] {{(pid=81409) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 13 04:07:38.244953 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.common [-] Adding resource routes to API projects: ['/projects//tags/' {}] {{(pid=81408) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 13 04:07:38.245103 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.common [-] Adding resource routes to API projects: ['/projects//users//roles' {}] {{(pid=81409) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 13 04:07:38.245237 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.common [-] Adding resource routes to API projects: ['/projects//users//roles/' {}] {{(pid=81408) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 13 04:07:38.245402 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.common [-] Adding resource routes to API projects: ['/projects//groups//roles/' {}] {{(pid=81409) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 13 04:07:38.245629 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.common [-] Adding resource routes to API projects: ['/projects//users//roles' {}] {{(pid=81408) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 13 04:07:38.245717 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.common [-] Adding resource routes to API projects: ['/projects//groups//roles' {}] {{(pid=81409) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 13 04:07:38.245913 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.common [-] Adding resource routes to API projects: ['/projects//groups//roles/' {}] {{(pid=81408) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 13 04:07:38.246199 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.common [-] Adding resource routes to API projects: ['/projects//groups//roles' {}] {{(pid=81408) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 13 04:07:38.261743 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.common [-] Adding standard routes to API regions for `type` (API Prefix: ) [/regions, /regions/] {{(pid=81409) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Oct 13 04:07:38.262183 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.common [-] Adding standard routes to API regions for `type` (API Prefix: ) [/regions, /regions/] {{(pid=81408) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Oct 13 04:07:38.264702 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.common [-] Adding standard routes to API registered_limit for `type` (API Prefix: ) [/registered_limits, /registered_limits/] {{(pid=81409) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Oct 13 04:07:38.265118 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.common [-] Adding standard routes to API registered_limit for `type` (API Prefix: ) [/registered_limits, /registered_limits/] {{(pid=81408) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Oct 13 04:07:38.267654 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.common [-] Adding resource routes to API role_assignments: ['/role_assignments' {}] {{(pid=81409) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 13 04:07:38.268040 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.common [-] Adding resource routes to API role_assignments: ['/role_assignments' {}] {{(pid=81408) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 13 04:07:38.269177 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.common [-] Adding resource routes to API role_inferences: ['/role_inferences' {}] {{(pid=81409) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 13 04:07:38.269482 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.common [-] Adding resource routes to API role_inferences: ['/role_inferences' {}] {{(pid=81408) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 13 04:07:38.270690 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.common [-] Adding standard routes to API roles for `type` (API Prefix: ) [/roles, /roles/] {{(pid=81409) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Oct 13 04:07:38.270979 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.common [-] Adding resource routes to API roles: ['/roles//implies' {}] {{(pid=81409) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 13 04:07:38.271058 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.common [-] Adding standard routes to API roles for `type` (API Prefix: ) [/roles, /roles/] {{(pid=81408) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Oct 13 04:07:38.271261 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.common [-] Adding resource routes to API roles: ['/roles//implies' {}] {{(pid=81408) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 13 04:07:38.271382 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.common [-] Adding resource routes to API roles: ['/roles//implies/' {}] {{(pid=81409) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 13 04:07:38.271668 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.common [-] Adding resource routes to API roles: ['/roles//implies/' {}] {{(pid=81408) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 13 04:07:38.277882 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.common [-] Adding resource routes to API s3tokens: ['/s3tokens' {}] {{(pid=81409) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 13 04:07:38.278054 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.common [-] Adding resource routes to API s3tokens: ['/s3tokens' {}] {{(pid=81408) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 13 04:07:38.279338 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.common [-] Adding standard routes to API services for `type` (API Prefix: ) [/services, /services/] {{(pid=81409) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Oct 13 04:07:38.279541 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.common [-] Adding standard routes to API services for `type` (API Prefix: ) [/services, /services/] {{(pid=81408) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Oct 13 04:07:38.282242 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.common [-] Adding resource routes to API system: ['/system/users//roles' {}] {{(pid=81409) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 13 04:07:38.282452 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.common [-] Adding resource routes to API system: ['/system/users//roles' {}] {{(pid=81408) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 13 04:07:38.282647 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.common [-] Adding resource routes to API system: ['/system/users//roles/' {}] {{(pid=81409) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 13 04:07:38.282851 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.common [-] Adding resource routes to API system: ['/system/users//roles/' {}] {{(pid=81408) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 13 04:07:38.282933 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.common [-] Adding resource routes to API system: ['/system/groups//roles' {}] {{(pid=81409) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 13 04:07:38.283141 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.common [-] Adding resource routes to API system: ['/system/groups//roles' {}] {{(pid=81408) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 13 04:07:38.283236 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.common [-] Adding resource routes to API system: ['/system/groups//roles/' {}] {{(pid=81409) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 13 04:07:38.283422 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.common [-] Adding resource routes to API system: ['/system/groups//roles/' {}] {{(pid=81408) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 13 04:07:38.291553 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.common [-] Adding standard routes to API trusts for `type` (API Prefix: /OS-TRUST) [/trusts, /trusts/] {{(pid=81409) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Oct 13 04:07:38.291847 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.common [-] Adding resource routes to API trusts: ['/trusts//roles' {}] {{(pid=81409) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 13 04:07:38.291928 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.common [-] Adding standard routes to API trusts for `type` (API Prefix: /OS-TRUST) [/trusts, /trusts/] {{(pid=81408) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Oct 13 04:07:38.292139 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.common [-] Adding resource routes to API trusts: ['/trusts//roles/' {}] {{(pid=81409) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 13 04:07:38.292270 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.common [-] Adding resource routes to API trusts: ['/trusts//roles' {}] {{(pid=81408) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 13 04:07:38.292480 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.common [-] Adding resource routes to API trusts: ['/trusts//roles/' {}] {{(pid=81408) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 13 04:07:38.299059 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.common [-] Adding standard routes to API users for `type` (API Prefix: ) [/users, /users/] {{(pid=81409) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Oct 13 04:07:38.299348 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//password' {}] {{(pid=81409) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 13 04:07:38.299443 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.common [-] Adding standard routes to API users for `type` (API Prefix: ) [/users, /users/] {{(pid=81408) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Oct 13 04:07:38.299636 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//password' {}] {{(pid=81408) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 13 04:07:38.299716 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//groups' {}] {{(pid=81409) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 13 04:07:38.299913 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//projects' {}] {{(pid=81409) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 13 04:07:38.299996 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//groups' {}] {{(pid=81408) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 13 04:07:38.300189 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//credentials/OS-EC2' {}] {{(pid=81409) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 13 04:07:38.300279 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//projects' {}] {{(pid=81408) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 13 04:07:38.300463 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//credentials/OS-EC2/' {}] {{(pid=81409) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 13 04:07:38.300544 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//credentials/OS-EC2' {}] {{(pid=81408) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 13 04:07:38.300740 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//OS-OAUTH1/access_tokens' {}] {{(pid=81409) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 13 04:07:38.300820 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//credentials/OS-EC2/' {}] {{(pid=81408) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 13 04:07:38.301013 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//OS-OAUTH1/access_tokens' {}] {{(pid=81408) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 13 04:07:38.301100 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//OS-OAUTH1/access_tokens/' {}] {{(pid=81409) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 13 04:07:38.301284 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//OS-OAUTH1/access_tokens/' {}] {{(pid=81408) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 13 04:07:38.301365 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//OS-OAUTH1/access_tokens//roles' {}] {{(pid=81409) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 13 04:07:38.301564 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//OS-OAUTH1/access_tokens//roles' {}] {{(pid=81408) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 13 04:07:38.301645 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//OS-OAUTH1/access_tokens//roles/' {}] {{(pid=81409) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 13 04:07:38.301845 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//application_credentials' {}] {{(pid=81409) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 13 04:07:38.301924 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//OS-OAUTH1/access_tokens//roles/' {}] {{(pid=81408) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 13 04:07:38.302124 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//application_credentials' {}] {{(pid=81408) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 13 04:07:38.302278 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//application_credentials/' {}] {{(pid=81409) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 13 04:07:38.302422 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//application_credentials/' {}] {{(pid=81408) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 13 04:07:38.302547 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//access_rules' {}] {{(pid=81409) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 13 04:07:38.302694 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//access_rules' {}] {{(pid=81408) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 13 04:07:38.302820 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//access_rules/' {}] {{(pid=81409) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 13 04:07:38.303010 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//access_rules/' {}] {{(pid=81408) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 13 04:07:38.337825 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: WSGI app 0 (mountpoint='') ready in 5 seconds on interpreter 0x557fce1b69c0 pid: 81408 (default app) Oct 13 04:07:38.338094 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: WSGI app 0 (mountpoint='') ready in 5 seconds on interpreter 0x557fce1b69c0 pid: 81409 (default app) Oct 13 04:07:38.341122 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41fc4e1f-98ec-49a6-b127-e3ffc2cfc849 None None] REQUEST_METHOD: `GET` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:07:38.341411 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41fc4e1f-98ec-49a6-b127-e3ffc2cfc849 None None] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:07:38.341685 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41fc4e1f-98ec-49a6-b127-e3ffc2cfc849 None None] PATH_INFO: `/v3` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:07:38.345816 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 1/1] 10.222.0.27 () {50 vars in 791 bytes} [Thu Oct 13 04:07:38 2022] GET /identity/v3/ => generated 255 bytes in 7 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 13 04:07:45.209610 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81f0c1c4-bf4b-4028-9134-2fe2dc94a623 None None] REQUEST_METHOD: `GET` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:07:45.210438 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81f0c1c4-bf4b-4028-9134-2fe2dc94a623 None None] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:07:45.210438 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81f0c1c4-bf4b-4028-9134-2fe2dc94a623 None None] PATH_INFO: `/` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:07:45.214891 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 1/2] 10.222.0.27 () {54 vars in 927 bytes} [Thu Oct 13 04:07:45 2022] GET /identity => generated 270 bytes in 9 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Oct 13 04:07:45.220652 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf0d950f-be26-47f7-87e2-bfbf7d66381e None None] REQUEST_METHOD: `POST` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:07:45.221267 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf0d950f-be26-47f7-87e2-bfbf7d66381e None None] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:07:45.221267 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf0d950f-be26-47f7-87e2-bfbf7d66381e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:07:45.293770 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: INFO dbcounter [None req-cf0d950f-be26-47f7-87e2-bfbf7d66381e None None] Registered counter for database keystone Oct 13 04:07:45.313202 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG oslo_db.sqlalchemy.engines [None req-cf0d950f-be26-47f7-87e2-bfbf7d66381e None None] MySQL server mode set to STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_ENGINE_SUBSTITUTION {{(pid=81408) _check_effective_sql_mode /usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/engines.py:314}} Oct 13 04:07:45.320291 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG dbcounter [-] [81408] Writer thread running {{(pid=81408) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:99}} Oct 13 04:07:45.521922 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG passlib.handlers.bcrypt [None req-cf0d950f-be26-47f7-87e2-bfbf7d66381e None None] detected 'bcrypt' backend, version '4.0.0' {{(pid=81408) _load_backend_mixin /usr/local/lib/python3.8/dist-packages/passlib/handlers/bcrypt.py:625}} Oct 13 04:07:45.522344 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG passlib.handlers.bcrypt [None req-cf0d950f-be26-47f7-87e2-bfbf7d66381e None None] 'bcrypt' backend lacks $2$ support, enabling workaround {{(pid=81408) _finalize_backend_mixin /usr/local/lib/python3.8/dist-packages/passlib/handlers/bcrypt.py:406}} Oct 13 04:07:45.580653 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.auth.core [None req-cf0d950f-be26-47f7-87e2-bfbf7d66381e None None] MFA Rules not processed for user `0e2d979563d6421995451a0e86bf3022`. Rule list: `[]` (Enabled: `True`). {{(pid=81408) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 13 04:07:45.582298 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.fernet_utils [None req-cf0d950f-be26-47f7-87e2-bfbf7d66381e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81408) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:07:45.710634 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 2/3] 10.222.0.27 () {58 vars in 1024 bytes} [Thu Oct 13 04:07:45 2022] POST /identity/v3/auth/tokens => generated 914 bytes in 491 msecs (HTTP/1.1 201) 6 headers in 384 bytes (2 switches on core 0) Oct 13 04:07:46.006900 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-81f0c1c4-bf4b-4028-9134-2fe2dc94a623 None None] Authenticating user token {{(pid=81409) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:07:46.075581 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: INFO dbcounter [None req-81f0c1c4-bf4b-4028-9134-2fe2dc94a623 None None] Registered counter for database keystone Oct 13 04:07:46.096181 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG oslo_db.sqlalchemy.engines [None req-81f0c1c4-bf4b-4028-9134-2fe2dc94a623 None None] MySQL server mode set to STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_ENGINE_SUBSTITUTION {{(pid=81409) _check_effective_sql_mode /usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/engines.py:314}} Oct 13 04:07:46.102615 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG dbcounter [-] [81409] Writer thread running {{(pid=81409) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:99}} Oct 13 04:07:46.219859 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-81f0c1c4-bf4b-4028-9134-2fe2dc94a623 None None] Validating token access rules against request {{(pid=81409) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:07:46.224368 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-82ededef-a9d0-47a4-96b5-07cb4bb5fa7b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8ba823f313144da3b683bacef8fd62e0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81409) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:07:46.224986 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82ededef-a9d0-47a4-96b5-07cb4bb5fa7b admin admin] REQUEST_METHOD: `GET` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:07:46.225264 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82ededef-a9d0-47a4-96b5-07cb4bb5fa7b admin admin] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:07:46.225533 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82ededef-a9d0-47a4-96b5-07cb4bb5fa7b admin admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:07:46.233875 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-82ededef-a9d0-47a4-96b5-07cb4bb5fa7b admin admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:07:46.284642 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: WARNING keystone.common.rbac_enforcer.enforcer [None req-82ededef-a9d0-47a4-96b5-07cb4bb5fa7b admin admin] Deprecated policy rules found. Use oslopolicy-policy-generator and oslopolicy-policy-upgrade to detect and resolve deprecated policies in your configuration. Oct 13 04:07:46.284936 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-82ededef-a9d0-47a4-96b5-07cb4bb5fa7b admin admin] RBAC: Authorization granted {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:07:46.294798 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 2/4] 10.222.0.27 () {58 vars in 1324 bytes} [Thu Oct 13 04:07:46 2022] GET /identity/v3/auth/tokens => generated 914 bytes in 288 msecs (HTTP/1.1 200) 6 headers in 379 bytes (1 switches on core 0) Oct 13 04:07:46.300799 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cf0d950f-be26-47f7-87e2-bfbf7d66381e None None] Authenticating user token {{(pid=81408) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:07:46.318121 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cf0d950f-be26-47f7-87e2-bfbf7d66381e None None] Validating token access rules against request {{(pid=81408) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:07:46.322641 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-40d355c1-bc21-4faa-99f5-2382c2aeeb98 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8ba823f313144da3b683bacef8fd62e0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81408) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:07:46.323740 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40d355c1-bc21-4faa-99f5-2382c2aeeb98 admin admin] REQUEST_METHOD: `GET` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:07:46.324021 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40d355c1-bc21-4faa-99f5-2382c2aeeb98 admin admin] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:07:46.324290 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40d355c1-bc21-4faa-99f5-2382c2aeeb98 admin admin] PATH_INFO: `/v3/projects/8ba823f313144da3b683bacef8fd62e0` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:07:46.325870 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-40d355c1-bc21-4faa-99f5-2382c2aeeb98 admin admin] RBAC: Authorizing `identity:get_project(project_id=8ba823f313144da3b683bacef8fd62e0)` {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:07:46.378744 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: WARNING keystone.common.rbac_enforcer.enforcer [None req-40d355c1-bc21-4faa-99f5-2382c2aeeb98 admin admin] Deprecated policy rules found. Use oslopolicy-policy-generator and oslopolicy-policy-upgrade to detect and resolve deprecated policies in your configuration. Oct 13 04:07:46.379096 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-40d355c1-bc21-4faa-99f5-2382c2aeeb98 admin admin] RBAC: Authorization granted {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:07:46.383637 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 3/5] 10.222.0.27 () {56 vars in 1207 bytes} [Thu Oct 13 04:07:46 2022] GET /identity/v3/projects/8ba823f313144da3b683bacef8fd62e0 => generated 344 bytes in 84 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 13 04:07:48.512721 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b259fba-af9a-435e-8ef7-5a73d4d455e7 None None] REQUEST_METHOD: `GET` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:07:48.513262 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b259fba-af9a-435e-8ef7-5a73d4d455e7 None None] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:07:48.513262 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b259fba-af9a-435e-8ef7-5a73d4d455e7 None None] PATH_INFO: `/` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:07:48.517421 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 3/6] 10.222.0.27 () {54 vars in 927 bytes} [Thu Oct 13 04:07:48 2022] GET /identity => generated 270 bytes in 6 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 13 04:07:48.523073 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f48406e9-27af-4a3c-a1f8-d06d2fbab37a None None] REQUEST_METHOD: `POST` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:07:48.523494 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f48406e9-27af-4a3c-a1f8-d06d2fbab37a None None] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:07:48.523635 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f48406e9-27af-4a3c-a1f8-d06d2fbab37a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:07:48.544056 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.auth.core [None req-f48406e9-27af-4a3c-a1f8-d06d2fbab37a None None] MFA Rules not processed for user `0e2d979563d6421995451a0e86bf3022`. Rule list: `[]` (Enabled: `True`). {{(pid=81408) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 13 04:07:48.545589 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.fernet_utils [None req-f48406e9-27af-4a3c-a1f8-d06d2fbab37a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81408) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:07:48.553403 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 4/7] 10.222.0.27 () {58 vars in 1024 bytes} [Thu Oct 13 04:07:48 2022] POST /identity/v3/auth/tokens => generated 914 bytes in 32 msecs (HTTP/1.1 201) 6 headers in 384 bytes (1 switches on core 0) Oct 13 04:07:48.821400 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4b259fba-af9a-435e-8ef7-5a73d4d455e7 None None] Authenticating user token {{(pid=81409) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:07:48.833472 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4b259fba-af9a-435e-8ef7-5a73d4d455e7 None None] Validating token access rules against request {{(pid=81409) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:07:48.838343 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-28443003-d729-42c1-bd14-933db9b97cd1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8ba823f313144da3b683bacef8fd62e0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81409) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:07:48.838936 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28443003-d729-42c1-bd14-933db9b97cd1 admin admin] REQUEST_METHOD: `GET` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:07:48.839212 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28443003-d729-42c1-bd14-933db9b97cd1 admin admin] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:07:48.839480 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28443003-d729-42c1-bd14-933db9b97cd1 admin admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:07:48.847258 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-28443003-d729-42c1-bd14-933db9b97cd1 admin admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:07:48.849011 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-28443003-d729-42c1-bd14-933db9b97cd1 admin admin] RBAC: Authorization granted {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:07:48.857702 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 4/8] 10.222.0.27 () {58 vars in 1324 bytes} [Thu Oct 13 04:07:48 2022] GET /identity/v3/auth/tokens => generated 914 bytes in 37 msecs (HTTP/1.1 200) 6 headers in 379 bytes (1 switches on core 0) Oct 13 04:07:48.863109 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f48406e9-27af-4a3c-a1f8-d06d2fbab37a None None] Authenticating user token {{(pid=81408) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:07:48.872588 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f48406e9-27af-4a3c-a1f8-d06d2fbab37a None None] Validating token access rules against request {{(pid=81408) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:07:48.877340 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b1bf9583-c05d-439a-b98e-61ddaf96a53d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8ba823f313144da3b683bacef8fd62e0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81408) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:07:48.877964 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1bf9583-c05d-439a-b98e-61ddaf96a53d admin admin] REQUEST_METHOD: `GET` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:07:48.878278 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1bf9583-c05d-439a-b98e-61ddaf96a53d admin admin] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:07:48.878557 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1bf9583-c05d-439a-b98e-61ddaf96a53d admin admin] PATH_INFO: `/v3/users/0e2d979563d6421995451a0e86bf3022` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:07:48.879855 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b1bf9583-c05d-439a-b98e-61ddaf96a53d admin admin] RBAC: Authorizing `identity:get_user(user_id=0e2d979563d6421995451a0e86bf3022)` {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:07:48.881673 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b1bf9583-c05d-439a-b98e-61ddaf96a53d admin admin] RBAC: Authorization granted {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:07:48.883832 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 5/9] 10.222.0.27 () {56 vars in 1198 bytes} [Thu Oct 13 04:07:48 2022] GET /identity/v3/users/0e2d979563d6421995451a0e86bf3022 => generated 247 bytes in 21 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 13 04:07:51.042366 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5d94ae5-87a3-4287-aacd-a6d3b0edc761 None None] REQUEST_METHOD: `GET` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:07:51.042918 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5d94ae5-87a3-4287-aacd-a6d3b0edc761 None None] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:07:51.042918 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5d94ae5-87a3-4287-aacd-a6d3b0edc761 None None] PATH_INFO: `/` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:07:51.044287 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 5/10] 10.222.0.27 () {54 vars in 927 bytes} [Thu Oct 13 04:07:51 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 13 04:07:51.049934 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6892dd4e-5e6a-4e8e-8569-163c42ea9cfc None None] REQUEST_METHOD: `POST` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:07:51.050468 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6892dd4e-5e6a-4e8e-8569-163c42ea9cfc None None] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:07:51.050559 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6892dd4e-5e6a-4e8e-8569-163c42ea9cfc None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:07:51.068853 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.auth.core [None req-6892dd4e-5e6a-4e8e-8569-163c42ea9cfc None None] MFA Rules not processed for user `0e2d979563d6421995451a0e86bf3022`. Rule list: `[]` (Enabled: `True`). {{(pid=81408) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 13 04:07:51.070045 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.fernet_utils [None req-6892dd4e-5e6a-4e8e-8569-163c42ea9cfc None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81408) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:07:51.140996 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-886d5b89-dc42-4a53-ad0c-d13b0697d4be None None] REQUEST_METHOD: `GET` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:07:51.141280 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-886d5b89-dc42-4a53-ad0c-d13b0697d4be None None] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:07:51.141554 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-886d5b89-dc42-4a53-ad0c-d13b0697d4be None None] PATH_INFO: `/` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:07:51.143147 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 6/11] 10.222.0.27 () {54 vars in 927 bytes} [Thu Oct 13 04:07:51 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Oct 13 04:07:51.148024 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9d5848e-d76c-4559-a589-d93f8cf6a992 None None] REQUEST_METHOD: `POST` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:07:51.148307 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9d5848e-d76c-4559-a589-d93f8cf6a992 None None] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:07:51.148581 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9d5848e-d76c-4559-a589-d93f8cf6a992 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:07:51.154184 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 6/12] 10.222.0.27 () {58 vars in 1024 bytes} [Thu Oct 13 04:07:51 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 105 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Oct 13 04:07:51.191222 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG passlib.handlers.bcrypt [None req-b9d5848e-d76c-4559-a589-d93f8cf6a992 None None] detected 'bcrypt' backend, version '4.0.0' {{(pid=81409) _load_backend_mixin /usr/local/lib/python3.8/dist-packages/passlib/handlers/bcrypt.py:625}} Oct 13 04:07:51.191602 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG passlib.handlers.bcrypt [None req-b9d5848e-d76c-4559-a589-d93f8cf6a992 None None] 'bcrypt' backend lacks $2$ support, enabling workaround {{(pid=81409) _finalize_backend_mixin /usr/local/lib/python3.8/dist-packages/passlib/handlers/bcrypt.py:406}} Oct 13 04:07:51.240914 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.auth.core [None req-b9d5848e-d76c-4559-a589-d93f8cf6a992 None None] MFA Rules not processed for user `0e2d979563d6421995451a0e86bf3022`. Rule list: `[]` (Enabled: `True`). {{(pid=81409) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 13 04:07:51.243020 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.fernet_utils [None req-b9d5848e-d76c-4559-a589-d93f8cf6a992 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81409) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:07:51.341771 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 7/13] 10.222.0.27 () {58 vars in 1024 bytes} [Thu Oct 13 04:07:51 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 194 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Oct 13 04:07:51.422270 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6892dd4e-5e6a-4e8e-8569-163c42ea9cfc None None] Authenticating user token {{(pid=81408) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:07:51.448294 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG dbcounter [-] [81408] Writing DB stats keystone:SELECT=100 {{(pid=81408) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:07:51.529843 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6892dd4e-5e6a-4e8e-8569-163c42ea9cfc None None] Validating token access rules against request {{(pid=81408) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:07:51.563283 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7d270614-4b09-43f3-b2f1-ee2521d8f828 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81408) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:07:51.563919 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d270614-4b09-43f3-b2f1-ee2521d8f828 None admin] REQUEST_METHOD: `GET` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:07:51.564218 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d270614-4b09-43f3-b2f1-ee2521d8f828 None admin] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:07:51.564477 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d270614-4b09-43f3-b2f1-ee2521d8f828 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:07:51.575310 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7d270614-4b09-43f3-b2f1-ee2521d8f828 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 13 04:07:51.584168 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application [None req-7d270614-4b09-43f3-b2f1-ee2521d8f828 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Oct 13 04:07:51.584168 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 13 04:07:51.584168 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 13 04:07:51.584168 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 13 04:07:51.584168 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 13 04:07:51.584168 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 13 04:07:51.584168 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 13 04:07:51.584168 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 13 04:07:51.584168 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 13 04:07:51.584168 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 13 04:07:51.584168 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 13 04:07:51.584168 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 13 04:07:51.584168 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 13 04:07:51.584168 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 13 04:07:51.584168 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 13 04:07:51.584168 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 13 04:07:51.585044 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 13 04:07:51.585044 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 13 04:07:51.585044 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 13 04:07:51.585044 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 13 04:07:51.585044 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 13 04:07:51.585044 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 13 04:07:51.585044 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 13 04:07:51.585044 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 13 04:07:51.585044 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 13 04:07:51.585044 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application return self.get_or_create( Oct 13 04:07:51.585044 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 13 04:07:51.585044 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application with Lock( Oct 13 04:07:51.585044 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 13 04:07:51.585044 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application return self._enter() Oct 13 04:07:51.585044 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 13 04:07:51.585044 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 13 04:07:51.585854 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 13 04:07:51.585854 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application return self.creator() Oct 13 04:07:51.585854 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 13 04:07:51.585854 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application created_value = creator( Oct 13 04:07:51.585854 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 13 04:07:51.585854 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 13 04:07:51.585854 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 13 04:07:51.585854 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 13 04:07:51.585854 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 13 04:07:51.585854 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 13 04:07:51.585854 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Oct 13 04:07:51.585854 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application Oct 13 04:07:51.585854 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 7/14] 10.222.0.27 () {56 vars in 1096 bytes} [Thu Oct 13 04:07:51 2022] GET /identity/v3/roles/admin => generated 83 bytes in 164 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 13 04:07:51.591403 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b9d5848e-d76c-4559-a589-d93f8cf6a992 None None] Authenticating user token {{(pid=81409) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:07:51.624810 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7d270614-4b09-43f3-b2f1-ee2521d8f828 None admin] Authenticating user token {{(pid=81408) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:07:51.694841 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b9d5848e-d76c-4559-a589-d93f8cf6a992 None None] Validating token access rules against request {{(pid=81409) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:07:51.697427 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG dbcounter [-] [81409] Writing DB stats keystone:SELECT=100 {{(pid=81409) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:07:51.706370 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG dbcounter [-] [81408] Writing DB stats keystone:SELECT=100 {{(pid=81408) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:07:51.725964 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7d270614-4b09-43f3-b2f1-ee2521d8f828 None admin] Validating token access rules against request {{(pid=81408) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:07:51.734592 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-477cbc79-9210-41fd-b6b9-11ce46d8f26d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81409) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:07:51.735459 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-477cbc79-9210-41fd-b6b9-11ce46d8f26d None admin] REQUEST_METHOD: `GET` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:07:51.735738 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-477cbc79-9210-41fd-b6b9-11ce46d8f26d None admin] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:07:51.736006 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-477cbc79-9210-41fd-b6b9-11ce46d8f26d None admin] PATH_INFO: `/v3/roles` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:07:51.736839 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-477cbc79-9210-41fd-b6b9-11ce46d8f26d None admin] RBAC: Adding query filter params (name=admin) {{(pid=81409) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 13 04:07:51.737142 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-477cbc79-9210-41fd-b6b9-11ce46d8f26d None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:07:51.738602 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-477cbc79-9210-41fd-b6b9-11ce46d8f26d None admin] RBAC: Authorization granted {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:07:51.750831 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 8/15] 10.222.0.27 () {56 vars in 1099 bytes} [Thu Oct 13 04:07:51 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 160 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 13 04:07:51.756313 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-477cbc79-9210-41fd-b6b9-11ce46d8f26d None admin] Authenticating user token {{(pid=81409) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:07:51.761087 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cdeb7419-512d-4579-98f1-babb0054d29d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81408) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:07:51.761701 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cdeb7419-512d-4579-98f1-babb0054d29d None admin] REQUEST_METHOD: `POST` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:07:51.761980 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cdeb7419-512d-4579-98f1-babb0054d29d None admin] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:07:51.762279 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cdeb7419-512d-4579-98f1-babb0054d29d None admin] PATH_INFO: `/v3/domains` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:07:51.762966 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cdeb7419-512d-4579-98f1-babb0054d29d None admin] RBAC: Authorizing `identity:create_domain()` {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:07:51.764357 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cdeb7419-512d-4579-98f1-babb0054d29d None admin] RBAC: Authorization granted {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:07:51.775416 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.sql.core [None req-cdeb7419-512d-4579-98f1-babb0054d29d None admin] Conflict project: (pymysql.err.IntegrityError) (1062, "Duplicate entry '<>-Default' for key 'project.ixu_project_name_domain_id'") Oct 13 04:07:51.775416 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [SQL: INSERT INTO project (id, name, domain_id, description, enabled, extra, parent_id, is_domain) VALUES (%(id)s, %(name)s, %(domain_id)s, %(description)s, %(enabled)s, %(extra)s, %(parent_id)s, %(is_domain)s)] Oct 13 04:07:51.775416 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [parameters: {'id': '4ce991dc72c247d789cb454a4f8b9375', 'name': 'Default', 'domain_id': '<>', 'description': '', 'enabled': 1, 'extra': '{}', 'parent_id': None, 'is_domain': 1}] Oct 13 04:07:51.775416 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: (Background on this error at: https://sqlalche.me/e/14/gkpj) {{(pid=81408) wrapper /opt/stack/keystone/keystone/common/sql/core.py:533}} Oct 13 04:07:51.793274 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application [None req-cdeb7419-512d-4579-98f1-babb0054d29d None admin] Conflict occurred attempting to store project - it is not permitted to have two projects acting as domains with the same name: Default.: keystone.exception.Conflict: Conflict occurred attempting to store project - it is not permitted to have two projects acting as domains with the same name: Default. Oct 13 04:07:51.793274 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 13 04:07:51.793274 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1900, in _execute_context Oct 13 04:07:51.793274 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application self.dialect.do_execute( Oct 13 04:07:51.793274 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/default.py", line 732, in do_execute Oct 13 04:07:51.793274 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application cursor.execute(statement, parameters) Oct 13 04:07:51.793274 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/cursors.py", line 148, in execute Oct 13 04:07:51.793274 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application result = self._query(query) Oct 13 04:07:51.793274 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/cursors.py", line 310, in _query Oct 13 04:07:51.793274 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application conn.query(q) Oct 13 04:07:51.793274 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 548, in query Oct 13 04:07:51.793274 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application self._affected_rows = self._read_query_result(unbuffered=unbuffered) Oct 13 04:07:51.793274 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 775, in _read_query_result Oct 13 04:07:51.793274 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application result.read() Oct 13 04:07:51.794132 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 1156, in read Oct 13 04:07:51.794132 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application first_packet = self.connection._read_packet() Oct 13 04:07:51.794132 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 725, in _read_packet Oct 13 04:07:51.794132 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application packet.raise_for_error() Oct 13 04:07:51.794132 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/protocol.py", line 221, in raise_for_error Oct 13 04:07:51.794132 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application err.raise_mysql_exception(self._data) Oct 13 04:07:51.794132 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/err.py", line 143, in raise_mysql_exception Oct 13 04:07:51.794132 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application raise errorclass(errno, errval) Oct 13 04:07:51.794132 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application pymysql.err.IntegrityError: (1062, "Duplicate entry '<>-Default' for key 'project.ixu_project_name_domain_id'") Oct 13 04:07:51.794132 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application Oct 13 04:07:51.794132 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application The above exception was the direct cause of the following exception: Oct 13 04:07:51.794132 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application Oct 13 04:07:51.794132 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 13 04:07:51.794132 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/sql/core.py", line 528, in wrapper Oct 13 04:07:51.794132 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application return method(*args, **kwargs) Oct 13 04:07:51.794132 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 241, in create_project Oct 13 04:07:51.794132 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application return project_ref.to_dict() Oct 13 04:07:51.794132 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ Oct 13 04:07:51.795070 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application next(self.gen) Oct 13 04:07:51.795070 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 1060, in _transaction_scope Oct 13 04:07:51.795070 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application yield resource Oct 13 04:07:51.795070 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ Oct 13 04:07:51.795070 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application next(self.gen) Oct 13 04:07:51.795070 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 662, in _session Oct 13 04:07:51.795070 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application self.session.rollback() Oct 13 04:07:51.795070 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_utils/excutils.py", line 227, in __exit__ Oct 13 04:07:51.795070 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application self.force_reraise() Oct 13 04:07:51.795070 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_utils/excutils.py", line 200, in force_reraise Oct 13 04:07:51.795070 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application raise self.value Oct 13 04:07:51.795070 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 659, in _session Oct 13 04:07:51.795070 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application self._end_session_transaction(self.session) Oct 13 04:07:51.795070 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 687, in _end_session_transaction Oct 13 04:07:51.795070 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application session.commit() Oct 13 04:07:51.795070 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 1451, in commit Oct 13 04:07:51.795070 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application self._transaction.commit(_to_root=self.future) Oct 13 04:07:51.795916 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 829, in commit Oct 13 04:07:51.795916 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application self._prepare_impl() Oct 13 04:07:51.795916 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 808, in _prepare_impl Oct 13 04:07:51.795916 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application self.session.flush() Oct 13 04:07:51.795916 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 3383, in flush Oct 13 04:07:51.795916 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application self._flush(objects) Oct 13 04:07:51.795916 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 3523, in _flush Oct 13 04:07:51.795916 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application transaction.rollback(_capture_exception=True) Oct 13 04:07:51.795916 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/util/langhelpers.py", line 70, in __exit__ Oct 13 04:07:51.795916 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application compat.raise_( Oct 13 04:07:51.795916 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/util/compat.py", line 208, in raise_ Oct 13 04:07:51.795916 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application raise exception Oct 13 04:07:51.795916 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 3483, in _flush Oct 13 04:07:51.795916 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application flush_context.execute() Oct 13 04:07:51.795916 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/unitofwork.py", line 456, in execute Oct 13 04:07:51.795916 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application rec.execute(self) Oct 13 04:07:51.795916 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/unitofwork.py", line 630, in execute Oct 13 04:07:51.796834 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application util.preloaded.orm_persistence.save_obj( Oct 13 04:07:51.796834 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/persistence.py", line 245, in save_obj Oct 13 04:07:51.796834 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application _emit_insert_statements( Oct 13 04:07:51.796834 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/persistence.py", line 1097, in _emit_insert_statements Oct 13 04:07:51.796834 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application c = connection._execute_20( Oct 13 04:07:51.796834 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1705, in _execute_20 Oct 13 04:07:51.796834 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application return meth(self, args_10style, kwargs_10style, execution_options) Oct 13 04:07:51.796834 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/sql/elements.py", line 333, in _execute_on_connection Oct 13 04:07:51.796834 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application return connection._execute_clauseelement( Oct 13 04:07:51.796834 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1572, in _execute_clauseelement Oct 13 04:07:51.796834 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application ret = self._execute_context( Oct 13 04:07:51.796834 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1943, in _execute_context Oct 13 04:07:51.796834 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application self._handle_dbapi_exception( Oct 13 04:07:51.796834 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 2122, in _handle_dbapi_exception Oct 13 04:07:51.796834 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application util.raise_(newraise, with_traceback=exc_info[2], from_=e) Oct 13 04:07:51.797569 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/util/compat.py", line 208, in raise_ Oct 13 04:07:51.797569 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application raise exception Oct 13 04:07:51.797569 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1900, in _execute_context Oct 13 04:07:51.797569 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application self.dialect.do_execute( Oct 13 04:07:51.797569 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/default.py", line 732, in do_execute Oct 13 04:07:51.797569 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application cursor.execute(statement, parameters) Oct 13 04:07:51.797569 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/cursors.py", line 148, in execute Oct 13 04:07:51.797569 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application result = self._query(query) Oct 13 04:07:51.797569 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/cursors.py", line 310, in _query Oct 13 04:07:51.797569 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application conn.query(q) Oct 13 04:07:51.797569 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 548, in query Oct 13 04:07:51.797569 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application self._affected_rows = self._read_query_result(unbuffered=unbuffered) Oct 13 04:07:51.797569 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 775, in _read_query_result Oct 13 04:07:51.797569 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application result.read() Oct 13 04:07:51.797569 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 1156, in read Oct 13 04:07:51.797569 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application first_packet = self.connection._read_packet() Oct 13 04:07:51.797569 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 725, in _read_packet Oct 13 04:07:51.799614 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application packet.raise_for_error() Oct 13 04:07:51.799614 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/protocol.py", line 221, in raise_for_error Oct 13 04:07:51.799614 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application err.raise_mysql_exception(self._data) Oct 13 04:07:51.799614 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/err.py", line 143, in raise_mysql_exception Oct 13 04:07:51.799614 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application raise errorclass(errno, errval) Oct 13 04:07:51.799614 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application oslo_db.exception.DBDuplicateEntry: (pymysql.err.IntegrityError) (1062, "Duplicate entry '<>-Default' for key 'project.ixu_project_name_domain_id'") Oct 13 04:07:51.799614 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application [SQL: INSERT INTO project (id, name, domain_id, description, enabled, extra, parent_id, is_domain) VALUES (%(id)s, %(name)s, %(domain_id)s, %(description)s, %(enabled)s, %(extra)s, %(parent_id)s, %(is_domain)s)] Oct 13 04:07:51.799614 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application [parameters: {'id': '4ce991dc72c247d789cb454a4f8b9375', 'name': 'Default', 'domain_id': '<>', 'description': '', 'enabled': 1, 'extra': '{}', 'parent_id': None, 'is_domain': 1}] Oct 13 04:07:51.799614 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application (Background on this error at: https://sqlalche.me/e/14/gkpj) Oct 13 04:07:51.799614 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application Oct 13 04:07:51.799614 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Oct 13 04:07:51.799614 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application Oct 13 04:07:51.799614 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 13 04:07:51.799614 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 216, in create_project Oct 13 04:07:51.799614 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application ret = self.driver.create_project(project_id, project) Oct 13 04:07:51.801540 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/sql/core.py", line 563, in wrapper Oct 13 04:07:51.801540 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application raise exception.Conflict(type=conflict_type, Oct 13 04:07:51.801540 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application keystone.exception.Conflict: Conflict occurred attempting to store project - Duplicate entry found with name Default. Oct 13 04:07:51.801540 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application Oct 13 04:07:51.801540 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Oct 13 04:07:51.801540 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application Oct 13 04:07:51.801540 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 13 04:07:51.801540 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 13 04:07:51.801540 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 13 04:07:51.801540 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 13 04:07:51.801540 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 13 04:07:51.801540 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 13 04:07:51.801540 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 13 04:07:51.801540 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 13 04:07:51.801540 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 13 04:07:51.801540 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 13 04:07:51.801540 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 13 04:07:51.803817 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 128, in post Oct 13 04:07:51.803817 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application ref = PROVIDERS.resource_api.create_domain( Oct 13 04:07:51.803817 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 13 04:07:51.803817 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 13 04:07:51.803817 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 765, in create_domain Oct 13 04:07:51.803817 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application is_domain_project = self.create_project( Oct 13 04:07:51.803817 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 13 04:07:51.803817 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 13 04:07:51.803817 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 218, in create_project Oct 13 04:07:51.803817 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application raise exception.Conflict( Oct 13 04:07:51.803817 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application keystone.exception.Conflict: Conflict occurred attempting to store project - it is not permitted to have two projects acting as domains with the same name: Default. Oct 13 04:07:51.803817 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application Oct 13 04:07:51.803817 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 8/16] 10.222.0.27 () {60 vars in 1137 bytes} [Thu Oct 13 04:07:51 2022] POST /identity/v3/domains => generated 190 bytes in 170 msecs (HTTP/1.1 409) 5 headers in 183 bytes (2 switches on core 0) Oct 13 04:07:51.803817 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cdeb7419-512d-4579-98f1-babb0054d29d None admin] Authenticating user token {{(pid=81408) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:07:51.857405 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-477cbc79-9210-41fd-b6b9-11ce46d8f26d None admin] Validating token access rules against request {{(pid=81409) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:07:51.891089 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-717aa262-e6af-4a8c-9e50-6a46c24b7ac9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81409) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:07:51.891865 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-717aa262-e6af-4a8c-9e50-6a46c24b7ac9 None admin] REQUEST_METHOD: `GET` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:07:51.892145 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-717aa262-e6af-4a8c-9e50-6a46c24b7ac9 None admin] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:07:51.892420 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-717aa262-e6af-4a8c-9e50-6a46c24b7ac9 None admin] PATH_INFO: `/v3/users/0e2d979563d6421995451a0e86bf3022` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:07:51.893908 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-717aa262-e6af-4a8c-9e50-6a46c24b7ac9 None admin] RBAC: Authorizing `identity:get_user(user_id=0e2d979563d6421995451a0e86bf3022)` {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:07:51.895399 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-717aa262-e6af-4a8c-9e50-6a46c24b7ac9 None admin] RBAC: Authorization granted {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:07:51.897144 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 9/17] 10.222.0.27 () {56 vars in 1177 bytes} [Thu Oct 13 04:07:51 2022] GET /identity/v3/users/0e2d979563d6421995451a0e86bf3022 => generated 247 bytes in 141 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 13 04:07:51.902792 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cdeb7419-512d-4579-98f1-babb0054d29d None admin] Validating token access rules against request {{(pid=81408) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:07:51.902927 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-717aa262-e6af-4a8c-9e50-6a46c24b7ac9 None admin] Authenticating user token {{(pid=81409) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:07:51.936771 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be13aa5d-a142-4bb8-bfcb-2b784b2bbda5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81408) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:07:51.937235 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG dbcounter [-] [81409] Writing DB stats keystone:SELECT=100 {{(pid=81409) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:07:51.937402 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be13aa5d-a142-4bb8-bfcb-2b784b2bbda5 None admin] REQUEST_METHOD: `GET` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:07:51.937682 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be13aa5d-a142-4bb8-bfcb-2b784b2bbda5 None admin] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:07:51.937955 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be13aa5d-a142-4bb8-bfcb-2b784b2bbda5 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:07:51.948761 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-be13aa5d-a142-4bb8-bfcb-2b784b2bbda5 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:07:51.950263 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-be13aa5d-a142-4bb8-bfcb-2b784b2bbda5 None admin] RBAC: Authorization granted {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:07:51.958965 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG dbcounter [-] [81408] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81408) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:07:51.968999 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application [None req-be13aa5d-a142-4bb8-bfcb-2b784b2bbda5 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 13 04:07:51.968999 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 13 04:07:51.968999 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 13 04:07:51.968999 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 13 04:07:51.968999 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 13 04:07:51.968999 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 13 04:07:51.968999 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 13 04:07:51.968999 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 13 04:07:51.968999 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 13 04:07:51.968999 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 13 04:07:51.968999 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 13 04:07:51.968999 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 13 04:07:51.968999 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 13 04:07:51.968999 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 13 04:07:51.968999 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 13 04:07:51.968999 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 13 04:07:51.970038 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 13 04:07:51.970038 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 13 04:07:51.970038 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 13 04:07:51.970038 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 13 04:07:51.970038 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 13 04:07:51.970038 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application return self.get_or_create( Oct 13 04:07:51.970038 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 13 04:07:51.970038 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application with Lock( Oct 13 04:07:51.970038 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 13 04:07:51.970038 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application return self._enter() Oct 13 04:07:51.970038 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 13 04:07:51.970038 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 13 04:07:51.970038 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 13 04:07:51.970038 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application return self.creator() Oct 13 04:07:51.970038 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 13 04:07:51.970038 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application created_value = creator( Oct 13 04:07:51.970038 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 13 04:07:51.970871 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 13 04:07:51.970871 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 13 04:07:51.970871 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application Oct 13 04:07:51.970871 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 9/18] 10.222.0.27 () {56 vars in 1108 bytes} [Thu Oct 13 04:07:51 2022] GET /identity/v3/domains/Default => generated 87 bytes in 170 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 13 04:07:51.976119 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be13aa5d-a142-4bb8-bfcb-2b784b2bbda5 None admin] Authenticating user token {{(pid=81408) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:07:52.006833 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-717aa262-e6af-4a8c-9e50-6a46c24b7ac9 None admin] Validating token access rules against request {{(pid=81409) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:07:52.040269 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b19027f3-e6ee-40d2-a4d7-e41e5d022267 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81409) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:07:52.040897 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b19027f3-e6ee-40d2-a4d7-e41e5d022267 None admin] REQUEST_METHOD: `GET` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:07:52.041175 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b19027f3-e6ee-40d2-a4d7-e41e5d022267 None admin] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:07:52.041442 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b19027f3-e6ee-40d2-a4d7-e41e5d022267 None admin] PATH_INFO: `/v3/domains/default` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:07:52.042582 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b19027f3-e6ee-40d2-a4d7-e41e5d022267 None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:07:52.044033 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b19027f3-e6ee-40d2-a4d7-e41e5d022267 None admin] RBAC: Authorization granted {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:07:52.045935 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 10/19] 10.222.0.27 () {56 vars in 1108 bytes} [Thu Oct 13 04:07:51 2022] GET /identity/v3/domains/default => generated 199 bytes in 143 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 13 04:07:52.051157 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b19027f3-e6ee-40d2-a4d7-e41e5d022267 None admin] Authenticating user token {{(pid=81409) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:07:52.076581 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be13aa5d-a142-4bb8-bfcb-2b784b2bbda5 None admin] Validating token access rules against request {{(pid=81408) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:07:52.110163 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3a12cf05-85b5-4fd0-885e-92133ccdaadc None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81408) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:07:52.110795 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a12cf05-85b5-4fd0-885e-92133ccdaadc None admin] REQUEST_METHOD: `GET` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:07:52.111092 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a12cf05-85b5-4fd0-885e-92133ccdaadc None admin] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:07:52.111367 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a12cf05-85b5-4fd0-885e-92133ccdaadc None admin] PATH_INFO: `/v3/domains` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:07:52.112007 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3a12cf05-85b5-4fd0-885e-92133ccdaadc None admin] RBAC: Adding query filter params (name=Default) {{(pid=81408) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 13 04:07:52.112309 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3a12cf05-85b5-4fd0-885e-92133ccdaadc None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:07:52.113709 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3a12cf05-85b5-4fd0-885e-92133ccdaadc None admin] RBAC: Authorization granted {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:07:52.130325 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 10/20] 10.222.0.27 () {56 vars in 1109 bytes} [Thu Oct 13 04:07:51 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 155 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 13 04:07:52.153937 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b19027f3-e6ee-40d2-a4d7-e41e5d022267 None admin] Validating token access rules against request {{(pid=81409) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:07:52.155615 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG dbcounter [-] [81409] Writing DB stats keystone:SELECT=100 {{(pid=81409) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:07:52.192425 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-738f0560-6d0c-41cb-b42f-e701ae395f16 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81409) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:07:52.193115 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-738f0560-6d0c-41cb-b42f-e701ae395f16 None admin] REQUEST_METHOD: `PUT` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:07:52.193394 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-738f0560-6d0c-41cb-b42f-e701ae395f16 None admin] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:07:52.193671 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-738f0560-6d0c-41cb-b42f-e701ae395f16 None admin] PATH_INFO: `/v3/domains/default/users/0e2d979563d6421995451a0e86bf3022/roles/ef55fbd94e1040be83695754075a7ed5` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:07:52.195715 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-738f0560-6d0c-41cb-b42f-e701ae395f16 None admin] RBAC: Authorizing `identity:create_grant(domain_id=default, user_id=0e2d979563d6421995451a0e86bf3022, role_id=ef55fbd94e1040be83695754075a7ed5)` {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:07:52.197232 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-738f0560-6d0c-41cb-b42f-e701ae395f16 None admin] RBAC: Authorization granted {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:07:52.317579 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 11/21] 10.222.0.27 () {58 vars in 1361 bytes} [Thu Oct 13 04:07:52 2022] PUT /identity/v3/domains/default/users/0e2d979563d6421995451a0e86bf3022/roles/ef55fbd94e1040be83695754075a7ed5 => generated 0 bytes in 267 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 13 04:07:54.753527 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ec2b54f-a978-48ec-8bae-ab31d9fa222c None None] REQUEST_METHOD: `GET` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:07:54.754040 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ec2b54f-a978-48ec-8bae-ab31d9fa222c None None] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:07:54.754112 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ec2b54f-a978-48ec-8bae-ab31d9fa222c None None] PATH_INFO: `/` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:07:54.755599 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 11/22] 10.222.0.27 () {54 vars in 927 bytes} [Thu Oct 13 04:07:54 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 13 04:07:54.772632 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7e0259f-e394-4f1d-929c-b73a76f39bb7 None None] REQUEST_METHOD: `POST` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:07:54.773128 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7e0259f-e394-4f1d-929c-b73a76f39bb7 None None] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:07:54.773199 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7e0259f-e394-4f1d-929c-b73a76f39bb7 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:07:54.803189 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.auth.core [None req-c7e0259f-e394-4f1d-929c-b73a76f39bb7 None None] MFA Rules not processed for user `0e2d979563d6421995451a0e86bf3022`. Rule list: `[]` (Enabled: `True`). {{(pid=81409) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 13 04:07:54.804768 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.fernet_utils [None req-c7e0259f-e394-4f1d-929c-b73a76f39bb7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81409) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:07:54.908086 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 12/23] 10.222.0.27 () {58 vars in 1024 bytes} [Thu Oct 13 04:07:54 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 137 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 13 04:07:54.917718 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f4dc428-da79-4be1-a288-27239597720d None None] REQUEST_METHOD: `GET` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:07:54.918003 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f4dc428-da79-4be1-a288-27239597720d None None] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:07:54.918318 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f4dc428-da79-4be1-a288-27239597720d None None] PATH_INFO: `/` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:07:54.919645 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 12/24] 10.222.0.27 () {54 vars in 927 bytes} [Thu Oct 13 04:07:54 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 13 04:07:54.927945 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-558b3e58-159a-494a-861b-c77853042b61 None None] REQUEST_METHOD: `POST` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:07:54.928239 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-558b3e58-159a-494a-861b-c77853042b61 None None] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:07:54.928518 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-558b3e58-159a-494a-861b-c77853042b61 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:07:54.955523 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.auth.core [None req-558b3e58-159a-494a-861b-c77853042b61 None None] MFA Rules not processed for user `0e2d979563d6421995451a0e86bf3022`. Rule list: `[]` (Enabled: `True`). {{(pid=81409) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 13 04:07:54.956775 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.fernet_utils [None req-558b3e58-159a-494a-861b-c77853042b61 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81409) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:07:55.059779 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 13/25] 10.222.0.27 () {58 vars in 1024 bytes} [Thu Oct 13 04:07:54 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 132 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 13 04:07:55.107062 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39d7335d-74eb-4638-964b-5a6f8e7a6491 None None] REQUEST_METHOD: `GET` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:07:55.107360 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39d7335d-74eb-4638-964b-5a6f8e7a6491 None None] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:07:55.107617 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39d7335d-74eb-4638-964b-5a6f8e7a6491 None None] PATH_INFO: `/` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:07:55.109124 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 13/26] 10.222.0.27 () {54 vars in 927 bytes} [Thu Oct 13 04:07:55 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 13 04:07:55.114745 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b042bb2-3bf6-42e8-801e-ae94a37d81f9 None None] REQUEST_METHOD: `POST` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:07:55.115029 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b042bb2-3bf6-42e8-801e-ae94a37d81f9 None None] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:07:55.115305 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b042bb2-3bf6-42e8-801e-ae94a37d81f9 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:07:55.147214 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.auth.core [None req-9b042bb2-3bf6-42e8-801e-ae94a37d81f9 None None] MFA Rules not processed for user `0e2d979563d6421995451a0e86bf3022`. Rule list: `[]` (Enabled: `True`). {{(pid=81409) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 13 04:07:55.148687 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.fernet_utils [None req-9b042bb2-3bf6-42e8-801e-ae94a37d81f9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81409) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:07:55.175257 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG dbcounter [-] [81409] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81409) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:07:55.239322 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-39d7335d-74eb-4638-964b-5a6f8e7a6491 None None] Authenticating user token {{(pid=81408) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:07:55.254732 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 14/27] 10.222.0.27 () {58 vars in 1024 bytes} [Thu Oct 13 04:07:55 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 141 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 13 04:07:55.305242 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG dbcounter [-] [81408] Writing DB stats keystone:SELECT=100 {{(pid=81408) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:07:55.351230 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-39d7335d-74eb-4638-964b-5a6f8e7a6491 None None] Validating token access rules against request {{(pid=81408) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:07:55.394283 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7021a613-8675-4647-9ab6-280b46c58c70 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81408) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:07:55.394965 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7021a613-8675-4647-9ab6-280b46c58c70 None admin] REQUEST_METHOD: `GET` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:07:55.395252 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7021a613-8675-4647-9ab6-280b46c58c70 None admin] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:07:55.395521 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7021a613-8675-4647-9ab6-280b46c58c70 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:07:55.412866 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7021a613-8675-4647-9ab6-280b46c58c70 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:07:55.413045 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7021a613-8675-4647-9ab6-280b46c58c70 None admin] RBAC: Authorization granted {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:07:55.439953 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application [None req-7021a613-8675-4647-9ab6-280b46c58c70 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 13 04:07:55.439953 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 13 04:07:55.439953 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 13 04:07:55.439953 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 13 04:07:55.439953 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 13 04:07:55.439953 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 13 04:07:55.439953 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 13 04:07:55.439953 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 13 04:07:55.439953 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 13 04:07:55.439953 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 13 04:07:55.439953 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 13 04:07:55.439953 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 13 04:07:55.439953 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 13 04:07:55.439953 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 13 04:07:55.439953 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 13 04:07:55.439953 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 13 04:07:55.440857 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 13 04:07:55.440857 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 13 04:07:55.440857 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 13 04:07:55.440857 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 13 04:07:55.440857 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 13 04:07:55.440857 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application return self.get_or_create( Oct 13 04:07:55.440857 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 13 04:07:55.440857 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application with Lock( Oct 13 04:07:55.440857 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 13 04:07:55.440857 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application return self._enter() Oct 13 04:07:55.440857 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 13 04:07:55.440857 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 13 04:07:55.440857 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 13 04:07:55.440857 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application return self.creator() Oct 13 04:07:55.440857 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 13 04:07:55.440857 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application created_value = creator( Oct 13 04:07:55.440857 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 13 04:07:55.441661 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 13 04:07:55.441661 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 13 04:07:55.441661 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application Oct 13 04:07:55.441661 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 14/28] 10.222.0.27 () {56 vars in 1108 bytes} [Thu Oct 13 04:07:55 2022] GET /identity/v3/domains/Default => generated 87 bytes in 202 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Oct 13 04:07:55.444939 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9b042bb2-3bf6-42e8-801e-ae94a37d81f9 None None] Authenticating user token {{(pid=81409) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:07:55.456660 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7021a613-8675-4647-9ab6-280b46c58c70 None admin] Authenticating user token {{(pid=81408) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:07:55.568749 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9b042bb2-3bf6-42e8-801e-ae94a37d81f9 None None] Validating token access rules against request {{(pid=81409) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:07:55.606455 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6f1ca534-af78-4013-aed5-241cede8f1c0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81409) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:07:55.607326 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f1ca534-af78-4013-aed5-241cede8f1c0 None admin] REQUEST_METHOD: `POST` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:07:55.607446 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f1ca534-af78-4013-aed5-241cede8f1c0 None admin] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:07:55.607647 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f1ca534-af78-4013-aed5-241cede8f1c0 None admin] PATH_INFO: `/v3/roles` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:07:55.608319 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6f1ca534-af78-4013-aed5-241cede8f1c0 None admin] RBAC: Authorizing `identity:create_role()` {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:07:55.609985 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6f1ca534-af78-4013-aed5-241cede8f1c0 None admin] RBAC: Authorization granted {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:07:55.632540 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 15/29] 10.222.0.27 () {60 vars in 1131 bytes} [Thu Oct 13 04:07:55 2022] POST /identity/v3/roles => generated 219 bytes in 188 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 13 04:07:55.633921 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6f1ca534-af78-4013-aed5-241cede8f1c0 None admin] Authenticating user token {{(pid=81409) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:07:55.665806 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG dbcounter [-] [81409] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81409) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:07:55.706922 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7021a613-8675-4647-9ab6-280b46c58c70 None admin] Validating token access rules against request {{(pid=81408) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:07:55.730901 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG dbcounter [-] [81408] Writing DB stats keystone:SELECT=100 {{(pid=81408) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:07:55.787691 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aca5a944-3020-4e31-981d-55cfd5f59493 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81408) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:07:55.788352 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aca5a944-3020-4e31-981d-55cfd5f59493 None admin] REQUEST_METHOD: `GET` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:07:55.788595 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aca5a944-3020-4e31-981d-55cfd5f59493 None admin] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:07:55.788860 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aca5a944-3020-4e31-981d-55cfd5f59493 None admin] PATH_INFO: `/v3/domains` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:07:55.789594 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aca5a944-3020-4e31-981d-55cfd5f59493 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81408) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 13 04:07:55.789992 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aca5a944-3020-4e31-981d-55cfd5f59493 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:07:55.791565 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aca5a944-3020-4e31-981d-55cfd5f59493 None admin] RBAC: Authorization granted {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:07:55.823704 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 15/30] 10.222.0.27 () {56 vars in 1109 bytes} [Thu Oct 13 04:07:55 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 367 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 13 04:07:55.825005 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af98ac8d-d89e-4d28-adad-31aa317167d3 None None] REQUEST_METHOD: `GET` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:07:55.825292 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af98ac8d-d89e-4d28-adad-31aa317167d3 None None] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:07:55.825564 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af98ac8d-d89e-4d28-adad-31aa317167d3 None None] PATH_INFO: `/` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:07:55.827210 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 16/31] 10.222.0.27 () {54 vars in 927 bytes} [Thu Oct 13 04:07:55 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 13 04:07:55.833595 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-af98ac8d-d89e-4d28-adad-31aa317167d3 None None] Authenticating user token {{(pid=81408) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:07:55.858942 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6f1ca534-af78-4013-aed5-241cede8f1c0 None admin] Validating token access rules against request {{(pid=81409) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:07:55.904763 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-336a2291-7650-42e7-95f1-44178f6b4442 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81409) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:07:55.905699 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-336a2291-7650-42e7-95f1-44178f6b4442 None admin] REQUEST_METHOD: `GET` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:07:55.906169 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-336a2291-7650-42e7-95f1-44178f6b4442 None admin] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:07:55.906663 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-336a2291-7650-42e7-95f1-44178f6b4442 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:07:55.922461 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-336a2291-7650-42e7-95f1-44178f6b4442 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 13 04:07:55.935698 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application [None req-336a2291-7650-42e7-95f1-44178f6b4442 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Oct 13 04:07:55.935698 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 13 04:07:55.935698 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 13 04:07:55.935698 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 13 04:07:55.935698 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 13 04:07:55.935698 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 13 04:07:55.935698 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 13 04:07:55.935698 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 13 04:07:55.935698 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 13 04:07:55.935698 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 13 04:07:55.935698 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 13 04:07:55.935698 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 13 04:07:55.935698 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 13 04:07:55.935698 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 13 04:07:55.935698 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 13 04:07:55.935698 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 13 04:07:55.936568 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 13 04:07:55.936568 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 13 04:07:55.936568 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 13 04:07:55.936568 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 13 04:07:55.936568 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 13 04:07:55.936568 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 13 04:07:55.936568 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 13 04:07:55.936568 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 13 04:07:55.936568 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 13 04:07:55.936568 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application return self.get_or_create( Oct 13 04:07:55.936568 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 13 04:07:55.936568 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application with Lock( Oct 13 04:07:55.936568 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 13 04:07:55.936568 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application return self._enter() Oct 13 04:07:55.936568 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 13 04:07:55.936568 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 13 04:07:55.937287 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 13 04:07:55.937287 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application return self.creator() Oct 13 04:07:55.937287 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 13 04:07:55.937287 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application created_value = creator( Oct 13 04:07:55.937287 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 13 04:07:55.937287 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 13 04:07:55.937287 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 13 04:07:55.937287 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 13 04:07:55.937287 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 13 04:07:55.937287 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 13 04:07:55.937287 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Oct 13 04:07:55.937287 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application Oct 13 04:07:55.937287 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 16/32] 10.222.0.27 () {56 vars in 1096 bytes} [Thu Oct 13 04:07:55 2022] GET /identity/v3/roles/admin => generated 83 bytes in 303 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 13 04:07:55.938400 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-daae8e14-cc35-4b5a-9e9f-64d05fcdd30e None None] REQUEST_METHOD: `POST` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:07:55.938664 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-daae8e14-cc35-4b5a-9e9f-64d05fcdd30e None None] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:07:55.938994 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-daae8e14-cc35-4b5a-9e9f-64d05fcdd30e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:07:55.959994 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.auth.core [None req-daae8e14-cc35-4b5a-9e9f-64d05fcdd30e None None] MFA Rules not processed for user `0e2d979563d6421995451a0e86bf3022`. Rule list: `[]` (Enabled: `True`). {{(pid=81409) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 13 04:07:55.961909 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.fernet_utils [None req-daae8e14-cc35-4b5a-9e9f-64d05fcdd30e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81409) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:07:55.981354 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-af98ac8d-d89e-4d28-adad-31aa317167d3 None None] Validating token access rules against request {{(pid=81408) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:07:56.022132 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7f2e44a0-0a55-44d5-86a8-95406cfe6211 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81408) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:07:56.023114 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f2e44a0-0a55-44d5-86a8-95406cfe6211 None admin] REQUEST_METHOD: `POST` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:07:56.023639 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f2e44a0-0a55-44d5-86a8-95406cfe6211 None admin] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:07:56.024127 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f2e44a0-0a55-44d5-86a8-95406cfe6211 None admin] PATH_INFO: `/v3/projects` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:07:56.025436 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7f2e44a0-0a55-44d5-86a8-95406cfe6211 None admin] RBAC: Authorizing `identity:create_project()` {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:07:56.027229 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7f2e44a0-0a55-44d5-86a8-95406cfe6211 None admin] RBAC: Authorization granted {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:07:56.045162 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 17/33] 10.222.0.27 () {58 vars in 1024 bytes} [Thu Oct 13 04:07:55 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 107 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 13 04:07:56.051064 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-daae8e14-cc35-4b5a-9e9f-64d05fcdd30e None None] Authenticating user token {{(pid=81409) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:07:56.056626 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG dbcounter [-] [81409] Writing DB stats keystone:SELECT=100 {{(pid=81409) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:07:56.183765 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 17/34] 10.222.0.27 () {60 vars in 1141 bytes} [Thu Oct 13 04:07:55 2022] POST /identity/v3/projects => generated 301 bytes in 350 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 13 04:07:56.241881 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-daae8e14-cc35-4b5a-9e9f-64d05fcdd30e None None] Validating token access rules against request {{(pid=81409) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:07:56.282078 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-046097fa-115f-455e-8628-67cbd7bc23cc None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81409) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:07:56.283145 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-046097fa-115f-455e-8628-67cbd7bc23cc None admin] REQUEST_METHOD: `GET` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:07:56.283638 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-046097fa-115f-455e-8628-67cbd7bc23cc None admin] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:07:56.284098 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-046097fa-115f-455e-8628-67cbd7bc23cc None admin] PATH_INFO: `/v3/roles` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:07:56.284975 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-046097fa-115f-455e-8628-67cbd7bc23cc None admin] RBAC: Adding query filter params (name=admin) {{(pid=81409) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 13 04:07:56.285476 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-046097fa-115f-455e-8628-67cbd7bc23cc None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:07:56.287149 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-046097fa-115f-455e-8628-67cbd7bc23cc None admin] RBAC: Authorization granted {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:07:56.297880 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 18/35] 10.222.0.27 () {56 vars in 1099 bytes} [Thu Oct 13 04:07:56 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 252 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 13 04:07:56.303815 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7f2e44a0-0a55-44d5-86a8-95406cfe6211 None admin] Authenticating user token {{(pid=81408) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:07:56.315637 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47e5575a-648a-473f-abf0-e8a2c5ef1715 None None] REQUEST_METHOD: `GET` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:07:56.319449 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47e5575a-648a-473f-abf0-e8a2c5ef1715 None None] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:07:56.320011 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47e5575a-648a-473f-abf0-e8a2c5ef1715 None None] PATH_INFO: `/` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:07:56.324216 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 19/36] 10.222.0.27 () {54 vars in 927 bytes} [Thu Oct 13 04:07:56 2022] GET /identity => generated 270 bytes in 18 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Oct 13 04:07:56.329829 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a64d498c-1eb2-4784-abcb-f061a38cf15a None None] REQUEST_METHOD: `POST` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:07:56.330449 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a64d498c-1eb2-4784-abcb-f061a38cf15a None None] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:07:56.330915 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a64d498c-1eb2-4784-abcb-f061a38cf15a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:07:56.359094 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG dbcounter [-] [81408] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81408) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:07:56.371821 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.auth.core [None req-a64d498c-1eb2-4784-abcb-f061a38cf15a None None] MFA Rules not processed for user `0e2d979563d6421995451a0e86bf3022`. Rule list: `[]` (Enabled: `True`). {{(pid=81409) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 13 04:07:56.373492 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.fernet_utils [None req-a64d498c-1eb2-4784-abcb-f061a38cf15a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81409) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:07:56.457403 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG dbcounter [-] [81409] Writing DB stats keystone:SELECT=100 {{(pid=81409) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:07:56.471599 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 20/37] 10.222.0.27 () {58 vars in 1024 bytes} [Thu Oct 13 04:07:56 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 143 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 13 04:07:56.473202 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a64d498c-1eb2-4784-abcb-f061a38cf15a None None] Authenticating user token {{(pid=81409) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:07:56.493308 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7f2e44a0-0a55-44d5-86a8-95406cfe6211 None admin] Validating token access rules against request {{(pid=81408) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:07:56.531502 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-259850d5-8203-4dca-956e-6a5f75838cee None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81408) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:07:56.532154 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-259850d5-8203-4dca-956e-6a5f75838cee None admin] REQUEST_METHOD: `GET` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:07:56.532431 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-259850d5-8203-4dca-956e-6a5f75838cee None admin] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:07:56.532707 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-259850d5-8203-4dca-956e-6a5f75838cee None admin] PATH_INFO: `/v3/users/0e2d979563d6421995451a0e86bf3022` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:07:56.533943 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-259850d5-8203-4dca-956e-6a5f75838cee None admin] RBAC: Authorizing `identity:get_user(user_id=0e2d979563d6421995451a0e86bf3022)` {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:07:56.535615 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-259850d5-8203-4dca-956e-6a5f75838cee None admin] RBAC: Authorization granted {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:07:56.537409 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 18/38] 10.222.0.27 () {56 vars in 1177 bytes} [Thu Oct 13 04:07:56 2022] GET /identity/v3/users/0e2d979563d6421995451a0e86bf3022 => generated 247 bytes in 234 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 13 04:07:56.544307 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-259850d5-8203-4dca-956e-6a5f75838cee None admin] Authenticating user token {{(pid=81408) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:07:56.592438 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a64d498c-1eb2-4784-abcb-f061a38cf15a None None] Validating token access rules against request {{(pid=81409) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:07:56.641066 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9c1962e3-c154-45a9-ac0f-96f1a146197c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81409) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:07:56.642290 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c1962e3-c154-45a9-ac0f-96f1a146197c None admin] REQUEST_METHOD: `POST` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:07:56.642775 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c1962e3-c154-45a9-ac0f-96f1a146197c None admin] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:07:56.643301 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c1962e3-c154-45a9-ac0f-96f1a146197c None admin] PATH_INFO: `/v3/roles` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:07:56.644244 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9c1962e3-c154-45a9-ac0f-96f1a146197c None admin] RBAC: Authorizing `identity:create_role()` {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:07:56.645936 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9c1962e3-c154-45a9-ac0f-96f1a146197c None admin] RBAC: Authorization granted {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:07:56.655927 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG dbcounter [-] [81408] Writing DB stats keystone:SELECT=100 {{(pid=81408) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:07:56.669453 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 21/39] 10.222.0.27 () {60 vars in 1131 bytes} [Thu Oct 13 04:07:56 2022] POST /identity/v3/roles => generated 225 bytes in 197 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 13 04:07:56.669927 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-259850d5-8203-4dca-956e-6a5f75838cee None admin] Validating token access rules against request {{(pid=81408) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:07:56.715652 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-11d9ed5a-7874-4cc0-a123-8874be2f958b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81408) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:07:56.716270 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11d9ed5a-7874-4cc0-a123-8874be2f958b None admin] REQUEST_METHOD: `GET` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:07:56.716592 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11d9ed5a-7874-4cc0-a123-8874be2f958b None admin] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:07:56.716920 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11d9ed5a-7874-4cc0-a123-8874be2f958b None admin] PATH_INFO: `/v3/domains/default` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:07:56.718319 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-11d9ed5a-7874-4cc0-a123-8874be2f958b None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:07:56.719915 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-11d9ed5a-7874-4cc0-a123-8874be2f958b None admin] RBAC: Authorization granted {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:07:56.721988 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 19/40] 10.222.0.27 () {56 vars in 1108 bytes} [Thu Oct 13 04:07:56 2022] GET /identity/v3/domains/default => generated 199 bytes in 178 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 13 04:07:56.727643 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0d9effc-26da-45d3-847d-80e53f335e39 None None] REQUEST_METHOD: `GET` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:07:56.727983 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0d9effc-26da-45d3-847d-80e53f335e39 None None] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:07:56.728261 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0d9effc-26da-45d3-847d-80e53f335e39 None None] PATH_INFO: `/` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:07:56.728533 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-11d9ed5a-7874-4cc0-a123-8874be2f958b None admin] Authenticating user token {{(pid=81408) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:07:56.729631 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 22/41] 10.222.0.27 () {54 vars in 927 bytes} [Thu Oct 13 04:07:56 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 13 04:07:56.735158 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25396b25-e476-4b08-915b-7cb80b9ee6d3 None None] REQUEST_METHOD: `POST` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:07:56.735438 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25396b25-e476-4b08-915b-7cb80b9ee6d3 None None] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:07:56.735711 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25396b25-e476-4b08-915b-7cb80b9ee6d3 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:07:56.757633 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.auth.core [None req-25396b25-e476-4b08-915b-7cb80b9ee6d3 None None] MFA Rules not processed for user `0e2d979563d6421995451a0e86bf3022`. Rule list: `[]` (Enabled: `True`). {{(pid=81409) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 13 04:07:56.759579 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.fernet_utils [None req-25396b25-e476-4b08-915b-7cb80b9ee6d3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81409) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:07:56.819279 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG dbcounter [-] [81409] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81409) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:07:56.840358 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 23/42] 10.222.0.27 () {58 vars in 1024 bytes} [Thu Oct 13 04:07:56 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 106 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Oct 13 04:07:56.841239 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-25396b25-e476-4b08-915b-7cb80b9ee6d3 None None] Authenticating user token {{(pid=81409) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:07:56.859135 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-11d9ed5a-7874-4cc0-a123-8874be2f958b None admin] Validating token access rules against request {{(pid=81408) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:07:56.903075 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-564a4ee4-f80d-42c0-842e-dd6dcf6b4f57 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81408) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:07:56.903697 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-564a4ee4-f80d-42c0-842e-dd6dcf6b4f57 None admin] REQUEST_METHOD: `GET` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:07:56.904253 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-564a4ee4-f80d-42c0-842e-dd6dcf6b4f57 None admin] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:07:56.904534 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-564a4ee4-f80d-42c0-842e-dd6dcf6b4f57 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:07:56.905271 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-564a4ee4-f80d-42c0-842e-dd6dcf6b4f57 None admin] RBAC: Adding query filter params (role.id=ef55fbd94e1040be83695754075a7ed5, scope.domain.id=default, user.id=0e2d979563d6421995451a0e86bf3022) {{(pid=81408) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 13 04:07:56.905578 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-564a4ee4-f80d-42c0-842e-dd6dcf6b4f57 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:07:56.907520 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-564a4ee4-f80d-42c0-842e-dd6dcf6b4f57 None admin] RBAC: Authorization granted {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:07:56.917507 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 20/43] 10.222.0.27 () {56 vars in 1322 bytes} [Thu Oct 13 04:07:56 2022] GET /identity/v3/role_assignments?user.id=0e2d979563d6421995451a0e86bf3022&scope.domain.id=default&role.id=ef55fbd94e1040be83695754075a7ed5 => generated 530 bytes in 190 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 13 04:07:56.965877 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-25396b25-e476-4b08-915b-7cb80b9ee6d3 None None] Validating token access rules against request {{(pid=81409) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:07:57.005182 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-13eb3f76-ec39-4cdb-9a05-4592a087564d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81409) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:07:57.005834 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13eb3f76-ec39-4cdb-9a05-4592a087564d None admin] REQUEST_METHOD: `GET` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:07:57.006114 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13eb3f76-ec39-4cdb-9a05-4592a087564d None admin] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:07:57.006428 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13eb3f76-ec39-4cdb-9a05-4592a087564d None admin] PATH_INFO: `/v3/domains/default` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:07:57.007605 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-13eb3f76-ec39-4cdb-9a05-4592a087564d None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:07:57.009625 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-13eb3f76-ec39-4cdb-9a05-4592a087564d None admin] RBAC: Authorization granted {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:07:57.014259 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 24/44] 10.222.0.27 () {56 vars in 1108 bytes} [Thu Oct 13 04:07:56 2022] GET /identity/v3/domains/default => generated 199 bytes in 171 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 13 04:07:57.017268 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-564a4ee4-f80d-42c0-842e-dd6dcf6b4f57 None admin] Authenticating user token {{(pid=81408) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:07:57.045307 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG dbcounter [-] [81408] Writing DB stats keystone:SELECT=100 {{(pid=81408) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:07:57.109823 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-13eb3f76-ec39-4cdb-9a05-4592a087564d None admin] Authenticating user token {{(pid=81409) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:07:57.134851 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-564a4ee4-f80d-42c0-842e-dd6dcf6b4f57 None admin] Validating token access rules against request {{(pid=81408) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:07:57.170164 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7daa7840-779c-45ac-8bc0-3102e572c6ca None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81408) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:07:57.170801 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7daa7840-779c-45ac-8bc0-3102e572c6ca None admin] REQUEST_METHOD: `POST` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:07:57.171079 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7daa7840-779c-45ac-8bc0-3102e572c6ca None admin] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:07:57.171348 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7daa7840-779c-45ac-8bc0-3102e572c6ca None admin] PATH_INFO: `/v3/projects` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:07:57.171998 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7daa7840-779c-45ac-8bc0-3102e572c6ca None admin] RBAC: Authorizing `identity:create_project()` {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:07:57.172714 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG dbcounter [-] [81409] Writing DB stats keystone:SELECT=100 {{(pid=81409) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:07:57.173627 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7daa7840-779c-45ac-8bc0-3102e572c6ca None admin] RBAC: Authorization granted {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:07:57.223646 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-13eb3f76-ec39-4cdb-9a05-4592a087564d None admin] Validating token access rules against request {{(pid=81409) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:07:57.261051 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1f87f8f6-ff93-42ee-b59b-99ed3dc0501c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81409) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:07:57.261670 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f87f8f6-ff93-42ee-b59b-99ed3dc0501c None admin] REQUEST_METHOD: `POST` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:07:57.261954 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f87f8f6-ff93-42ee-b59b-99ed3dc0501c None admin] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:07:57.262292 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f87f8f6-ff93-42ee-b59b-99ed3dc0501c None admin] PATH_INFO: `/v3/roles` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:07:57.262956 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1f87f8f6-ff93-42ee-b59b-99ed3dc0501c None admin] RBAC: Authorizing `identity:create_role()` {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:07:57.264364 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1f87f8f6-ff93-42ee-b59b-99ed3dc0501c None admin] RBAC: Authorization granted {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:07:57.280421 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 25/45] 10.222.0.27 () {60 vars in 1131 bytes} [Thu Oct 13 04:07:57 2022] POST /identity/v3/roles => generated 223 bytes in 171 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 13 04:07:57.310823 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 21/46] 10.222.0.27 () {60 vars in 1141 bytes} [Thu Oct 13 04:07:57 2022] POST /identity/v3/projects => generated 312 bytes in 294 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 13 04:07:59.466505 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a2e3844-e42e-4f44-9d0a-d388ca13aac5 None None] REQUEST_METHOD: `GET` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:07:59.466974 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a2e3844-e42e-4f44-9d0a-d388ca13aac5 None None] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:07:59.467052 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a2e3844-e42e-4f44-9d0a-d388ca13aac5 None None] PATH_INFO: `/` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:07:59.468428 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 26/47] 10.222.0.27 () {54 vars in 927 bytes} [Thu Oct 13 04:07:59 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 13 04:07:59.474195 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12f5c00b-a1f4-49c2-a6cd-51d47d3f12c4 None None] REQUEST_METHOD: `POST` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:07:59.474634 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12f5c00b-a1f4-49c2-a6cd-51d47d3f12c4 None None] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:07:59.474754 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12f5c00b-a1f4-49c2-a6cd-51d47d3f12c4 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:07:59.493464 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.auth.core [None req-12f5c00b-a1f4-49c2-a6cd-51d47d3f12c4 None None] MFA Rules not processed for user `0e2d979563d6421995451a0e86bf3022`. Rule list: `[]` (Enabled: `True`). {{(pid=81408) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 13 04:07:59.495044 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.fernet_utils [None req-12f5c00b-a1f4-49c2-a6cd-51d47d3f12c4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81408) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:07:59.558612 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 22/48] 10.222.0.27 () {58 vars in 1024 bytes} [Thu Oct 13 04:07:59 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 86 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Oct 13 04:07:59.826140 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9a2e3844-e42e-4f44-9d0a-d388ca13aac5 None None] Authenticating user token {{(pid=81409) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:07:59.925198 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9a2e3844-e42e-4f44-9d0a-d388ca13aac5 None None] Validating token access rules against request {{(pid=81409) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:07:59.937523 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG dbcounter [-] [81409] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81409) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:07:59.959103 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0590936c-36f0-4d9c-ba14-ca09a1ab97e5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81409) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:07:59.959725 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0590936c-36f0-4d9c-ba14-ca09a1ab97e5 None admin] REQUEST_METHOD: `GET` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:07:59.960006 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0590936c-36f0-4d9c-ba14-ca09a1ab97e5 None admin] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:07:59.960281 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0590936c-36f0-4d9c-ba14-ca09a1ab97e5 None admin] PATH_INFO: `/v3/domains/default` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:07:59.961430 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0590936c-36f0-4d9c-ba14-ca09a1ab97e5 None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:07:59.963408 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0590936c-36f0-4d9c-ba14-ca09a1ab97e5 None admin] RBAC: Authorization granted {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:07:59.964960 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 27/49] 10.222.0.27 () {56 vars in 1108 bytes} [Thu Oct 13 04:07:59 2022] GET /identity/v3/domains/default => generated 199 bytes in 139 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 13 04:07:59.970524 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-12f5c00b-a1f4-49c2-a6cd-51d47d3f12c4 None None] Authenticating user token {{(pid=81408) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:07:59.973859 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG dbcounter [-] [81408] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81408) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:08:00.069507 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-12f5c00b-a1f4-49c2-a6cd-51d47d3f12c4 None None] Validating token access rules against request {{(pid=81408) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:08:00.103543 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f6012e73-272d-4457-a278-0dd031cce1f7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81408) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:08:00.104088 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6012e73-272d-4457-a278-0dd031cce1f7 None admin] REQUEST_METHOD: `POST` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:08:00.104374 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6012e73-272d-4457-a278-0dd031cce1f7 None admin] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:08:00.104653 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6012e73-272d-4457-a278-0dd031cce1f7 None admin] PATH_INFO: `/v3/projects` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:08:00.105302 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f6012e73-272d-4457-a278-0dd031cce1f7 None admin] RBAC: Authorizing `identity:create_project()` {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:08:00.106822 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f6012e73-272d-4457-a278-0dd031cce1f7 None admin] RBAC: Authorization granted {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:08:00.132305 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 23/50] 10.222.0.27 () {60 vars in 1140 bytes} [Thu Oct 13 04:07:59 2022] POST /identity/v3/projects => generated 298 bytes in 162 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 13 04:08:02.278299 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ddeba9bf-9dad-45d7-a8e8-429265803758 None None] REQUEST_METHOD: `GET` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:08:02.278807 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ddeba9bf-9dad-45d7-a8e8-429265803758 None None] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:08:02.278888 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ddeba9bf-9dad-45d7-a8e8-429265803758 None None] PATH_INFO: `/` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:08:02.280208 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 28/51] 10.222.0.27 () {54 vars in 927 bytes} [Thu Oct 13 04:08:02 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 13 04:08:02.285342 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12902a62-429b-42b7-a4a5-c6d7401785cf None None] REQUEST_METHOD: `POST` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:08:02.285701 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12902a62-429b-42b7-a4a5-c6d7401785cf None None] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:08:02.285900 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12902a62-429b-42b7-a4a5-c6d7401785cf None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:08:02.304522 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.auth.core [None req-12902a62-429b-42b7-a4a5-c6d7401785cf None None] MFA Rules not processed for user `0e2d979563d6421995451a0e86bf3022`. Rule list: `[]` (Enabled: `True`). {{(pid=81408) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 13 04:08:02.306499 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.fernet_utils [None req-12902a62-429b-42b7-a4a5-c6d7401785cf None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81408) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:08:02.355148 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG dbcounter [-] [81408] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81408) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:08:02.372471 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 24/52] 10.222.0.27 () {58 vars in 1024 bytes} [Thu Oct 13 04:08:02 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 88 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 13 04:08:02.642192 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ddeba9bf-9dad-45d7-a8e8-429265803758 None None] Authenticating user token {{(pid=81409) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:08:02.741662 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ddeba9bf-9dad-45d7-a8e8-429265803758 None None] Validating token access rules against request {{(pid=81409) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:08:02.773891 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6280b1e0-a523-42ca-9e59-8eb9b2bfa45c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81409) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:08:02.774592 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6280b1e0-a523-42ca-9e59-8eb9b2bfa45c None admin] REQUEST_METHOD: `GET` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:08:02.774872 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6280b1e0-a523-42ca-9e59-8eb9b2bfa45c None admin] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:08:02.775148 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6280b1e0-a523-42ca-9e59-8eb9b2bfa45c None admin] PATH_INFO: `/v3/domains/default` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:08:02.776262 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6280b1e0-a523-42ca-9e59-8eb9b2bfa45c None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:08:02.777709 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6280b1e0-a523-42ca-9e59-8eb9b2bfa45c None admin] RBAC: Authorization granted {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:08:02.779277 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 29/53] 10.222.0.27 () {56 vars in 1108 bytes} [Thu Oct 13 04:08:02 2022] GET /identity/v3/domains/default => generated 199 bytes in 138 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 13 04:08:02.784668 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-12902a62-429b-42b7-a4a5-c6d7401785cf None None] Authenticating user token {{(pid=81408) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:08:02.881124 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-12902a62-429b-42b7-a4a5-c6d7401785cf None None] Validating token access rules against request {{(pid=81408) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:08:02.914663 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dd6f3744-eb6e-4e74-9a0e-39ada88cfb68 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81408) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:08:02.915274 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd6f3744-eb6e-4e74-9a0e-39ada88cfb68 None admin] REQUEST_METHOD: `POST` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:08:02.915603 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd6f3744-eb6e-4e74-9a0e-39ada88cfb68 None admin] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:08:02.915879 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd6f3744-eb6e-4e74-9a0e-39ada88cfb68 None admin] PATH_INFO: `/v3/users` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:08:02.916525 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dd6f3744-eb6e-4e74-9a0e-39ada88cfb68 None admin] RBAC: Authorizing `identity:create_user()` {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:08:02.917989 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dd6f3744-eb6e-4e74-9a0e-39ada88cfb68 None admin] RBAC: Authorization granted {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:08:02.943625 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 25/54] 10.222.0.27 () {60 vars in 1132 bytes} [Thu Oct 13 04:08:02 2022] POST /identity/v3/users => generated 275 bytes in 159 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 13 04:08:06.204444 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9197aac5-e8ac-46e3-bfc2-a57e002b7ebf None None] REQUEST_METHOD: `GET` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:08:06.205456 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9197aac5-e8ac-46e3-bfc2-a57e002b7ebf None None] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:08:06.205959 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9197aac5-e8ac-46e3-bfc2-a57e002b7ebf None None] PATH_INFO: `/` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:08:06.207543 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 30/55] 10.222.0.27 () {54 vars in 927 bytes} [Thu Oct 13 04:08:06 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 13 04:08:06.214086 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a785787c-fb0b-4287-a9c3-ee67435aef6a None None] REQUEST_METHOD: `POST` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:08:06.214997 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a785787c-fb0b-4287-a9c3-ee67435aef6a None None] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:08:06.215483 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a785787c-fb0b-4287-a9c3-ee67435aef6a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:08:06.238845 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.auth.core [None req-a785787c-fb0b-4287-a9c3-ee67435aef6a None None] MFA Rules not processed for user `0e2d979563d6421995451a0e86bf3022`. Rule list: `[]` (Enabled: `True`). {{(pid=81408) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 13 04:08:06.240333 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.fernet_utils [None req-a785787c-fb0b-4287-a9c3-ee67435aef6a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81408) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:08:06.291338 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG dbcounter [-] [81408] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81408) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:08:06.386076 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 26/56] 10.222.0.27 () {58 vars in 1024 bytes} [Thu Oct 13 04:08:06 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 173 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 13 04:08:06.446298 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f2c3dea-6711-4310-a89a-15b8bcb80ea2 None None] REQUEST_METHOD: `GET` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:08:06.446910 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f2c3dea-6711-4310-a89a-15b8bcb80ea2 None None] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:08:06.447368 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f2c3dea-6711-4310-a89a-15b8bcb80ea2 None None] PATH_INFO: `/` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:08:06.448999 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 31/57] 10.222.0.27 () {54 vars in 927 bytes} [Thu Oct 13 04:08:06 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 13 04:08:06.455069 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9bd195f-f9f5-4307-bf6d-7c0fc76abc22 None None] REQUEST_METHOD: `POST` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:08:06.455608 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9bd195f-f9f5-4307-bf6d-7c0fc76abc22 None None] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:08:06.456078 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9bd195f-f9f5-4307-bf6d-7c0fc76abc22 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:08:06.479190 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.auth.core [None req-e9bd195f-f9f5-4307-bf6d-7c0fc76abc22 None None] MFA Rules not processed for user `0e2d979563d6421995451a0e86bf3022`. Rule list: `[]` (Enabled: `True`). {{(pid=81408) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 13 04:08:06.480712 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.fernet_utils [None req-e9bd195f-f9f5-4307-bf6d-7c0fc76abc22 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81408) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:08:06.592708 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 27/58] 10.222.0.27 () {58 vars in 1024 bytes} [Thu Oct 13 04:08:06 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 139 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Oct 13 04:08:06.635265 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a18fd5a-7cc1-49d5-9e30-a6bfd5c54594 None None] REQUEST_METHOD: `GET` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:08:06.635910 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a18fd5a-7cc1-49d5-9e30-a6bfd5c54594 None None] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:08:06.636386 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a18fd5a-7cc1-49d5-9e30-a6bfd5c54594 None None] PATH_INFO: `/` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:08:06.638182 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 32/59] 10.222.0.27 () {54 vars in 927 bytes} [Thu Oct 13 04:08:06 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 13 04:08:06.654348 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e9be4e6-3e53-4a27-953f-dacdd1a1a40f None None] REQUEST_METHOD: `POST` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:08:06.654655 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e9be4e6-3e53-4a27-953f-dacdd1a1a40f None None] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:08:06.654908 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e9be4e6-3e53-4a27-953f-dacdd1a1a40f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:08:06.687545 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.auth.core [None req-0e9be4e6-3e53-4a27-953f-dacdd1a1a40f None None] MFA Rules not processed for user `0e2d979563d6421995451a0e86bf3022`. Rule list: `[]` (Enabled: `True`). {{(pid=81408) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 13 04:08:06.688805 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.fernet_utils [None req-0e9be4e6-3e53-4a27-953f-dacdd1a1a40f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81408) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:08:06.811999 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 28/60] 10.222.0.27 () {58 vars in 1024 bytes} [Thu Oct 13 04:08:06 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 159 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 13 04:08:06.831865 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f5f005a-a3ee-4f14-afd9-9e7884d4b012 None None] REQUEST_METHOD: `GET` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:08:06.832505 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f5f005a-a3ee-4f14-afd9-9e7884d4b012 None None] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:08:06.832982 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f5f005a-a3ee-4f14-afd9-9e7884d4b012 None None] PATH_INFO: `/` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:08:06.834597 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 33/61] 10.222.0.27 () {54 vars in 927 bytes} [Thu Oct 13 04:08:06 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 13 04:08:06.852730 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0e9fcd1-8c70-474c-8651-dbd297213c6e None None] REQUEST_METHOD: `POST` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:08:06.853021 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0e9fcd1-8c70-474c-8651-dbd297213c6e None None] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:08:06.853301 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0e9fcd1-8c70-474c-8651-dbd297213c6e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:08:06.862080 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8f5f005a-a3ee-4f14-afd9-9e7884d4b012 None None] Authenticating user token {{(pid=81409) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:08:06.875014 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.auth.core [None req-d0e9fcd1-8c70-474c-8651-dbd297213c6e None None] MFA Rules not processed for user `0e2d979563d6421995451a0e86bf3022`. Rule list: `[]` (Enabled: `True`). {{(pid=81408) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 13 04:08:06.876210 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.fernet_utils [None req-d0e9fcd1-8c70-474c-8651-dbd297213c6e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81408) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:08:06.939992 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG dbcounter [-] [81408] Writing DB stats keystone:SELECT=100 {{(pid=81408) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:08:06.959623 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG dbcounter [-] [81409] Writing DB stats keystone:SELECT=100 {{(pid=81409) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:08:06.996201 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 29/62] 10.222.0.27 () {58 vars in 1024 bytes} [Thu Oct 13 04:08:06 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 145 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 13 04:08:07.007646 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d7e7218-789f-4935-93fb-7d48e1e6bd37 None None] REQUEST_METHOD: `GET` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:08:07.007959 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d7e7218-789f-4935-93fb-7d48e1e6bd37 None None] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:08:07.008238 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d7e7218-789f-4935-93fb-7d48e1e6bd37 None None] PATH_INFO: `/` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:08:07.009835 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 30/63] 10.222.0.27 () {54 vars in 927 bytes} [Thu Oct 13 04:08:06 2022] GET /identity => generated 270 bytes in 13 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 13 04:08:07.010440 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9d7e7218-789f-4935-93fb-7d48e1e6bd37 None None] Authenticating user token {{(pid=81408) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:08:07.029948 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8f5f005a-a3ee-4f14-afd9-9e7884d4b012 None None] Validating token access rules against request {{(pid=81409) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:08:07.079221 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-78ad0755-c0b1-448f-904c-b00435a94c9e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81409) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:08:07.080467 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78ad0755-c0b1-448f-904c-b00435a94c9e None admin] REQUEST_METHOD: `GET` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:08:07.081235 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78ad0755-c0b1-448f-904c-b00435a94c9e None admin] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:08:07.081957 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78ad0755-c0b1-448f-904c-b00435a94c9e None admin] PATH_INFO: `/v3/roles/member` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:08:07.095387 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-78ad0755-c0b1-448f-904c-b00435a94c9e None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 13 04:08:07.102012 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application [None req-78ad0755-c0b1-448f-904c-b00435a94c9e None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Oct 13 04:08:07.102012 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 13 04:08:07.102012 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 13 04:08:07.102012 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 13 04:08:07.102012 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 13 04:08:07.102012 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 13 04:08:07.102012 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 13 04:08:07.102012 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 13 04:08:07.102012 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 13 04:08:07.102012 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 13 04:08:07.102012 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 13 04:08:07.102012 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 13 04:08:07.102012 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 13 04:08:07.102012 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 13 04:08:07.102012 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 13 04:08:07.102012 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 13 04:08:07.103160 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 13 04:08:07.103160 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 13 04:08:07.103160 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 13 04:08:07.103160 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 13 04:08:07.103160 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 13 04:08:07.103160 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 13 04:08:07.103160 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 13 04:08:07.103160 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 13 04:08:07.103160 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 13 04:08:07.103160 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application return self.get_or_create( Oct 13 04:08:07.103160 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 13 04:08:07.103160 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application with Lock( Oct 13 04:08:07.103160 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 13 04:08:07.103160 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application return self._enter() Oct 13 04:08:07.103160 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 13 04:08:07.103160 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 13 04:08:07.103950 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 13 04:08:07.103950 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application return self.creator() Oct 13 04:08:07.103950 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 13 04:08:07.103950 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application created_value = creator( Oct 13 04:08:07.103950 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 13 04:08:07.103950 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 13 04:08:07.103950 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 13 04:08:07.103950 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 13 04:08:07.103950 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 13 04:08:07.103950 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 13 04:08:07.103950 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Oct 13 04:08:07.103950 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application Oct 13 04:08:07.106045 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 34/64] 10.222.0.27 () {56 vars in 1099 bytes} [Thu Oct 13 04:08:06 2022] GET /identity/v3/roles/member => generated 84 bytes in 244 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Oct 13 04:08:07.107601 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f27dce3-e2df-4c16-9343-b0f885f3cd70 None None] REQUEST_METHOD: `POST` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:08:07.108071 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f27dce3-e2df-4c16-9343-b0f885f3cd70 None None] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:08:07.108517 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f27dce3-e2df-4c16-9343-b0f885f3cd70 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:08:07.132644 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.auth.core [None req-7f27dce3-e2df-4c16-9343-b0f885f3cd70 None None] MFA Rules not processed for user `0e2d979563d6421995451a0e86bf3022`. Rule list: `[]` (Enabled: `True`). {{(pid=81409) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 13 04:08:07.134956 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.fernet_utils [None req-7f27dce3-e2df-4c16-9343-b0f885f3cd70 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81409) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:08:07.157003 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9d7e7218-789f-4935-93fb-7d48e1e6bd37 None None] Validating token access rules against request {{(pid=81408) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:08:07.193967 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3af5e8f1-a163-4e36-9af4-320c25ef9e8a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81408) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:08:07.194707 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3af5e8f1-a163-4e36-9af4-320c25ef9e8a None admin] REQUEST_METHOD: `GET` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:08:07.194998 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3af5e8f1-a163-4e36-9af4-320c25ef9e8a None admin] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:08:07.195273 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3af5e8f1-a163-4e36-9af4-320c25ef9e8a None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:08:07.208020 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3af5e8f1-a163-4e36-9af4-320c25ef9e8a None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 13 04:08:07.213801 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application [None req-3af5e8f1-a163-4e36-9af4-320c25ef9e8a None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Oct 13 04:08:07.213801 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 13 04:08:07.213801 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 13 04:08:07.213801 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 13 04:08:07.213801 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 13 04:08:07.213801 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 13 04:08:07.213801 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 13 04:08:07.213801 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 13 04:08:07.213801 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 13 04:08:07.213801 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 13 04:08:07.213801 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 13 04:08:07.213801 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 13 04:08:07.213801 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 13 04:08:07.213801 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 13 04:08:07.213801 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 13 04:08:07.213801 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 13 04:08:07.215476 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 13 04:08:07.215476 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 13 04:08:07.215476 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 13 04:08:07.215476 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 13 04:08:07.215476 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 13 04:08:07.215476 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 13 04:08:07.215476 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 13 04:08:07.215476 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 13 04:08:07.215476 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 13 04:08:07.215476 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application return self.get_or_create( Oct 13 04:08:07.215476 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 13 04:08:07.215476 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application with Lock( Oct 13 04:08:07.215476 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 13 04:08:07.215476 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application return self._enter() Oct 13 04:08:07.215476 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 13 04:08:07.215476 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 13 04:08:07.216203 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 13 04:08:07.216203 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application return self.creator() Oct 13 04:08:07.216203 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 13 04:08:07.216203 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application created_value = creator( Oct 13 04:08:07.216203 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 13 04:08:07.216203 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 13 04:08:07.216203 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 13 04:08:07.216203 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 13 04:08:07.216203 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 13 04:08:07.216203 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 13 04:08:07.216203 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Oct 13 04:08:07.216203 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application Oct 13 04:08:07.216203 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 31/65] 10.222.0.27 () {56 vars in 1096 bytes} [Thu Oct 13 04:08:07 2022] GET /identity/v3/roles/admin => generated 83 bytes in 205 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 13 04:08:07.216203 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3af5e8f1-a163-4e36-9af4-320c25ef9e8a None admin] Authenticating user token {{(pid=81408) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:08:07.246562 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG dbcounter [-] [81408] Writing DB stats keystone:SELECT=100 {{(pid=81408) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:08:07.315748 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 35/66] 10.222.0.27 () {58 vars in 1024 bytes} [Thu Oct 13 04:08:07 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 209 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 13 04:08:07.316614 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7f27dce3-e2df-4c16-9343-b0f885f3cd70 None None] Authenticating user token {{(pid=81409) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:08:07.331344 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3af5e8f1-a163-4e36-9af4-320c25ef9e8a None admin] Validating token access rules against request {{(pid=81408) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:08:07.356869 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG dbcounter [-] [81409] Writing DB stats keystone:SELECT=100 {{(pid=81409) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:08:07.374584 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8aee8f73-a510-4773-bac6-a78d9e20af7e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81408) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:08:07.375201 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8aee8f73-a510-4773-bac6-a78d9e20af7e None admin] REQUEST_METHOD: `GET` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:08:07.375490 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8aee8f73-a510-4773-bac6-a78d9e20af7e None admin] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:08:07.375945 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8aee8f73-a510-4773-bac6-a78d9e20af7e None admin] PATH_INFO: `/v3/roles` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:08:07.376440 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8aee8f73-a510-4773-bac6-a78d9e20af7e None admin] RBAC: Adding query filter params (name=member) {{(pid=81408) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 13 04:08:07.376746 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8aee8f73-a510-4773-bac6-a78d9e20af7e None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:08:07.378141 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8aee8f73-a510-4773-bac6-a78d9e20af7e None admin] RBAC: Authorization granted {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:08:07.390816 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 32/67] 10.222.0.27 () {56 vars in 1101 bytes} [Thu Oct 13 04:08:07 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 175 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 13 04:08:07.396020 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8aee8f73-a510-4773-bac6-a78d9e20af7e None admin] Authenticating user token {{(pid=81408) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:08:07.425100 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7f27dce3-e2df-4c16-9343-b0f885f3cd70 None None] Validating token access rules against request {{(pid=81409) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:08:07.461420 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3242929d-5733-441b-b917-9909ccc23702 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81409) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:08:07.462620 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3242929d-5733-441b-b917-9909ccc23702 None admin] REQUEST_METHOD: `GET` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:08:07.462897 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3242929d-5733-441b-b917-9909ccc23702 None admin] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:08:07.463167 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3242929d-5733-441b-b917-9909ccc23702 None admin] PATH_INFO: `/v3/roles/anotherrole` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:08:07.475051 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3242929d-5733-441b-b917-9909ccc23702 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 13 04:08:07.481430 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application [None req-3242929d-5733-441b-b917-9909ccc23702 None admin] Could not find role: anotherrole.: keystone.exception.RoleNotFound: Could not find role: anotherrole. Oct 13 04:08:07.481430 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 13 04:08:07.481430 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 13 04:08:07.481430 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 13 04:08:07.481430 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 13 04:08:07.481430 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 13 04:08:07.481430 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 13 04:08:07.481430 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 13 04:08:07.481430 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 13 04:08:07.481430 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 13 04:08:07.481430 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 13 04:08:07.481430 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 13 04:08:07.481430 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 13 04:08:07.481430 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 13 04:08:07.481430 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 13 04:08:07.481430 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 13 04:08:07.482291 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 13 04:08:07.482291 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 13 04:08:07.482291 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 13 04:08:07.482291 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 13 04:08:07.482291 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 13 04:08:07.482291 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 13 04:08:07.482291 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 13 04:08:07.482291 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 13 04:08:07.482291 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 13 04:08:07.482291 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application return self.get_or_create( Oct 13 04:08:07.482291 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 13 04:08:07.482291 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application with Lock( Oct 13 04:08:07.482291 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 13 04:08:07.482291 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application return self._enter() Oct 13 04:08:07.482291 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 13 04:08:07.482291 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 13 04:08:07.483043 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 13 04:08:07.483043 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application return self.creator() Oct 13 04:08:07.483043 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 13 04:08:07.483043 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application created_value = creator( Oct 13 04:08:07.483043 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 13 04:08:07.483043 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 13 04:08:07.483043 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 13 04:08:07.483043 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 13 04:08:07.483043 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 13 04:08:07.483043 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 13 04:08:07.483043 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: anotherrole. Oct 13 04:08:07.483043 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application Oct 13 04:08:07.483043 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 36/68] 10.222.0.27 () {56 vars in 1114 bytes} [Thu Oct 13 04:08:07 2022] GET /identity/v3/roles/anotherrole => generated 89 bytes in 167 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 13 04:08:07.486259 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3242929d-5733-441b-b917-9909ccc23702 None admin] Authenticating user token {{(pid=81409) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:08:07.492680 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG dbcounter [-] [81408] Writing DB stats keystone:SELECT=100 {{(pid=81408) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:08:07.511784 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8aee8f73-a510-4773-bac6-a78d9e20af7e None admin] Validating token access rules against request {{(pid=81408) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:08:07.546958 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c92cff29-0071-4263-a8d8-dddc3e7aa8c1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81408) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:08:07.547557 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c92cff29-0071-4263-a8d8-dddc3e7aa8c1 None admin] REQUEST_METHOD: `GET` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:08:07.547834 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c92cff29-0071-4263-a8d8-dddc3e7aa8c1 None admin] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:08:07.548105 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c92cff29-0071-4263-a8d8-dddc3e7aa8c1 None admin] PATH_INFO: `/v3/roles` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:08:07.548765 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c92cff29-0071-4263-a8d8-dddc3e7aa8c1 None admin] RBAC: Adding query filter params (name=admin) {{(pid=81408) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 13 04:08:07.549065 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c92cff29-0071-4263-a8d8-dddc3e7aa8c1 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:08:07.550496 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c92cff29-0071-4263-a8d8-dddc3e7aa8c1 None admin] RBAC: Authorization granted {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:08:07.563264 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 33/69] 10.222.0.27 () {56 vars in 1099 bytes} [Thu Oct 13 04:08:07 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 167 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 13 04:08:07.563264 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c92cff29-0071-4263-a8d8-dddc3e7aa8c1 None admin] Authenticating user token {{(pid=81408) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:08:07.593556 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG dbcounter [-] [81409] Writing DB stats keystone:SELECT=100 {{(pid=81409) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:08:07.605596 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3242929d-5733-441b-b917-9909ccc23702 None admin] Validating token access rules against request {{(pid=81409) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:08:07.640940 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6b073af4-64a3-4a41-be91-6e5234cc42af None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81409) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:08:07.641571 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b073af4-64a3-4a41-be91-6e5234cc42af None admin] REQUEST_METHOD: `GET` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:08:07.641850 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b073af4-64a3-4a41-be91-6e5234cc42af None admin] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:08:07.642127 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b073af4-64a3-4a41-be91-6e5234cc42af None admin] PATH_INFO: `/v3/domains/default` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:08:07.643292 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6b073af4-64a3-4a41-be91-6e5234cc42af None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:08:07.644763 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6b073af4-64a3-4a41-be91-6e5234cc42af None admin] RBAC: Authorization granted {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:08:07.646375 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 37/70] 10.222.0.27 () {56 vars in 1108 bytes} [Thu Oct 13 04:08:07 2022] GET /identity/v3/domains/default => generated 199 bytes in 163 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 13 04:08:07.647140 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6b073af4-64a3-4a41-be91-6e5234cc42af None admin] Authenticating user token {{(pid=81409) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:08:07.662663 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c92cff29-0071-4263-a8d8-dddc3e7aa8c1 None admin] Validating token access rules against request {{(pid=81408) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:08:07.697823 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e7f49eee-2840-4046-b63f-a824458f83c6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81408) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:08:07.698474 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7f49eee-2840-4046-b63f-a824458f83c6 None admin] REQUEST_METHOD: `GET` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:08:07.698756 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7f49eee-2840-4046-b63f-a824458f83c6 None admin] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:08:07.699030 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7f49eee-2840-4046-b63f-a824458f83c6 None admin] PATH_INFO: `/v3/users/761e23aab9234a7c9c88dc41d2ddbe8e` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:08:07.713367 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e7f49eee-2840-4046-b63f-a824458f83c6 None admin] RBAC: Authorizing `identity:get_user(user_id=761e23aab9234a7c9c88dc41d2ddbe8e)` {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:08:07.714852 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e7f49eee-2840-4046-b63f-a824458f83c6 None admin] RBAC: Authorization granted {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:08:07.716463 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 34/71] 10.222.0.27 () {56 vars in 1177 bytes} [Thu Oct 13 04:08:07 2022] GET /identity/v3/users/761e23aab9234a7c9c88dc41d2ddbe8e => generated 275 bytes in 157 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 13 04:08:07.717227 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e7f49eee-2840-4046-b63f-a824458f83c6 None admin] Authenticating user token {{(pid=81408) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:08:07.720439 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG dbcounter [-] [81408] Writing DB stats keystone:SELECT=100 {{(pid=81408) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:08:07.748791 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6b073af4-64a3-4a41-be91-6e5234cc42af None admin] Validating token access rules against request {{(pid=81409) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:08:07.783316 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-18026f5b-a1d1-4c7a-b731-6db49a9fc94d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81409) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:08:07.783935 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18026f5b-a1d1-4c7a-b731-6db49a9fc94d None admin] REQUEST_METHOD: `GET` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:08:07.784217 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18026f5b-a1d1-4c7a-b731-6db49a9fc94d None admin] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:08:07.784494 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18026f5b-a1d1-4c7a-b731-6db49a9fc94d None admin] PATH_INFO: `/v3/roles` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:08:07.785155 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-18026f5b-a1d1-4c7a-b731-6db49a9fc94d None admin] RBAC: Adding query filter params (name=anotherrole) {{(pid=81409) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 13 04:08:07.785462 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-18026f5b-a1d1-4c7a-b731-6db49a9fc94d None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:08:07.786925 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-18026f5b-a1d1-4c7a-b731-6db49a9fc94d None admin] RBAC: Authorization granted {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:08:07.794781 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 38/72] 10.222.0.27 () {56 vars in 1111 bytes} [Thu Oct 13 04:08:07 2022] GET /identity/v3/roles?name=anotherrole => generated 334 bytes in 148 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 13 04:08:07.799422 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-18026f5b-a1d1-4c7a-b731-6db49a9fc94d None admin] Authenticating user token {{(pid=81409) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:08:07.821554 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e7f49eee-2840-4046-b63f-a824458f83c6 None admin] Validating token access rules against request {{(pid=81408) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:08:07.827843 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG dbcounter [-] [81409] Writing DB stats keystone:SELECT=100 {{(pid=81409) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:08:07.855212 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6c02215a-f655-4413-8e01-f25b7bf32694 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81408) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:08:07.855774 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c02215a-f655-4413-8e01-f25b7bf32694 None admin] REQUEST_METHOD: `GET` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:08:07.856070 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c02215a-f655-4413-8e01-f25b7bf32694 None admin] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:08:07.856326 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c02215a-f655-4413-8e01-f25b7bf32694 None admin] PATH_INFO: `/v3/users/0e2d979563d6421995451a0e86bf3022` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:08:07.857457 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6c02215a-f655-4413-8e01-f25b7bf32694 None admin] RBAC: Authorizing `identity:get_user(user_id=0e2d979563d6421995451a0e86bf3022)` {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:08:07.858952 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6c02215a-f655-4413-8e01-f25b7bf32694 None admin] RBAC: Authorization granted {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:08:07.860522 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 35/73] 10.222.0.27 () {56 vars in 1177 bytes} [Thu Oct 13 04:08:07 2022] GET /identity/v3/users/0e2d979563d6421995451a0e86bf3022 => generated 247 bytes in 144 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 13 04:08:07.861271 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6c02215a-f655-4413-8e01-f25b7bf32694 None admin] Authenticating user token {{(pid=81408) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:08:07.903741 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-18026f5b-a1d1-4c7a-b731-6db49a9fc94d None admin] Validating token access rules against request {{(pid=81409) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:08:07.930012 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG dbcounter [-] [81408] Writing DB stats keystone:SELECT=100 {{(pid=81408) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:08:07.937406 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ed517ae0-8d4d-4bb7-b369-bb3a3c560950 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81409) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:08:07.938038 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed517ae0-8d4d-4bb7-b369-bb3a3c560950 None admin] REQUEST_METHOD: `GET` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:08:07.938351 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed517ae0-8d4d-4bb7-b369-bb3a3c560950 None admin] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:08:07.938632 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed517ae0-8d4d-4bb7-b369-bb3a3c560950 None admin] PATH_INFO: `/v3/roles/member` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:08:07.948722 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ed517ae0-8d4d-4bb7-b369-bb3a3c560950 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 13 04:08:07.953997 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application [None req-ed517ae0-8d4d-4bb7-b369-bb3a3c560950 None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Oct 13 04:08:07.953997 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 13 04:08:07.953997 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 13 04:08:07.953997 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 13 04:08:07.953997 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 13 04:08:07.953997 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 13 04:08:07.953997 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 13 04:08:07.953997 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 13 04:08:07.953997 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 13 04:08:07.953997 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 13 04:08:07.953997 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 13 04:08:07.953997 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 13 04:08:07.953997 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 13 04:08:07.953997 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 13 04:08:07.953997 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 13 04:08:07.953997 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 13 04:08:07.955081 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 13 04:08:07.955081 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 13 04:08:07.955081 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 13 04:08:07.955081 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 13 04:08:07.955081 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 13 04:08:07.955081 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 13 04:08:07.955081 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 13 04:08:07.955081 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 13 04:08:07.955081 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 13 04:08:07.955081 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application return self.get_or_create( Oct 13 04:08:07.955081 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 13 04:08:07.955081 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application with Lock( Oct 13 04:08:07.955081 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 13 04:08:07.955081 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application return self._enter() Oct 13 04:08:07.955081 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 13 04:08:07.955081 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 13 04:08:07.956005 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 13 04:08:07.956005 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application return self.creator() Oct 13 04:08:07.956005 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 13 04:08:07.956005 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application created_value = creator( Oct 13 04:08:07.956005 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 13 04:08:07.956005 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 13 04:08:07.956005 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 13 04:08:07.956005 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 13 04:08:07.956005 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 13 04:08:07.956005 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 13 04:08:07.956005 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Oct 13 04:08:07.956005 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application Oct 13 04:08:07.956005 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 39/74] 10.222.0.27 () {56 vars in 1099 bytes} [Thu Oct 13 04:08:07 2022] GET /identity/v3/roles/member => generated 84 bytes in 160 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 13 04:08:07.956005 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ed517ae0-8d4d-4bb7-b369-bb3a3c560950 None admin] Authenticating user token {{(pid=81409) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:08:07.960173 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6c02215a-f655-4413-8e01-f25b7bf32694 None admin] Validating token access rules against request {{(pid=81408) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:08:07.997714 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b26b678c-8446-4e35-ad2c-3f09787b9242 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81408) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:08:07.998350 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b26b678c-8446-4e35-ad2c-3f09787b9242 None admin] REQUEST_METHOD: `POST` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:08:07.998636 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b26b678c-8446-4e35-ad2c-3f09787b9242 None admin] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:08:07.998915 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b26b678c-8446-4e35-ad2c-3f09787b9242 None admin] PATH_INFO: `/v3/users` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:08:07.999578 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b26b678c-8446-4e35-ad2c-3f09787b9242 None admin] RBAC: Authorizing `identity:create_user()` {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:08:08.001042 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b26b678c-8446-4e35-ad2c-3f09787b9242 None admin] RBAC: Authorization granted {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:08:08.034242 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 36/75] 10.222.0.27 () {60 vars in 1132 bytes} [Thu Oct 13 04:08:07 2022] POST /identity/v3/users => generated 289 bytes in 160 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 13 04:08:08.034242 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b26b678c-8446-4e35-ad2c-3f09787b9242 None admin] Authenticating user token {{(pid=81408) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:08:08.055105 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG dbcounter [-] [81409] Writing DB stats keystone:SELECT=100 {{(pid=81409) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:08:08.071305 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ed517ae0-8d4d-4bb7-b369-bb3a3c560950 None admin] Validating token access rules against request {{(pid=81409) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:08:08.110055 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0ea5a002-eb93-42f1-b85a-53f21e3d9e8b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81409) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:08:08.110668 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ea5a002-eb93-42f1-b85a-53f21e3d9e8b None admin] REQUEST_METHOD: `GET` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:08:08.110875 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ea5a002-eb93-42f1-b85a-53f21e3d9e8b None admin] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:08:08.111147 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ea5a002-eb93-42f1-b85a-53f21e3d9e8b None admin] PATH_INFO: `/v3/projects/eeec419e34e44c1f881179933c828d97` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:08:08.112360 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0ea5a002-eb93-42f1-b85a-53f21e3d9e8b None admin] RBAC: Authorizing `identity:get_project(project_id=eeec419e34e44c1f881179933c828d97)` {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:08:08.113834 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0ea5a002-eb93-42f1-b85a-53f21e3d9e8b None admin] RBAC: Authorization granted {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:08:08.115616 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 40/76] 10.222.0.27 () {56 vars in 1186 bytes} [Thu Oct 13 04:08:07 2022] GET /identity/v3/projects/eeec419e34e44c1f881179933c828d97 => generated 298 bytes in 160 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 13 04:08:08.116441 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0ea5a002-eb93-42f1-b85a-53f21e3d9e8b None admin] Authenticating user token {{(pid=81409) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:08:08.153094 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b26b678c-8446-4e35-ad2c-3f09787b9242 None admin] Validating token access rules against request {{(pid=81408) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:08:08.188527 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG dbcounter [-] [81408] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81408) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:08:08.200256 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ab379a9f-ea5e-4211-97fd-ea299d40bba8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81408) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:08:08.201674 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab379a9f-ea5e-4211-97fd-ea299d40bba8 None admin] REQUEST_METHOD: `GET` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:08:08.201674 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab379a9f-ea5e-4211-97fd-ea299d40bba8 None admin] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:08:08.201674 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab379a9f-ea5e-4211-97fd-ea299d40bba8 None admin] PATH_INFO: `/v3/users/761e23aab9234a7c9c88dc41d2ddbe8e` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:08:08.203500 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ab379a9f-ea5e-4211-97fd-ea299d40bba8 None admin] RBAC: Authorizing `identity:get_user(user_id=761e23aab9234a7c9c88dc41d2ddbe8e)` {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:08:08.204317 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ab379a9f-ea5e-4211-97fd-ea299d40bba8 None admin] RBAC: Authorization granted {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:08:08.205915 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 37/77] 10.222.0.27 () {56 vars in 1177 bytes} [Thu Oct 13 04:08:08 2022] GET /identity/v3/users/761e23aab9234a7c9c88dc41d2ddbe8e => generated 275 bytes in 184 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 13 04:08:08.210780 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ab379a9f-ea5e-4211-97fd-ea299d40bba8 None admin] Authenticating user token {{(pid=81408) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:08:08.237261 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0ea5a002-eb93-42f1-b85a-53f21e3d9e8b None admin] Validating token access rules against request {{(pid=81409) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:08:08.279346 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eaecb090-48b6-4fdc-8e24-f74a585bd4d2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81409) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:08:08.280301 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eaecb090-48b6-4fdc-8e24-f74a585bd4d2 None admin] REQUEST_METHOD: `GET` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:08:08.280586 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eaecb090-48b6-4fdc-8e24-f74a585bd4d2 None admin] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:08:08.280861 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eaecb090-48b6-4fdc-8e24-f74a585bd4d2 None admin] PATH_INFO: `/v3/projects/eeec419e34e44c1f881179933c828d97` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:08:08.282044 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eaecb090-48b6-4fdc-8e24-f74a585bd4d2 None admin] RBAC: Authorizing `identity:get_project(project_id=eeec419e34e44c1f881179933c828d97)` {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:08:08.285501 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eaecb090-48b6-4fdc-8e24-f74a585bd4d2 None admin] RBAC: Authorization granted {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:08:08.286451 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 41/78] 10.222.0.27 () {56 vars in 1186 bytes} [Thu Oct 13 04:08:08 2022] GET /identity/v3/projects/eeec419e34e44c1f881179933c828d97 => generated 298 bytes in 170 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 13 04:08:08.287035 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eaecb090-48b6-4fdc-8e24-f74a585bd4d2 None admin] Authenticating user token {{(pid=81409) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:08:08.327509 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG dbcounter [-] [81409] Writing DB stats keystone:SELECT=100 {{(pid=81409) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:08:08.336973 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ab379a9f-ea5e-4211-97fd-ea299d40bba8 None admin] Validating token access rules against request {{(pid=81408) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:08:08.383031 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8382bb98-efdd-4074-abda-196e156a751f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81408) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:08:08.383603 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8382bb98-efdd-4074-abda-196e156a751f None admin] REQUEST_METHOD: `GET` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:08:08.383881 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8382bb98-efdd-4074-abda-196e156a751f None admin] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:08:08.384160 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8382bb98-efdd-4074-abda-196e156a751f None admin] PATH_INFO: `/v3/roles` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:08:08.384835 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8382bb98-efdd-4074-abda-196e156a751f None admin] RBAC: Adding query filter params (name=member) {{(pid=81408) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 13 04:08:08.385140 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8382bb98-efdd-4074-abda-196e156a751f None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:08:08.387021 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8382bb98-efdd-4074-abda-196e156a751f None admin] RBAC: Authorization granted {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:08:08.395933 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 38/79] 10.222.0.27 () {56 vars in 1101 bytes} [Thu Oct 13 04:08:08 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 189 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 13 04:08:08.399681 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8382bb98-efdd-4074-abda-196e156a751f None admin] Authenticating user token {{(pid=81408) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:08:08.427585 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eaecb090-48b6-4fdc-8e24-f74a585bd4d2 None admin] Validating token access rules against request {{(pid=81409) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:08:08.470574 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0094b8ae-8984-4d85-9e5e-fcfe7435864c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81409) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:08:08.471240 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0094b8ae-8984-4d85-9e5e-fcfe7435864c None admin] REQUEST_METHOD: `PUT` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:08:08.471516 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0094b8ae-8984-4d85-9e5e-fcfe7435864c None admin] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:08:08.471788 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0094b8ae-8984-4d85-9e5e-fcfe7435864c None admin] PATH_INFO: `/v3/projects/eeec419e34e44c1f881179933c828d97/users/761e23aab9234a7c9c88dc41d2ddbe8e/roles/cc56c4ae13a7424ea771c15da4b153c1` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:08:08.474599 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0094b8ae-8984-4d85-9e5e-fcfe7435864c None admin] RBAC: Authorizing `identity:create_grant(project_id=eeec419e34e44c1f881179933c828d97, user_id=761e23aab9234a7c9c88dc41d2ddbe8e, role_id=cc56c4ae13a7424ea771c15da4b153c1)` {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:08:08.476575 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0094b8ae-8984-4d85-9e5e-fcfe7435864c None admin] RBAC: Authorization granted {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:08:08.480964 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG dbcounter [-] [81408] Writing DB stats keystone:SELECT=100 {{(pid=81408) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:08:08.496602 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 42/80] 10.222.0.27 () {58 vars in 1439 bytes} [Thu Oct 13 04:08:08 2022] PUT /identity/v3/projects/eeec419e34e44c1f881179933c828d97/users/761e23aab9234a7c9c88dc41d2ddbe8e/roles/cc56c4ae13a7424ea771c15da4b153c1 => generated 0 bytes in 210 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 13 04:08:08.498093 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0094b8ae-8984-4d85-9e5e-fcfe7435864c None admin] Authenticating user token {{(pid=81409) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:08:08.541568 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8382bb98-efdd-4074-abda-196e156a751f None admin] Validating token access rules against request {{(pid=81408) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:08:08.612091 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-260f9fb2-6ec5-486a-b12e-bb65a1dfb1f9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81408) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:08:08.612749 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-260f9fb2-6ec5-486a-b12e-bb65a1dfb1f9 None admin] REQUEST_METHOD: `GET` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:08:08.613027 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-260f9fb2-6ec5-486a-b12e-bb65a1dfb1f9 None admin] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:08:08.613298 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-260f9fb2-6ec5-486a-b12e-bb65a1dfb1f9 None admin] PATH_INFO: `/v3/projects/eeec419e34e44c1f881179933c828d97` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:08:08.617319 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-260f9fb2-6ec5-486a-b12e-bb65a1dfb1f9 None admin] RBAC: Authorizing `identity:get_project(project_id=eeec419e34e44c1f881179933c828d97)` {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:08:08.619428 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-260f9fb2-6ec5-486a-b12e-bb65a1dfb1f9 None admin] RBAC: Authorization granted {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:08:08.621438 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 39/81] 10.222.0.27 () {56 vars in 1186 bytes} [Thu Oct 13 04:08:08 2022] GET /identity/v3/projects/eeec419e34e44c1f881179933c828d97 => generated 298 bytes in 225 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 13 04:08:08.622004 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-260f9fb2-6ec5-486a-b12e-bb65a1dfb1f9 None admin] Authenticating user token {{(pid=81408) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:08:08.638856 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG dbcounter [-] [81409] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81409) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:08:08.679467 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0094b8ae-8984-4d85-9e5e-fcfe7435864c None admin] Validating token access rules against request {{(pid=81409) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:08:08.735711 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2e037549-c4f8-4e20-8e36-87494ab2d325 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81409) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:08:08.736383 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e037549-c4f8-4e20-8e36-87494ab2d325 None admin] REQUEST_METHOD: `PUT` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:08:08.736665 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e037549-c4f8-4e20-8e36-87494ab2d325 None admin] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:08:08.736942 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e037549-c4f8-4e20-8e36-87494ab2d325 None admin] PATH_INFO: `/v3/projects/eeec419e34e44c1f881179933c828d97/users/0e2d979563d6421995451a0e86bf3022/roles/ef55fbd94e1040be83695754075a7ed5` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:08:08.739695 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2e037549-c4f8-4e20-8e36-87494ab2d325 None admin] RBAC: Authorizing `identity:create_grant(project_id=eeec419e34e44c1f881179933c828d97, user_id=0e2d979563d6421995451a0e86bf3022, role_id=ef55fbd94e1040be83695754075a7ed5)` {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:08:08.741152 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2e037549-c4f8-4e20-8e36-87494ab2d325 None admin] RBAC: Authorization granted {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:08:08.762003 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 43/82] 10.222.0.27 () {58 vars in 1439 bytes} [Thu Oct 13 04:08:08 2022] PUT /identity/v3/projects/eeec419e34e44c1f881179933c828d97/users/0e2d979563d6421995451a0e86bf3022/roles/ef55fbd94e1040be83695754075a7ed5 => generated 0 bytes in 265 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 13 04:08:08.763542 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2e037549-c4f8-4e20-8e36-87494ab2d325 None admin] Authenticating user token {{(pid=81409) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:08:08.816340 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-260f9fb2-6ec5-486a-b12e-bb65a1dfb1f9 None admin] Validating token access rules against request {{(pid=81408) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:08:08.854561 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG dbcounter [-] [81408] Writing DB stats keystone:SELECT=100 {{(pid=81408) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:08:08.864923 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e2d1b322-36da-4566-9e0d-dfb40446525d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81408) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:08:08.865621 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2d1b322-36da-4566-9e0d-dfb40446525d None admin] REQUEST_METHOD: `GET` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:08:08.865908 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2d1b322-36da-4566-9e0d-dfb40446525d None admin] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:08:08.866187 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2d1b322-36da-4566-9e0d-dfb40446525d None admin] PATH_INFO: `/v3/users/761e23aab9234a7c9c88dc41d2ddbe8e` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:08:08.867527 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e2d1b322-36da-4566-9e0d-dfb40446525d None admin] RBAC: Authorizing `identity:get_user(user_id=761e23aab9234a7c9c88dc41d2ddbe8e)` {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:08:08.870253 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e2d1b322-36da-4566-9e0d-dfb40446525d None admin] RBAC: Authorization granted {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:08:08.871013 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 40/83] 10.222.0.27 () {56 vars in 1177 bytes} [Thu Oct 13 04:08:08 2022] GET /identity/v3/users/761e23aab9234a7c9c88dc41d2ddbe8e => generated 275 bytes in 249 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 13 04:08:08.876777 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e2d1b322-36da-4566-9e0d-dfb40446525d None admin] Authenticating user token {{(pid=81408) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:08:08.979079 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2e037549-c4f8-4e20-8e36-87494ab2d325 None admin] Validating token access rules against request {{(pid=81409) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:08:09.033163 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e2d1b322-36da-4566-9e0d-dfb40446525d None admin] Validating token access rules against request {{(pid=81408) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:08:09.065309 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-70558b06-15d1-4ee5-90fd-0ca22733f2a0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81409) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:08:09.066007 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70558b06-15d1-4ee5-90fd-0ca22733f2a0 None admin] REQUEST_METHOD: `PUT` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:08:09.067706 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70558b06-15d1-4ee5-90fd-0ca22733f2a0 None admin] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:08:09.067997 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70558b06-15d1-4ee5-90fd-0ca22733f2a0 None admin] PATH_INFO: `/v3/projects/eeec419e34e44c1f881179933c828d97/users/761e23aab9234a7c9c88dc41d2ddbe8e/roles/743995e93fef47fd91204c3d701054b0` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:08:09.072567 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-70558b06-15d1-4ee5-90fd-0ca22733f2a0 None admin] RBAC: Authorizing `identity:create_grant(project_id=eeec419e34e44c1f881179933c828d97, user_id=761e23aab9234a7c9c88dc41d2ddbe8e, role_id=743995e93fef47fd91204c3d701054b0)` {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:08:09.074060 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-70558b06-15d1-4ee5-90fd-0ca22733f2a0 None admin] RBAC: Authorization granted {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:08:09.088219 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a8617cb5-247f-4b4e-a3cb-b6494824c984 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81408) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:08:09.089121 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8617cb5-247f-4b4e-a3cb-b6494824c984 None admin] REQUEST_METHOD: `GET` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:08:09.089409 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8617cb5-247f-4b4e-a3cb-b6494824c984 None admin] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:08:09.089691 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8617cb5-247f-4b4e-a3cb-b6494824c984 None admin] PATH_INFO: `/v3/projects/fb592092172d4c10b772d5c6d2dc9c62` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:08:09.093529 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a8617cb5-247f-4b4e-a3cb-b6494824c984 None admin] RBAC: Authorizing `identity:get_project(project_id=fb592092172d4c10b772d5c6d2dc9c62)` {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:08:09.098949 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 44/84] 10.222.0.27 () {58 vars in 1439 bytes} [Thu Oct 13 04:08:08 2022] PUT /identity/v3/projects/eeec419e34e44c1f881179933c828d97/users/761e23aab9234a7c9c88dc41d2ddbe8e/roles/743995e93fef47fd91204c3d701054b0 => generated 0 bytes in 336 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 13 04:08:09.099686 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a8617cb5-247f-4b4e-a3cb-b6494824c984 None admin] RBAC: Authorization granted {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:08:09.101803 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 41/85] 10.222.0.27 () {56 vars in 1186 bytes} [Thu Oct 13 04:08:08 2022] GET /identity/v3/projects/fb592092172d4c10b772d5c6d2dc9c62 => generated 312 bytes in 225 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 13 04:08:09.108736 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-70558b06-15d1-4ee5-90fd-0ca22733f2a0 None admin] Authenticating user token {{(pid=81409) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:08:09.113258 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG dbcounter [-] [81409] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=81409) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:08:09.238119 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-70558b06-15d1-4ee5-90fd-0ca22733f2a0 None admin] Validating token access rules against request {{(pid=81409) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:08:09.278051 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-559f352a-179a-4619-9971-852625b0c7b3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81409) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:08:09.278959 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-559f352a-179a-4619-9971-852625b0c7b3 None admin] REQUEST_METHOD: `PUT` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:08:09.279244 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-559f352a-179a-4619-9971-852625b0c7b3 None admin] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:08:09.279520 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-559f352a-179a-4619-9971-852625b0c7b3 None admin] PATH_INFO: `/v3/projects/fb592092172d4c10b772d5c6d2dc9c62/users/761e23aab9234a7c9c88dc41d2ddbe8e/roles/cc56c4ae13a7424ea771c15da4b153c1` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:08:09.281655 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-559f352a-179a-4619-9971-852625b0c7b3 None admin] RBAC: Authorizing `identity:create_grant(project_id=fb592092172d4c10b772d5c6d2dc9c62, user_id=761e23aab9234a7c9c88dc41d2ddbe8e, role_id=cc56c4ae13a7424ea771c15da4b153c1)` {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:08:09.283225 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-559f352a-179a-4619-9971-852625b0c7b3 None admin] RBAC: Authorization granted {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:08:09.300793 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 45/86] 10.222.0.27 () {58 vars in 1439 bytes} [Thu Oct 13 04:08:09 2022] PUT /identity/v3/projects/fb592092172d4c10b772d5c6d2dc9c62/users/761e23aab9234a7c9c88dc41d2ddbe8e/roles/cc56c4ae13a7424ea771c15da4b153c1 => generated 0 bytes in 192 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 13 04:08:10.843608 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bbcf4d8b-48f2-41b3-a22d-22cc05b77176 None None] REQUEST_METHOD: `GET` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:08:10.844613 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bbcf4d8b-48f2-41b3-a22d-22cc05b77176 None None] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:08:10.845106 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bbcf4d8b-48f2-41b3-a22d-22cc05b77176 None None] PATH_INFO: `/` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:08:10.846702 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 42/87] 10.222.0.27 () {54 vars in 927 bytes} [Thu Oct 13 04:08:10 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 13 04:08:10.864535 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-beb05939-ad79-46f9-b99b-9710a10d1db9 None None] REQUEST_METHOD: `POST` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:08:10.865452 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-beb05939-ad79-46f9-b99b-9710a10d1db9 None None] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:08:10.865926 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-beb05939-ad79-46f9-b99b-9710a10d1db9 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:08:10.895204 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.auth.core [None req-beb05939-ad79-46f9-b99b-9710a10d1db9 None None] MFA Rules not processed for user `0e2d979563d6421995451a0e86bf3022`. Rule list: `[]` (Enabled: `True`). {{(pid=81409) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 13 04:08:10.896758 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.fernet_utils [None req-beb05939-ad79-46f9-b99b-9710a10d1db9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81409) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:08:10.970311 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG dbcounter [-] [81409] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81409) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:08:10.986350 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 46/88] 10.222.0.27 () {58 vars in 1024 bytes} [Thu Oct 13 04:08:10 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 123 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 13 04:08:11.378697 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-993ef480-6830-4e21-a6dd-5a22e44f08ad None None] REQUEST_METHOD: `GET` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:08:11.379384 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-993ef480-6830-4e21-a6dd-5a22e44f08ad None None] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:08:11.379876 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-993ef480-6830-4e21-a6dd-5a22e44f08ad None None] PATH_INFO: `/` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:08:11.381434 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 43/89] 10.222.0.27 () {54 vars in 927 bytes} [Thu Oct 13 04:08:11 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 13 04:08:11.386832 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4d04270-9a37-45de-a396-5fcf8b36692b None None] REQUEST_METHOD: `POST` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:08:11.387356 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4d04270-9a37-45de-a396-5fcf8b36692b None None] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:08:11.387848 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4d04270-9a37-45de-a396-5fcf8b36692b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:08:11.428274 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.auth.core [None req-d4d04270-9a37-45de-a396-5fcf8b36692b None None] MFA Rules not processed for user `0e2d979563d6421995451a0e86bf3022`. Rule list: `[]` (Enabled: `True`). {{(pid=81409) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 13 04:08:11.429749 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.fernet_utils [None req-d4d04270-9a37-45de-a396-5fcf8b36692b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81409) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:08:11.516339 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 47/90] 10.222.0.27 () {58 vars in 1024 bytes} [Thu Oct 13 04:08:11 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 130 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 13 04:08:11.532755 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49dfb485-d5d8-421d-a9f2-2bf1feabcc4e None None] REQUEST_METHOD: `GET` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:08:11.533047 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49dfb485-d5d8-421d-a9f2-2bf1feabcc4e None None] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:08:11.533430 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49dfb485-d5d8-421d-a9f2-2bf1feabcc4e None None] PATH_INFO: `/` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:08:11.534751 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 44/91] 10.222.0.27 () {54 vars in 927 bytes} [Thu Oct 13 04:08:11 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 13 04:08:11.535933 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d4d04270-9a37-45de-a396-5fcf8b36692b None None] Authenticating user token {{(pid=81409) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:08:11.543649 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9bad6617-7850-4bf1-8a16-b4d369c600f9 None None] REQUEST_METHOD: `POST` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:08:11.544157 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9bad6617-7850-4bf1-8a16-b4d369c600f9 None None] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:08:11.544612 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9bad6617-7850-4bf1-8a16-b4d369c600f9 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:08:11.569341 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.auth.core [None req-9bad6617-7850-4bf1-8a16-b4d369c600f9 None None] MFA Rules not processed for user `0e2d979563d6421995451a0e86bf3022`. Rule list: `[]` (Enabled: `True`). {{(pid=81408) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 13 04:08:11.572850 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.fernet_utils [None req-9bad6617-7850-4bf1-8a16-b4d369c600f9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81408) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:08:11.690670 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG dbcounter [-] [81408] Writing DB stats keystone:SELECT=100 {{(pid=81408) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:08:11.709151 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 45/92] 10.222.0.27 () {58 vars in 1024 bytes} [Thu Oct 13 04:08:11 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 166 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Oct 13 04:08:11.726137 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d4d04270-9a37-45de-a396-5fcf8b36692b None None] Validating token access rules against request {{(pid=81409) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:08:11.756351 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG dbcounter [-] [81409] Writing DB stats keystone:SELECT=100 {{(pid=81409) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:08:11.771024 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b44f896c-eea4-42ea-b5e9-f92d6fa86964 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81409) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:08:11.772016 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b44f896c-eea4-42ea-b5e9-f92d6fa86964 None admin] REQUEST_METHOD: `GET` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:08:11.772472 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b44f896c-eea4-42ea-b5e9-f92d6fa86964 None admin] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:08:11.772921 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b44f896c-eea4-42ea-b5e9-f92d6fa86964 None admin] PATH_INFO: `/v3/roles/reader` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:08:11.791458 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b44f896c-eea4-42ea-b5e9-f92d6fa86964 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 13 04:08:11.796561 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application [None req-b44f896c-eea4-42ea-b5e9-f92d6fa86964 None admin] Could not find role: reader.: keystone.exception.RoleNotFound: Could not find role: reader. Oct 13 04:08:11.796561 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 13 04:08:11.796561 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 13 04:08:11.796561 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 13 04:08:11.796561 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 13 04:08:11.796561 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 13 04:08:11.796561 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 13 04:08:11.796561 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 13 04:08:11.796561 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 13 04:08:11.796561 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 13 04:08:11.796561 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 13 04:08:11.796561 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 13 04:08:11.796561 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 13 04:08:11.796561 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 13 04:08:11.796561 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 13 04:08:11.796561 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 13 04:08:11.797446 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 13 04:08:11.797446 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 13 04:08:11.797446 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 13 04:08:11.797446 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 13 04:08:11.797446 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 13 04:08:11.797446 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 13 04:08:11.797446 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 13 04:08:11.797446 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 13 04:08:11.797446 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 13 04:08:11.797446 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application return self.get_or_create( Oct 13 04:08:11.797446 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 13 04:08:11.797446 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application with Lock( Oct 13 04:08:11.797446 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 13 04:08:11.797446 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application return self._enter() Oct 13 04:08:11.797446 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 13 04:08:11.797446 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 13 04:08:11.798229 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 13 04:08:11.798229 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application return self.creator() Oct 13 04:08:11.798229 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 13 04:08:11.798229 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application created_value = creator( Oct 13 04:08:11.798229 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 13 04:08:11.798229 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 13 04:08:11.798229 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 13 04:08:11.798229 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 13 04:08:11.798229 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 13 04:08:11.798229 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 13 04:08:11.798229 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: reader. Oct 13 04:08:11.798229 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application Oct 13 04:08:11.798229 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 48/93] 10.222.0.27 () {56 vars in 1099 bytes} [Thu Oct 13 04:08:11 2022] GET /identity/v3/roles/reader => generated 84 bytes in 262 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 13 04:08:11.803696 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9bad6617-7850-4bf1-8a16-b4d369c600f9 None None] Authenticating user token {{(pid=81408) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:08:11.960009 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9bad6617-7850-4bf1-8a16-b4d369c600f9 None None] Validating token access rules against request {{(pid=81408) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:08:12.012340 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-30ea07f4-311f-41ee-b3a4-9572213ebaeb None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81408) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:08:12.012954 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30ea07f4-311f-41ee-b3a4-9572213ebaeb None admin] REQUEST_METHOD: `GET` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:08:12.013232 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30ea07f4-311f-41ee-b3a4-9572213ebaeb None admin] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:08:12.013503 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30ea07f4-311f-41ee-b3a4-9572213ebaeb None admin] PATH_INFO: `/v3/roles` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:08:12.014529 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-30ea07f4-311f-41ee-b3a4-9572213ebaeb None admin] RBAC: Adding query filter params (name=reader) {{(pid=81408) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 13 04:08:12.014829 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-30ea07f4-311f-41ee-b3a4-9572213ebaeb None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:08:12.015579 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1cd1f93-6926-45bb-8c7d-3c008d0fcca4 None None] REQUEST_METHOD: `GET` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:08:12.015994 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1cd1f93-6926-45bb-8c7d-3c008d0fcca4 None None] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:08:12.016228 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-30ea07f4-311f-41ee-b3a4-9572213ebaeb None admin] RBAC: Authorization granted {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:08:12.016708 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1cd1f93-6926-45bb-8c7d-3c008d0fcca4 None None] PATH_INFO: `/` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:08:12.018068 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 49/94] 10.222.0.27 () {54 vars in 927 bytes} [Thu Oct 13 04:08:12 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 13 04:08:12.025709 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 46/95] 10.222.0.27 () {56 vars in 1101 bytes} [Thu Oct 13 04:08:11 2022] GET /identity/v3/roles?name=reader => generated 341 bytes in 222 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 13 04:08:12.031420 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f1cd1f93-6926-45bb-8c7d-3c008d0fcca4 None None] Authenticating user token {{(pid=81409) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:08:12.031913 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b363d58b-4882-41a2-87bf-502c6b2ab21c None None] REQUEST_METHOD: `POST` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:08:12.032206 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b363d58b-4882-41a2-87bf-502c6b2ab21c None None] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:08:12.032483 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b363d58b-4882-41a2-87bf-502c6b2ab21c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:08:12.052679 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.auth.core [None req-b363d58b-4882-41a2-87bf-502c6b2ab21c None None] MFA Rules not processed for user `0e2d979563d6421995451a0e86bf3022`. Rule list: `[]` (Enabled: `True`). {{(pid=81408) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 13 04:08:12.053996 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.fernet_utils [None req-b363d58b-4882-41a2-87bf-502c6b2ab21c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81408) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:08:12.108106 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG dbcounter [-] [81408] Writing DB stats keystone:SELECT=100 {{(pid=81408) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:08:12.126877 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 47/96] 10.222.0.27 () {58 vars in 1024 bytes} [Thu Oct 13 04:08:12 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 96 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 13 04:08:12.127769 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b363d58b-4882-41a2-87bf-502c6b2ab21c None None] Authenticating user token {{(pid=81408) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:08:12.210564 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f1cd1f93-6926-45bb-8c7d-3c008d0fcca4 None None] Validating token access rules against request {{(pid=81409) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:08:12.270268 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b363d58b-4882-41a2-87bf-502c6b2ab21c None None] Validating token access rules against request {{(pid=81408) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:08:12.280211 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c5839c31-2d59-4070-81a2-ed3f9a7f438d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81409) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:08:12.280850 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5839c31-2d59-4070-81a2-ed3f9a7f438d None admin] REQUEST_METHOD: `GET` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:08:12.281126 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5839c31-2d59-4070-81a2-ed3f9a7f438d None admin] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:08:12.281398 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5839c31-2d59-4070-81a2-ed3f9a7f438d None admin] PATH_INFO: `/v3/users/4cba1177c8534417a4db1df4fe654898` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:08:12.305022 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c5839c31-2d59-4070-81a2-ed3f9a7f438d None admin] RBAC: Authorizing `identity:get_user(user_id=4cba1177c8534417a4db1df4fe654898)` {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:08:12.306881 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c5839c31-2d59-4070-81a2-ed3f9a7f438d None admin] RBAC: Authorization granted {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:08:12.308917 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 50/97] 10.222.0.27 () {56 vars in 1177 bytes} [Thu Oct 13 04:08:12 2022] GET /identity/v3/users/4cba1177c8534417a4db1df4fe654898 => generated 289 bytes in 278 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 13 04:08:12.309923 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c5839c31-2d59-4070-81a2-ed3f9a7f438d None admin] Authenticating user token {{(pid=81409) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:08:12.347447 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-15af9e48-a35b-4216-80fe-b2584922bbd3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81408) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:08:12.348709 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15af9e48-a35b-4216-80fe-b2584922bbd3 None admin] REQUEST_METHOD: `GET` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:08:12.349350 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15af9e48-a35b-4216-80fe-b2584922bbd3 None admin] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:08:12.350933 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15af9e48-a35b-4216-80fe-b2584922bbd3 None admin] PATH_INFO: `/v3/domains/default` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:08:12.353761 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-15af9e48-a35b-4216-80fe-b2584922bbd3 None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:08:12.356187 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-15af9e48-a35b-4216-80fe-b2584922bbd3 None admin] RBAC: Authorization granted {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:08:12.358291 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 48/98] 10.222.0.27 () {56 vars in 1108 bytes} [Thu Oct 13 04:08:12 2022] GET /identity/v3/domains/default => generated 199 bytes in 231 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 13 04:08:12.361308 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-15af9e48-a35b-4216-80fe-b2584922bbd3 None admin] Authenticating user token {{(pid=81408) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:08:12.382160 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG dbcounter [-] [81409] Writing DB stats keystone:SELECT=100 {{(pid=81409) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:08:12.450709 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG dbcounter [-] [81408] Writing DB stats keystone:SELECT=100 {{(pid=81408) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:08:12.494046 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c5839c31-2d59-4070-81a2-ed3f9a7f438d None admin] Validating token access rules against request {{(pid=81409) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:08:12.498280 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-15af9e48-a35b-4216-80fe-b2584922bbd3 None admin] Validating token access rules against request {{(pid=81408) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:08:12.531849 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0d66836e-be1e-4805-b777-10c525821935 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81409) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:08:12.532492 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d66836e-be1e-4805-b777-10c525821935 None admin] REQUEST_METHOD: `GET` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:08:12.532773 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d66836e-be1e-4805-b777-10c525821935 None admin] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:08:12.533058 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d66836e-be1e-4805-b777-10c525821935 None admin] PATH_INFO: `/v3/roles/member` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:08:12.542237 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff8e5831-6fe5-44a7-bbfa-f82711a70cfc None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81408) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:08:12.543149 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff8e5831-6fe5-44a7-bbfa-f82711a70cfc None admin] REQUEST_METHOD: `GET` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:08:12.543645 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff8e5831-6fe5-44a7-bbfa-f82711a70cfc None admin] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:08:12.544352 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff8e5831-6fe5-44a7-bbfa-f82711a70cfc None admin] PATH_INFO: `/v3/projects/eeec419e34e44c1f881179933c828d97` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:08:12.544661 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0d66836e-be1e-4805-b777-10c525821935 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 13 04:08:12.545896 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ff8e5831-6fe5-44a7-bbfa-f82711a70cfc None admin] RBAC: Authorizing `identity:get_project(project_id=eeec419e34e44c1f881179933c828d97)` {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:08:12.547561 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ff8e5831-6fe5-44a7-bbfa-f82711a70cfc None admin] RBAC: Authorization granted {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:08:12.549420 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 49/99] 10.222.0.27 () {56 vars in 1186 bytes} [Thu Oct 13 04:08:12 2022] GET /identity/v3/projects/eeec419e34e44c1f881179933c828d97 => generated 298 bytes in 189 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 13 04:08:12.549666 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application [None req-0d66836e-be1e-4805-b777-10c525821935 None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Oct 13 04:08:12.549666 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 13 04:08:12.549666 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 13 04:08:12.549666 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 13 04:08:12.549666 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 13 04:08:12.549666 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 13 04:08:12.549666 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 13 04:08:12.549666 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 13 04:08:12.549666 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 13 04:08:12.549666 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 13 04:08:12.549666 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 13 04:08:12.549666 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 13 04:08:12.549666 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 13 04:08:12.549666 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 13 04:08:12.549666 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 13 04:08:12.549666 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 13 04:08:12.550496 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 13 04:08:12.550496 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 13 04:08:12.550496 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 13 04:08:12.550496 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 13 04:08:12.550496 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 13 04:08:12.550496 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 13 04:08:12.550496 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 13 04:08:12.550496 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 13 04:08:12.550496 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 13 04:08:12.550496 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application return self.get_or_create( Oct 13 04:08:12.550496 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 13 04:08:12.550496 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application with Lock( Oct 13 04:08:12.550496 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 13 04:08:12.550496 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application return self._enter() Oct 13 04:08:12.550496 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 13 04:08:12.550496 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 13 04:08:12.551242 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 13 04:08:12.551242 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application return self.creator() Oct 13 04:08:12.551242 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 13 04:08:12.551242 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application created_value = creator( Oct 13 04:08:12.551242 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 13 04:08:12.551242 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 13 04:08:12.551242 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 13 04:08:12.551242 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 13 04:08:12.551242 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 13 04:08:12.551242 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 13 04:08:12.551242 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Oct 13 04:08:12.551242 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application Oct 13 04:08:12.551883 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff8e5831-6fe5-44a7-bbfa-f82711a70cfc None admin] Authenticating user token {{(pid=81408) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:08:12.551948 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 51/100] 10.222.0.27 () {56 vars in 1099 bytes} [Thu Oct 13 04:08:12 2022] GET /identity/v3/roles/member => generated 84 bytes in 242 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 13 04:08:12.552668 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0d66836e-be1e-4805-b777-10c525821935 None admin] Authenticating user token {{(pid=81409) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:08:12.669028 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0d66836e-be1e-4805-b777-10c525821935 None admin] Validating token access rules against request {{(pid=81409) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:08:12.671486 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG dbcounter [-] [81409] Writing DB stats keystone:SELECT=100 {{(pid=81409) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:08:12.676826 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff8e5831-6fe5-44a7-bbfa-f82711a70cfc None admin] Validating token access rules against request {{(pid=81408) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:08:12.700621 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG dbcounter [-] [81408] Writing DB stats keystone:SELECT=100 {{(pid=81408) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:08:12.708729 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d5f553a0-a33e-49dc-b7b6-9eb5f88febd4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81409) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:08:12.709368 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5f553a0-a33e-49dc-b7b6-9eb5f88febd4 None admin] REQUEST_METHOD: `GET` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:08:12.709649 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5f553a0-a33e-49dc-b7b6-9eb5f88febd4 None admin] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:08:12.709922 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5f553a0-a33e-49dc-b7b6-9eb5f88febd4 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:08:12.715925 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d65b90d0-0f9f-4200-a2eb-01b96a222d95 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81408) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:08:12.716709 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d65b90d0-0f9f-4200-a2eb-01b96a222d95 None admin] REQUEST_METHOD: `POST` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:08:12.716990 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d65b90d0-0f9f-4200-a2eb-01b96a222d95 None admin] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:08:12.717312 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d65b90d0-0f9f-4200-a2eb-01b96a222d95 None admin] PATH_INFO: `/v3/projects` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:08:12.717970 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d65b90d0-0f9f-4200-a2eb-01b96a222d95 None admin] RBAC: Authorizing `identity:create_project()` {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:08:12.719480 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d65b90d0-0f9f-4200-a2eb-01b96a222d95 None admin] RBAC: Authorization granted {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:08:12.725066 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d5f553a0-a33e-49dc-b7b6-9eb5f88febd4 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 13 04:08:12.730951 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application [None req-d5f553a0-a33e-49dc-b7b6-9eb5f88febd4 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Oct 13 04:08:12.730951 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 13 04:08:12.730951 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 13 04:08:12.730951 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 13 04:08:12.730951 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 13 04:08:12.730951 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 13 04:08:12.730951 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 13 04:08:12.730951 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 13 04:08:12.730951 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 13 04:08:12.730951 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 13 04:08:12.730951 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 13 04:08:12.730951 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 13 04:08:12.730951 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 13 04:08:12.730951 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 13 04:08:12.730951 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 13 04:08:12.730951 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 13 04:08:12.731762 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 13 04:08:12.731762 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 13 04:08:12.731762 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 13 04:08:12.731762 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 13 04:08:12.731762 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 13 04:08:12.731762 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 13 04:08:12.731762 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 13 04:08:12.731762 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 13 04:08:12.731762 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 13 04:08:12.731762 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application return self.get_or_create( Oct 13 04:08:12.731762 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 13 04:08:12.731762 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application with Lock( Oct 13 04:08:12.731762 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 13 04:08:12.731762 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application return self._enter() Oct 13 04:08:12.731762 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 13 04:08:12.731762 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 13 04:08:12.738125 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 13 04:08:12.738125 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application return self.creator() Oct 13 04:08:12.738125 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 13 04:08:12.738125 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application created_value = creator( Oct 13 04:08:12.738125 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 13 04:08:12.738125 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 13 04:08:12.738125 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 13 04:08:12.738125 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 13 04:08:12.738125 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 13 04:08:12.738125 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 13 04:08:12.738125 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Oct 13 04:08:12.738125 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application Oct 13 04:08:12.738125 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 52/101] 10.222.0.27 () {56 vars in 1096 bytes} [Thu Oct 13 04:08:12 2022] GET /identity/v3/roles/admin => generated 83 bytes in 180 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 13 04:08:12.738125 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d5f553a0-a33e-49dc-b7b6-9eb5f88febd4 None admin] Authenticating user token {{(pid=81409) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:08:12.762247 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 50/102] 10.222.0.27 () {60 vars in 1141 bytes} [Thu Oct 13 04:08:12 2022] POST /identity/v3/projects => generated 302 bytes in 207 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 13 04:08:12.762247 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d65b90d0-0f9f-4200-a2eb-01b96a222d95 None admin] Authenticating user token {{(pid=81408) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:08:12.863930 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d5f553a0-a33e-49dc-b7b6-9eb5f88febd4 None admin] Validating token access rules against request {{(pid=81409) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:08:12.877591 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d65b90d0-0f9f-4200-a2eb-01b96a222d95 None admin] Validating token access rules against request {{(pid=81408) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:08:12.902069 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-58707302-a3b2-40e4-8ddb-5cc80b99b1cd None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81409) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:08:12.902772 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58707302-a3b2-40e4-8ddb-5cc80b99b1cd None admin] REQUEST_METHOD: `PUT` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:08:12.903050 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58707302-a3b2-40e4-8ddb-5cc80b99b1cd None admin] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:08:12.903329 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58707302-a3b2-40e4-8ddb-5cc80b99b1cd None admin] PATH_INFO: `/v3/projects/eeec419e34e44c1f881179933c828d97/users/4cba1177c8534417a4db1df4fe654898/roles/2306a441792b49f19af2d869acf5b3b6` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:08:12.905393 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-58707302-a3b2-40e4-8ddb-5cc80b99b1cd None admin] RBAC: Authorizing `identity:create_grant(project_id=eeec419e34e44c1f881179933c828d97, user_id=4cba1177c8534417a4db1df4fe654898, role_id=2306a441792b49f19af2d869acf5b3b6)` {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:08:12.906922 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-58707302-a3b2-40e4-8ddb-5cc80b99b1cd None admin] RBAC: Authorization granted {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:08:12.916666 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-545938d9-1c48-4894-9168-76c5052588ad None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81408) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:08:12.917299 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-545938d9-1c48-4894-9168-76c5052588ad None admin] REQUEST_METHOD: `GET` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:08:12.917588 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-545938d9-1c48-4894-9168-76c5052588ad None admin] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:08:12.917868 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-545938d9-1c48-4894-9168-76c5052588ad None admin] PATH_INFO: `/v3/roles` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:08:12.918591 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-545938d9-1c48-4894-9168-76c5052588ad None admin] RBAC: Adding query filter params (name=member) {{(pid=81408) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 13 04:08:12.918900 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-545938d9-1c48-4894-9168-76c5052588ad None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:08:12.920354 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-545938d9-1c48-4894-9168-76c5052588ad None admin] RBAC: Authorization granted {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:08:12.928948 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 53/103] 10.222.0.27 () {58 vars in 1439 bytes} [Thu Oct 13 04:08:12 2022] PUT /identity/v3/projects/eeec419e34e44c1f881179933c828d97/users/4cba1177c8534417a4db1df4fe654898/roles/2306a441792b49f19af2d869acf5b3b6 => generated 0 bytes in 188 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 13 04:08:12.928948 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4bae59b7-987b-4e94-9c50-2bf41cb50a88 None None] REQUEST_METHOD: `GET` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:08:12.928948 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4bae59b7-987b-4e94-9c50-2bf41cb50a88 None None] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:08:12.928948 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4bae59b7-987b-4e94-9c50-2bf41cb50a88 None None] PATH_INFO: `/` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:08:12.928948 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 54/104] 10.222.0.27 () {54 vars in 927 bytes} [Thu Oct 13 04:08:12 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 13 04:08:12.928948 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4bae59b7-987b-4e94-9c50-2bf41cb50a88 None None] Authenticating user token {{(pid=81409) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:08:12.939708 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 51/105] 10.222.0.27 () {56 vars in 1101 bytes} [Thu Oct 13 04:08:12 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 182 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 13 04:08:12.941038 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e73718e-4483-417c-b796-07e44ce4ba72 None None] REQUEST_METHOD: `GET` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:08:12.941324 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e73718e-4483-417c-b796-07e44ce4ba72 None None] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:08:12.941648 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e73718e-4483-417c-b796-07e44ce4ba72 None None] PATH_INFO: `/` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:08:12.943224 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 52/106] 10.222.0.27 () {54 vars in 927 bytes} [Thu Oct 13 04:08:12 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 13 04:08:12.944291 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d256143-2826-4541-99ca-325c7697614f None None] REQUEST_METHOD: `POST` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:08:12.944552 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d256143-2826-4541-99ca-325c7697614f None None] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:08:12.944837 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d256143-2826-4541-99ca-325c7697614f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:08:12.959798 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG dbcounter [-] [81409] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81409) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:08:12.971442 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.auth.core [None req-6d256143-2826-4541-99ca-325c7697614f None None] MFA Rules not processed for user `0e2d979563d6421995451a0e86bf3022`. Rule list: `[]` (Enabled: `True`). {{(pid=81408) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 13 04:08:12.973655 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.fernet_utils [None req-6d256143-2826-4541-99ca-325c7697614f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81408) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:08:13.016939 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG dbcounter [-] [81408] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81408) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:08:13.071971 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 53/107] 10.222.0.27 () {58 vars in 1024 bytes} [Thu Oct 13 04:08:12 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 128 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 13 04:08:13.073044 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6d256143-2826-4541-99ca-325c7697614f None None] Authenticating user token {{(pid=81408) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:08:13.079535 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4bae59b7-987b-4e94-9c50-2bf41cb50a88 None None] Validating token access rules against request {{(pid=81409) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:08:13.129423 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-16b65e62-7e55-4d59-8e57-69ea3941e2af None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81409) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:08:13.130322 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16b65e62-7e55-4d59-8e57-69ea3941e2af None admin] REQUEST_METHOD: `GET` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:08:13.130748 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16b65e62-7e55-4d59-8e57-69ea3941e2af None admin] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:08:13.131424 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16b65e62-7e55-4d59-8e57-69ea3941e2af None admin] PATH_INFO: `/v3/roles` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:08:13.132649 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-16b65e62-7e55-4d59-8e57-69ea3941e2af None admin] RBAC: Adding query filter params (name=admin) {{(pid=81409) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 13 04:08:13.133104 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-16b65e62-7e55-4d59-8e57-69ea3941e2af None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:08:13.135443 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-16b65e62-7e55-4d59-8e57-69ea3941e2af None admin] RBAC: Authorization granted {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:08:13.147298 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 55/108] 10.222.0.27 () {56 vars in 1099 bytes} [Thu Oct 13 04:08:12 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 222 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 13 04:08:13.148849 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb9dc6e4-bf00-45d5-8eff-46581f49a5c3 None None] REQUEST_METHOD: `POST` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:08:13.149842 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb9dc6e4-bf00-45d5-8eff-46581f49a5c3 None None] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:08:13.150462 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb9dc6e4-bf00-45d5-8eff-46581f49a5c3 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:08:13.170985 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.auth.core [None req-cb9dc6e4-bf00-45d5-8eff-46581f49a5c3 None None] MFA Rules not processed for user `0e2d979563d6421995451a0e86bf3022`. Rule list: `[]` (Enabled: `True`). {{(pid=81409) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 13 04:08:13.172465 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.fernet_utils [None req-cb9dc6e4-bf00-45d5-8eff-46581f49a5c3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81409) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:08:13.199391 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6d256143-2826-4541-99ca-325c7697614f None None] Validating token access rules against request {{(pid=81408) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:08:13.248437 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-63e5a5d3-f7c1-4770-82e1-2b6f8a541708 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81408) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:08:13.249234 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63e5a5d3-f7c1-4770-82e1-2b6f8a541708 None admin] REQUEST_METHOD: `GET` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:08:13.249635 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63e5a5d3-f7c1-4770-82e1-2b6f8a541708 None admin] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:08:13.249918 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63e5a5d3-f7c1-4770-82e1-2b6f8a541708 None admin] PATH_INFO: `/v3/users/761e23aab9234a7c9c88dc41d2ddbe8e` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:08:13.251781 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-63e5a5d3-f7c1-4770-82e1-2b6f8a541708 None admin] RBAC: Authorizing `identity:get_user(user_id=761e23aab9234a7c9c88dc41d2ddbe8e)` {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:08:13.253874 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-63e5a5d3-f7c1-4770-82e1-2b6f8a541708 None admin] RBAC: Authorization granted {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:08:13.256206 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 54/109] 10.222.0.27 () {56 vars in 1177 bytes} [Thu Oct 13 04:08:13 2022] GET /identity/v3/users/761e23aab9234a7c9c88dc41d2ddbe8e => generated 275 bytes in 184 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 13 04:08:13.257248 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-63e5a5d3-f7c1-4770-82e1-2b6f8a541708 None admin] Authenticating user token {{(pid=81408) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:08:13.268001 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 56/110] 10.222.0.27 () {58 vars in 1024 bytes} [Thu Oct 13 04:08:13 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 120 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 13 04:08:13.268759 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cb9dc6e4-bf00-45d5-8eff-46581f49a5c3 None None] Authenticating user token {{(pid=81409) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:08:13.299142 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG dbcounter [-] [81409] Writing DB stats keystone:SELECT=100 {{(pid=81409) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:08:13.311055 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG dbcounter [-] [81408] Writing DB stats keystone:SELECT=100 {{(pid=81408) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:08:13.435417 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-63e5a5d3-f7c1-4770-82e1-2b6f8a541708 None admin] Validating token access rules against request {{(pid=81408) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:08:13.450246 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cb9dc6e4-bf00-45d5-8eff-46581f49a5c3 None None] Validating token access rules against request {{(pid=81409) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:08:13.503904 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b7afb343-7723-4866-a61b-718ceca5e140 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81408) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:08:13.504820 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7afb343-7723-4866-a61b-718ceca5e140 None admin] REQUEST_METHOD: `GET` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:08:13.505245 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7afb343-7723-4866-a61b-718ceca5e140 None admin] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:08:13.505662 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7afb343-7723-4866-a61b-718ceca5e140 None admin] PATH_INFO: `/v3/users/0e2d979563d6421995451a0e86bf3022` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:08:13.507667 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b7afb343-7723-4866-a61b-718ceca5e140 None admin] RBAC: Authorizing `identity:get_user(user_id=0e2d979563d6421995451a0e86bf3022)` {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:08:13.510763 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b7afb343-7723-4866-a61b-718ceca5e140 None admin] RBAC: Authorization granted {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:08:13.514364 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 55/111] 10.222.0.27 () {56 vars in 1177 bytes} [Thu Oct 13 04:08:13 2022] GET /identity/v3/users/0e2d979563d6421995451a0e86bf3022 => generated 247 bytes in 257 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 13 04:08:13.515612 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b7afb343-7723-4866-a61b-718ceca5e140 None admin] Authenticating user token {{(pid=81408) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:08:13.517152 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-57b9680a-af1a-4778-bc06-5da497cb58b7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81409) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:08:13.518072 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57b9680a-af1a-4778-bc06-5da497cb58b7 None admin] REQUEST_METHOD: `GET` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:08:13.518512 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57b9680a-af1a-4778-bc06-5da497cb58b7 None admin] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:08:13.518944 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57b9680a-af1a-4778-bc06-5da497cb58b7 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:08:13.625438 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-57b9680a-af1a-4778-bc06-5da497cb58b7 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:08:13.627147 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-57b9680a-af1a-4778-bc06-5da497cb58b7 None admin] RBAC: Authorization granted {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:08:13.653784 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG dbcounter [-] [81409] Writing DB stats keystone:SELECT=100 {{(pid=81409) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:08:13.659679 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG dbcounter [-] [81408] Writing DB stats keystone:SELECT=100 {{(pid=81408) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:08:13.663491 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b7afb343-7723-4866-a61b-718ceca5e140 None admin] Validating token access rules against request {{(pid=81408) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:08:13.701248 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-68b35854-6c09-4cde-aa49-4cc4a7236d2d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81408) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:08:13.702125 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68b35854-6c09-4cde-aa49-4cc4a7236d2d None admin] REQUEST_METHOD: `GET` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:08:13.702575 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68b35854-6c09-4cde-aa49-4cc4a7236d2d None admin] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:08:13.703058 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68b35854-6c09-4cde-aa49-4cc4a7236d2d None admin] PATH_INFO: `/v3/roles/anotherrole` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:08:13.714631 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-68b35854-6c09-4cde-aa49-4cc4a7236d2d None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 13 04:08:13.719904 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application [None req-68b35854-6c09-4cde-aa49-4cc4a7236d2d None admin] Could not find role: anotherrole.: keystone.exception.RoleNotFound: Could not find role: anotherrole. Oct 13 04:08:13.719904 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 13 04:08:13.719904 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 13 04:08:13.719904 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 13 04:08:13.719904 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 13 04:08:13.719904 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 13 04:08:13.719904 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 13 04:08:13.719904 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 13 04:08:13.719904 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 13 04:08:13.719904 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 13 04:08:13.719904 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 13 04:08:13.719904 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 13 04:08:13.719904 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 13 04:08:13.719904 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 13 04:08:13.719904 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 13 04:08:13.719904 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 13 04:08:13.720727 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 13 04:08:13.720727 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 13 04:08:13.720727 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 13 04:08:13.720727 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 13 04:08:13.720727 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 13 04:08:13.720727 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 13 04:08:13.720727 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 13 04:08:13.720727 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 13 04:08:13.720727 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 13 04:08:13.720727 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application return self.get_or_create( Oct 13 04:08:13.720727 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 13 04:08:13.720727 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application with Lock( Oct 13 04:08:13.720727 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 13 04:08:13.720727 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application return self._enter() Oct 13 04:08:13.720727 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 13 04:08:13.720727 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 13 04:08:13.721493 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 13 04:08:13.721493 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application return self.creator() Oct 13 04:08:13.721493 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 13 04:08:13.721493 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application created_value = creator( Oct 13 04:08:13.721493 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 13 04:08:13.721493 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 13 04:08:13.721493 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 13 04:08:13.721493 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 13 04:08:13.721493 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 13 04:08:13.721493 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 13 04:08:13.721493 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: anotherrole. Oct 13 04:08:13.721493 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application Oct 13 04:08:13.729149 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 56/112] 10.222.0.27 () {56 vars in 1114 bytes} [Thu Oct 13 04:08:13 2022] GET /identity/v3/roles/anotherrole => generated 89 bytes in 209 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 13 04:08:13.730328 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-68b35854-6c09-4cde-aa49-4cc4a7236d2d None admin] Authenticating user token {{(pid=81408) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:08:13.746060 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 57/113] 10.222.0.27 () {58 vars in 1282 bytes} [Thu Oct 13 04:08:13 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 477 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Oct 13 04:08:13.746786 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-57b9680a-af1a-4778-bc06-5da497cb58b7 None admin] Authenticating user token {{(pid=81409) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:08:13.840452 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-68b35854-6c09-4cde-aa49-4cc4a7236d2d None admin] Validating token access rules against request {{(pid=81408) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:08:13.859902 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-57b9680a-af1a-4778-bc06-5da497cb58b7 None admin] Validating token access rules against request {{(pid=81409) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:08:13.878730 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG dbcounter [-] [81409] Writing DB stats keystone:SELECT=100 {{(pid=81409) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:08:13.881656 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f6d56a5e-7f2b-4975-8dd2-c49f37d118e3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81408) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:08:13.882872 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6d56a5e-7f2b-4975-8dd2-c49f37d118e3 None admin] REQUEST_METHOD: `GET` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:08:13.883396 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6d56a5e-7f2b-4975-8dd2-c49f37d118e3 None admin] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:08:13.884150 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6d56a5e-7f2b-4975-8dd2-c49f37d118e3 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:08:13.902183 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6a4e5207-b4ee-4b6f-979e-8a41cecf8e57 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81409) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:08:13.902882 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a4e5207-b4ee-4b6f-979e-8a41cecf8e57 None admin] REQUEST_METHOD: `GET` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:08:13.903337 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a4e5207-b4ee-4b6f-979e-8a41cecf8e57 None admin] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:08:13.903614 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a4e5207-b4ee-4b6f-979e-8a41cecf8e57 None admin] PATH_INFO: `/v3/roles/member` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:08:13.917737 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6a4e5207-b4ee-4b6f-979e-8a41cecf8e57 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 13 04:08:13.922500 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG dbcounter [-] [81408] Writing DB stats keystone:SELECT=100 {{(pid=81408) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:08:13.925525 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application [None req-6a4e5207-b4ee-4b6f-979e-8a41cecf8e57 None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Oct 13 04:08:13.925525 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 13 04:08:13.925525 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 13 04:08:13.925525 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 13 04:08:13.925525 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 13 04:08:13.925525 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 13 04:08:13.925525 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 13 04:08:13.925525 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 13 04:08:13.925525 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 13 04:08:13.925525 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 13 04:08:13.925525 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 13 04:08:13.925525 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 13 04:08:13.925525 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 13 04:08:13.925525 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 13 04:08:13.925525 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 13 04:08:13.925525 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 13 04:08:13.926446 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 13 04:08:13.926446 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 13 04:08:13.926446 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 13 04:08:13.926446 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 13 04:08:13.926446 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 13 04:08:13.926446 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 13 04:08:13.926446 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 13 04:08:13.926446 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 13 04:08:13.926446 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 13 04:08:13.926446 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application return self.get_or_create( Oct 13 04:08:13.926446 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 13 04:08:13.926446 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application with Lock( Oct 13 04:08:13.926446 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 13 04:08:13.926446 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application return self._enter() Oct 13 04:08:13.926446 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 13 04:08:13.926446 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 13 04:08:13.927236 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 13 04:08:13.927236 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application return self.creator() Oct 13 04:08:13.927236 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 13 04:08:13.927236 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application created_value = creator( Oct 13 04:08:13.927236 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 13 04:08:13.927236 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 13 04:08:13.927236 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 13 04:08:13.927236 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 13 04:08:13.927236 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 13 04:08:13.927236 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 13 04:08:13.927236 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Oct 13 04:08:13.927236 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application Oct 13 04:08:13.934356 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 58/114] 10.222.0.27 () {56 vars in 1099 bytes} [Thu Oct 13 04:08:13 2022] GET /identity/v3/roles/member => generated 84 bytes in 188 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 13 04:08:13.935459 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6a4e5207-b4ee-4b6f-979e-8a41cecf8e57 None admin] Authenticating user token {{(pid=81409) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:08:13.978951 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f6d56a5e-7f2b-4975-8dd2-c49f37d118e3 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:08:13.981257 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f6d56a5e-7f2b-4975-8dd2-c49f37d118e3 None admin] RBAC: Authorization granted {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:08:14.058415 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6a4e5207-b4ee-4b6f-979e-8a41cecf8e57 None admin] Validating token access rules against request {{(pid=81409) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:08:14.096049 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 57/115] 10.222.0.27 () {58 vars in 1282 bytes} [Thu Oct 13 04:08:13 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 366 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Oct 13 04:08:14.097400 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f6d56a5e-7f2b-4975-8dd2-c49f37d118e3 None admin] Authenticating user token {{(pid=81408) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:08:14.103057 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-38576f65-c0c3-4cc7-9fea-1d5c4cb66ba6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81409) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:08:14.103686 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38576f65-c0c3-4cc7-9fea-1d5c4cb66ba6 None admin] REQUEST_METHOD: `GET` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:08:14.103965 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38576f65-c0c3-4cc7-9fea-1d5c4cb66ba6 None admin] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:08:14.104236 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38576f65-c0c3-4cc7-9fea-1d5c4cb66ba6 None admin] PATH_INFO: `/v3/roles` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:08:14.104910 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-38576f65-c0c3-4cc7-9fea-1d5c4cb66ba6 None admin] RBAC: Adding query filter params (name=anotherrole) {{(pid=81409) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 13 04:08:14.105213 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-38576f65-c0c3-4cc7-9fea-1d5c4cb66ba6 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:08:14.106638 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-38576f65-c0c3-4cc7-9fea-1d5c4cb66ba6 None admin] RBAC: Authorization granted {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:08:14.119832 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 59/116] 10.222.0.27 () {56 vars in 1111 bytes} [Thu Oct 13 04:08:13 2022] GET /identity/v3/roles?name=anotherrole => generated 334 bytes in 180 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 13 04:08:14.119832 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-38576f65-c0c3-4cc7-9fea-1d5c4cb66ba6 None admin] Authenticating user token {{(pid=81409) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:08:14.160404 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG dbcounter [-] [81409] Writing DB stats keystone:SELECT=100 {{(pid=81409) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:08:14.179367 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG dbcounter [-] [81408] Writing DB stats keystone:SELECT=100 {{(pid=81408) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:08:14.231599 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f6d56a5e-7f2b-4975-8dd2-c49f37d118e3 None admin] Validating token access rules against request {{(pid=81408) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:08:14.241703 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-38576f65-c0c3-4cc7-9fea-1d5c4cb66ba6 None admin] Validating token access rules against request {{(pid=81409) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:08:14.275955 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-859c1aac-e4c2-4dc5-88f5-069a4fd019f1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81408) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:08:14.276758 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-859c1aac-e4c2-4dc5-88f5-069a4fd019f1 None admin] REQUEST_METHOD: `GET` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:08:14.277053 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-859c1aac-e4c2-4dc5-88f5-069a4fd019f1 None admin] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:08:14.277332 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-859c1aac-e4c2-4dc5-88f5-069a4fd019f1 None admin] PATH_INFO: `/v3/projects/eeec419e34e44c1f881179933c828d97` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:08:14.278822 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-859c1aac-e4c2-4dc5-88f5-069a4fd019f1 None admin] RBAC: Authorizing `identity:get_project(project_id=eeec419e34e44c1f881179933c828d97)` {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:08:14.280503 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-859c1aac-e4c2-4dc5-88f5-069a4fd019f1 None admin] RBAC: Authorization granted {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:08:14.281082 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4af81c04-371c-4329-a5ad-3b819dc6a0ed None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81409) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:08:14.281919 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4af81c04-371c-4329-a5ad-3b819dc6a0ed None admin] REQUEST_METHOD: `GET` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:08:14.282389 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 58/117] 10.222.0.27 () {56 vars in 1186 bytes} [Thu Oct 13 04:08:14 2022] GET /identity/v3/projects/eeec419e34e44c1f881179933c828d97 => generated 298 bytes in 185 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 13 04:08:14.282909 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4af81c04-371c-4329-a5ad-3b819dc6a0ed None admin] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:08:14.283011 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-859c1aac-e4c2-4dc5-88f5-069a4fd019f1 None admin] Authenticating user token {{(pid=81408) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:08:14.283541 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4af81c04-371c-4329-a5ad-3b819dc6a0ed None admin] PATH_INFO: `/v3/roles` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:08:14.284402 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4af81c04-371c-4329-a5ad-3b819dc6a0ed None admin] RBAC: Adding query filter params (name=member) {{(pid=81409) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 13 04:08:14.284903 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4af81c04-371c-4329-a5ad-3b819dc6a0ed None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:08:14.286603 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4af81c04-371c-4329-a5ad-3b819dc6a0ed None admin] RBAC: Authorization granted {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:08:14.296524 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 60/118] 10.222.0.27 () {56 vars in 1101 bytes} [Thu Oct 13 04:08:14 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 180 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 13 04:08:14.297639 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4af81c04-371c-4329-a5ad-3b819dc6a0ed None admin] Authenticating user token {{(pid=81409) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:08:14.413517 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG dbcounter [-] [81409] Writing DB stats keystone:SELECT=100 {{(pid=81409) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:08:14.413671 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-859c1aac-e4c2-4dc5-88f5-069a4fd019f1 None admin] Validating token access rules against request {{(pid=81408) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:08:14.421054 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4af81c04-371c-4329-a5ad-3b819dc6a0ed None admin] Validating token access rules against request {{(pid=81409) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:08:14.446052 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG dbcounter [-] [81408] Writing DB stats keystone:SELECT=100 {{(pid=81408) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:08:14.458622 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-49de1eea-2680-4260-9838-f19b582d240e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81408) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:08:14.460237 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49de1eea-2680-4260-9838-f19b582d240e None admin] REQUEST_METHOD: `GET` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:08:14.460726 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49de1eea-2680-4260-9838-f19b582d240e None admin] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:08:14.461201 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49de1eea-2680-4260-9838-f19b582d240e None admin] PATH_INFO: `/v3/projects/eeec419e34e44c1f881179933c828d97` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:08:14.463136 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-49de1eea-2680-4260-9838-f19b582d240e None admin] RBAC: Authorizing `identity:get_project(project_id=eeec419e34e44c1f881179933c828d97)` {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:08:14.464686 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-26a42f24-2ef7-4f22-90e5-d7ec0419bbf7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81409) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:08:14.465072 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-49de1eea-2680-4260-9838-f19b582d240e None admin] RBAC: Authorization granted {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:08:14.465323 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26a42f24-2ef7-4f22-90e5-d7ec0419bbf7 None admin] REQUEST_METHOD: `GET` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:08:14.465605 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26a42f24-2ef7-4f22-90e5-d7ec0419bbf7 None admin] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:08:14.465882 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26a42f24-2ef7-4f22-90e5-d7ec0419bbf7 None admin] PATH_INFO: `/v3/users/761e23aab9234a7c9c88dc41d2ddbe8e` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:08:14.467052 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-26a42f24-2ef7-4f22-90e5-d7ec0419bbf7 None admin] RBAC: Authorizing `identity:get_user(user_id=761e23aab9234a7c9c88dc41d2ddbe8e)` {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:08:14.467804 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 59/119] 10.222.0.27 () {56 vars in 1186 bytes} [Thu Oct 13 04:08:14 2022] GET /identity/v3/projects/eeec419e34e44c1f881179933c828d97 => generated 298 bytes in 185 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 13 04:08:14.468955 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-49de1eea-2680-4260-9838-f19b582d240e None admin] Authenticating user token {{(pid=81408) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:08:14.473699 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-26a42f24-2ef7-4f22-90e5-d7ec0419bbf7 None admin] RBAC: Authorization granted {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:08:14.479881 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 61/120] 10.222.0.27 () {56 vars in 1177 bytes} [Thu Oct 13 04:08:14 2022] GET /identity/v3/users/761e23aab9234a7c9c88dc41d2ddbe8e => generated 275 bytes in 178 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 13 04:08:14.479881 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-26a42f24-2ef7-4f22-90e5-d7ec0419bbf7 None admin] Authenticating user token {{(pid=81409) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:08:14.584810 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-26a42f24-2ef7-4f22-90e5-d7ec0419bbf7 None admin] Validating token access rules against request {{(pid=81409) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:08:14.617561 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-49de1eea-2680-4260-9838-f19b582d240e None admin] Validating token access rules against request {{(pid=81408) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:08:14.622459 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e331872b-8dd8-4503-8d8a-01257bce469f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81409) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:08:14.623097 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e331872b-8dd8-4503-8d8a-01257bce469f None admin] REQUEST_METHOD: `GET` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:08:14.623381 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e331872b-8dd8-4503-8d8a-01257bce469f None admin] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:08:14.623653 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e331872b-8dd8-4503-8d8a-01257bce469f None admin] PATH_INFO: `/v3/users/761e23aab9234a7c9c88dc41d2ddbe8e` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:08:14.624783 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e331872b-8dd8-4503-8d8a-01257bce469f None admin] RBAC: Authorizing `identity:get_user(user_id=761e23aab9234a7c9c88dc41d2ddbe8e)` {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:08:14.626285 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e331872b-8dd8-4503-8d8a-01257bce469f None admin] RBAC: Authorization granted {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:08:14.628068 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 62/121] 10.222.0.27 () {56 vars in 1177 bytes} [Thu Oct 13 04:08:14 2022] GET /identity/v3/users/761e23aab9234a7c9c88dc41d2ddbe8e => generated 275 bytes in 152 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 13 04:08:14.628835 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e331872b-8dd8-4503-8d8a-01257bce469f None admin] Authenticating user token {{(pid=81409) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:08:14.665629 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cbe60bda-cd51-415d-85d1-1ea77a8c981f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81408) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:08:14.667245 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cbe60bda-cd51-415d-85d1-1ea77a8c981f None admin] REQUEST_METHOD: `GET` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:08:14.667732 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cbe60bda-cd51-415d-85d1-1ea77a8c981f None admin] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:08:14.668209 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cbe60bda-cd51-415d-85d1-1ea77a8c981f None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:08:14.669235 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG dbcounter [-] [81409] Writing DB stats keystone:SELECT=100 {{(pid=81409) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:08:14.669818 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cbe60bda-cd51-415d-85d1-1ea77a8c981f None admin] RBAC: Adding query filter params (role.id=cc56c4ae13a7424ea771c15da4b153c1, scope.project.id=eeec419e34e44c1f881179933c828d97, user.id=761e23aab9234a7c9c88dc41d2ddbe8e) {{(pid=81408) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 13 04:08:14.671132 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cbe60bda-cd51-415d-85d1-1ea77a8c981f None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:08:14.672866 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cbe60bda-cd51-415d-85d1-1ea77a8c981f None admin] RBAC: Authorization granted {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:08:14.681499 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 60/122] 10.222.0.27 () {56 vars in 1374 bytes} [Thu Oct 13 04:08:14 2022] GET /identity/v3/role_assignments?user.id=761e23aab9234a7c9c88dc41d2ddbe8e&scope.project.id=eeec419e34e44c1f881179933c828d97&role.id=cc56c4ae13a7424ea771c15da4b153c1 => generated 608 bytes in 213 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 13 04:08:14.682939 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cbe60bda-cd51-415d-85d1-1ea77a8c981f None admin] Authenticating user token {{(pid=81408) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:08:14.751000 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e331872b-8dd8-4503-8d8a-01257bce469f None admin] Validating token access rules against request {{(pid=81409) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:08:14.772290 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG dbcounter [-] [81408] Writing DB stats keystone:SELECT=100 {{(pid=81408) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:08:14.823055 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cecc8780-0099-4430-a68a-5c9a7fc17fe4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81409) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:08:14.823722 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cecc8780-0099-4430-a68a-5c9a7fc17fe4 None admin] REQUEST_METHOD: `GET` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:08:14.824389 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cecc8780-0099-4430-a68a-5c9a7fc17fe4 None admin] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:08:14.824389 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cecc8780-0099-4430-a68a-5c9a7fc17fe4 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:08:14.825160 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cecc8780-0099-4430-a68a-5c9a7fc17fe4 None admin] RBAC: Adding query filter params (role.id=ef55fbd94e1040be83695754075a7ed5, scope.project.id=eeec419e34e44c1f881179933c828d97, user.id=0e2d979563d6421995451a0e86bf3022) {{(pid=81409) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 13 04:08:14.825475 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cecc8780-0099-4430-a68a-5c9a7fc17fe4 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:08:14.827029 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cecc8780-0099-4430-a68a-5c9a7fc17fe4 None admin] RBAC: Authorization granted {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:08:14.831813 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cbe60bda-cd51-415d-85d1-1ea77a8c981f None admin] Validating token access rules against request {{(pid=81408) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:08:14.845534 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 63/123] 10.222.0.27 () {56 vars in 1374 bytes} [Thu Oct 13 04:08:14 2022] GET /identity/v3/role_assignments?user.id=0e2d979563d6421995451a0e86bf3022&scope.project.id=eeec419e34e44c1f881179933c828d97&role.id=ef55fbd94e1040be83695754075a7ed5 => generated 608 bytes in 217 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 13 04:08:14.846464 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cecc8780-0099-4430-a68a-5c9a7fc17fe4 None admin] Authenticating user token {{(pid=81409) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:08:14.891103 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0d5a85cd-03bc-480f-9066-b52d831a855e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81408) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:08:14.892143 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d5a85cd-03bc-480f-9066-b52d831a855e None admin] REQUEST_METHOD: `GET` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:08:14.892648 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d5a85cd-03bc-480f-9066-b52d831a855e None admin] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:08:14.893144 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d5a85cd-03bc-480f-9066-b52d831a855e None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:08:14.990176 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0d5a85cd-03bc-480f-9066-b52d831a855e None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:08:14.992477 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0d5a85cd-03bc-480f-9066-b52d831a855e None admin] RBAC: Authorization granted {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:08:15.064514 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG dbcounter [-] [81408] Writing DB stats keystone:SELECT=100 {{(pid=81408) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:08:15.116928 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 61/124] 10.222.0.27 () {58 vars in 1282 bytes} [Thu Oct 13 04:08:14 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 435 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Oct 13 04:08:15.125774 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0d5a85cd-03bc-480f-9066-b52d831a855e None admin] Authenticating user token {{(pid=81408) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:08:15.154042 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG dbcounter [-] [81409] Writing DB stats keystone:SELECT=100 {{(pid=81409) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:08:15.157647 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cecc8780-0099-4430-a68a-5c9a7fc17fe4 None admin] Validating token access rules against request {{(pid=81409) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:08:15.201832 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7570312a-51e6-4dcf-bf88-e411b8f271e5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81409) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:08:15.202650 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7570312a-51e6-4dcf-bf88-e411b8f271e5 None admin] REQUEST_METHOD: `GET` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:08:15.202882 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7570312a-51e6-4dcf-bf88-e411b8f271e5 None admin] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:08:15.203159 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7570312a-51e6-4dcf-bf88-e411b8f271e5 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:08:15.261213 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0d5a85cd-03bc-480f-9066-b52d831a855e None admin] Validating token access rules against request {{(pid=81408) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:08:15.287795 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7570312a-51e6-4dcf-bf88-e411b8f271e5 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:08:15.289275 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7570312a-51e6-4dcf-bf88-e411b8f271e5 None admin] RBAC: Authorization granted {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:08:15.305047 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2e010b5c-85d8-4cb1-8921-68084acbceb1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81408) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:08:15.306245 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e010b5c-85d8-4cb1-8921-68084acbceb1 None admin] REQUEST_METHOD: `GET` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:08:15.306593 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e010b5c-85d8-4cb1-8921-68084acbceb1 None admin] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:08:15.306810 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e010b5c-85d8-4cb1-8921-68084acbceb1 None admin] PATH_INFO: `/v3/projects/eeec419e34e44c1f881179933c828d97` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:08:15.307958 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2e010b5c-85d8-4cb1-8921-68084acbceb1 None admin] RBAC: Authorizing `identity:get_project(project_id=eeec419e34e44c1f881179933c828d97)` {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:08:15.309432 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2e010b5c-85d8-4cb1-8921-68084acbceb1 None admin] RBAC: Authorization granted {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:08:15.311324 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 62/125] 10.222.0.27 () {56 vars in 1186 bytes} [Thu Oct 13 04:08:15 2022] GET /identity/v3/projects/eeec419e34e44c1f881179933c828d97 => generated 298 bytes in 186 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 13 04:08:15.316110 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f107e6f3-7f38-44ea-b246-965074299062 None None] REQUEST_METHOD: `GET` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:08:15.316110 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f107e6f3-7f38-44ea-b246-965074299062 None None] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:08:15.316110 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f107e6f3-7f38-44ea-b246-965074299062 None None] PATH_INFO: `/` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:08:15.316110 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 63/126] 10.222.0.27 () {54 vars in 927 bytes} [Thu Oct 13 04:08:15 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 13 04:08:15.319130 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f107e6f3-7f38-44ea-b246-965074299062 None None] Authenticating user token {{(pid=81408) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:08:15.348536 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG dbcounter [-] [81408] Writing DB stats keystone:SELECT=100 {{(pid=81408) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:08:15.410591 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG dbcounter [-] [81409] Writing DB stats keystone:SELECT=100 {{(pid=81409) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:08:15.418116 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 64/127] 10.222.0.27 () {58 vars in 1282 bytes} [Thu Oct 13 04:08:14 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 572 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Oct 13 04:08:15.419266 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f94c7beb-9926-4866-b531-67df0196ce9b None None] REQUEST_METHOD: `POST` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:08:15.419553 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f94c7beb-9926-4866-b531-67df0196ce9b None None] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:08:15.419830 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f94c7beb-9926-4866-b531-67df0196ce9b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:08:15.438340 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.auth.core [None req-f94c7beb-9926-4866-b531-67df0196ce9b None None] MFA Rules not processed for user `0e2d979563d6421995451a0e86bf3022`. Rule list: `[]` (Enabled: `True`). {{(pid=81409) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 13 04:08:15.439551 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.fernet_utils [None req-f94c7beb-9926-4866-b531-67df0196ce9b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81409) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:08:15.443285 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f107e6f3-7f38-44ea-b246-965074299062 None None] Validating token access rules against request {{(pid=81408) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:08:15.481190 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-43d630e7-7fb7-4e68-a46b-41806eefd7ac None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81408) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:08:15.481824 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43d630e7-7fb7-4e68-a46b-41806eefd7ac None admin] REQUEST_METHOD: `GET` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:08:15.482105 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43d630e7-7fb7-4e68-a46b-41806eefd7ac None admin] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:08:15.482417 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43d630e7-7fb7-4e68-a46b-41806eefd7ac None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:08:15.483173 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-43d630e7-7fb7-4e68-a46b-41806eefd7ac None admin] RBAC: Adding query filter params (role.id=743995e93fef47fd91204c3d701054b0, scope.project.id=eeec419e34e44c1f881179933c828d97, user.id=761e23aab9234a7c9c88dc41d2ddbe8e) {{(pid=81408) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 13 04:08:15.483488 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-43d630e7-7fb7-4e68-a46b-41806eefd7ac None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:08:15.485002 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-43d630e7-7fb7-4e68-a46b-41806eefd7ac None admin] RBAC: Authorization granted {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:08:15.491683 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 64/128] 10.222.0.27 () {56 vars in 1374 bytes} [Thu Oct 13 04:08:15 2022] GET /identity/v3/role_assignments?user.id=761e23aab9234a7c9c88dc41d2ddbe8e&scope.project.id=eeec419e34e44c1f881179933c828d97&role.id=743995e93fef47fd91204c3d701054b0 => generated 608 bytes in 173 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 13 04:08:15.492455 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-43d630e7-7fb7-4e68-a46b-41806eefd7ac None admin] Authenticating user token {{(pid=81408) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:08:15.535188 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 65/129] 10.222.0.27 () {58 vars in 1024 bytes} [Thu Oct 13 04:08:15 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 116 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 13 04:08:15.586658 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ff0129b-d303-4f68-ae9b-a4dacc458529 None None] REQUEST_METHOD: `GET` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:08:15.586947 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ff0129b-d303-4f68-ae9b-a4dacc458529 None None] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:08:15.587229 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ff0129b-d303-4f68-ae9b-a4dacc458529 None None] PATH_INFO: `/` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:08:15.588644 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 66/130] 10.222.0.27 () {54 vars in 927 bytes} [Thu Oct 13 04:08:15 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Oct 13 04:08:15.594283 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2e1af54-4b34-48db-bd89-619428be947a None None] REQUEST_METHOD: `POST` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:08:15.594572 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2e1af54-4b34-48db-bd89-619428be947a None None] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:08:15.594851 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2e1af54-4b34-48db-bd89-619428be947a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:08:15.614587 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.auth.core [None req-a2e1af54-4b34-48db-bd89-619428be947a None None] MFA Rules not processed for user `0e2d979563d6421995451a0e86bf3022`. Rule list: `[]` (Enabled: `True`). {{(pid=81409) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 13 04:08:15.615806 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.fernet_utils [None req-a2e1af54-4b34-48db-bd89-619428be947a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81409) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:08:15.622009 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG dbcounter [-] [81408] Writing DB stats keystone:SELECT=100 {{(pid=81408) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:08:15.636528 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-43d630e7-7fb7-4e68-a46b-41806eefd7ac None admin] Validating token access rules against request {{(pid=81408) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:08:15.680864 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ac76bbbd-b5df-474e-b539-d75f8b287e87 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81408) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:08:15.681500 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac76bbbd-b5df-474e-b539-d75f8b287e87 None admin] REQUEST_METHOD: `GET` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:08:15.681831 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac76bbbd-b5df-474e-b539-d75f8b287e87 None admin] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:08:15.682117 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac76bbbd-b5df-474e-b539-d75f8b287e87 None admin] PATH_INFO: `/v3/projects/fb592092172d4c10b772d5c6d2dc9c62` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:08:15.684057 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ac76bbbd-b5df-474e-b539-d75f8b287e87 None admin] RBAC: Authorizing `identity:get_project(project_id=fb592092172d4c10b772d5c6d2dc9c62)` {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:08:15.685521 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ac76bbbd-b5df-474e-b539-d75f8b287e87 None admin] RBAC: Authorization granted {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:08:15.688177 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 65/131] 10.222.0.27 () {56 vars in 1186 bytes} [Thu Oct 13 04:08:15 2022] GET /identity/v3/projects/fb592092172d4c10b772d5c6d2dc9c62 => generated 312 bytes in 196 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 13 04:08:15.693919 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ac76bbbd-b5df-474e-b539-d75f8b287e87 None admin] Authenticating user token {{(pid=81408) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:08:15.708665 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 67/132] 10.222.0.27 () {58 vars in 1024 bytes} [Thu Oct 13 04:08:15 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 115 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Oct 13 04:08:15.805998 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ac76bbbd-b5df-474e-b539-d75f8b287e87 None admin] Validating token access rules against request {{(pid=81408) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:08:15.811541 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a2e1af54-4b34-48db-bd89-619428be947a None None] Authenticating user token {{(pid=81409) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:08:15.843117 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-53ab3198-7307-40d6-9b66-84c270fb98f5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81408) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:08:15.843727 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53ab3198-7307-40d6-9b66-84c270fb98f5 None admin] REQUEST_METHOD: `GET` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:08:15.844015 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53ab3198-7307-40d6-9b66-84c270fb98f5 None admin] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:08:15.844292 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53ab3198-7307-40d6-9b66-84c270fb98f5 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:08:15.845029 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-53ab3198-7307-40d6-9b66-84c270fb98f5 None admin] RBAC: Adding query filter params (role.id=cc56c4ae13a7424ea771c15da4b153c1, scope.project.id=fb592092172d4c10b772d5c6d2dc9c62, user.id=761e23aab9234a7c9c88dc41d2ddbe8e) {{(pid=81408) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 13 04:08:15.845335 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-53ab3198-7307-40d6-9b66-84c270fb98f5 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:08:15.846822 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-53ab3198-7307-40d6-9b66-84c270fb98f5 None admin] RBAC: Authorization granted {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:08:15.853550 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 66/133] 10.222.0.27 () {56 vars in 1374 bytes} [Thu Oct 13 04:08:15 2022] GET /identity/v3/role_assignments?user.id=761e23aab9234a7c9c88dc41d2ddbe8e&scope.project.id=fb592092172d4c10b772d5c6d2dc9c62&role.id=cc56c4ae13a7424ea771c15da4b153c1 => generated 608 bytes in 160 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 13 04:08:15.885227 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG dbcounter [-] [81409] Writing DB stats keystone:SELECT=100 {{(pid=81409) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:08:15.933506 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a2e1af54-4b34-48db-bd89-619428be947a None None] Validating token access rules against request {{(pid=81409) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:08:15.970539 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-39aab07d-56a4-4be0-b4a1-6ec63a96d7d8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81409) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:08:15.971198 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39aab07d-56a4-4be0-b4a1-6ec63a96d7d8 None admin] REQUEST_METHOD: `GET` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:08:15.971481 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39aab07d-56a4-4be0-b4a1-6ec63a96d7d8 None admin] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:08:15.971757 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39aab07d-56a4-4be0-b4a1-6ec63a96d7d8 None admin] PATH_INFO: `/v3/domains/default` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:08:15.972944 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-39aab07d-56a4-4be0-b4a1-6ec63a96d7d8 None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:08:15.974690 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-39aab07d-56a4-4be0-b4a1-6ec63a96d7d8 None admin] RBAC: Authorization granted {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:08:15.976354 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 68/134] 10.222.0.27 () {56 vars in 1108 bytes} [Thu Oct 13 04:08:15 2022] GET /identity/v3/domains/default => generated 199 bytes in 165 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 13 04:08:15.982913 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-53ab3198-7307-40d6-9b66-84c270fb98f5 None admin] Authenticating user token {{(pid=81408) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:08:15.985027 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-39aab07d-56a4-4be0-b4a1-6ec63a96d7d8 None admin] Authenticating user token {{(pid=81409) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:08:16.010136 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG dbcounter [-] [81408] Writing DB stats keystone:SELECT=100 {{(pid=81408) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:08:16.099177 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-39aab07d-56a4-4be0-b4a1-6ec63a96d7d8 None admin] Validating token access rules against request {{(pid=81409) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:08:16.103352 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-53ab3198-7307-40d6-9b66-84c270fb98f5 None admin] Validating token access rules against request {{(pid=81408) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:08:16.124917 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG dbcounter [-] [81409] Writing DB stats keystone:SELECT=100 {{(pid=81409) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:08:16.133411 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0ac7c7b6-db0c-4ff8-b29f-40af25556c9b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81409) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:08:16.134065 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ac7c7b6-db0c-4ff8-b29f-40af25556c9b None admin] REQUEST_METHOD: `GET` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:08:16.134373 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ac7c7b6-db0c-4ff8-b29f-40af25556c9b None admin] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:08:16.134673 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ac7c7b6-db0c-4ff8-b29f-40af25556c9b None admin] PATH_INFO: `/v3/roles/reader` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:08:16.137787 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1b08014b-5ded-40b9-b621-14cf663356e3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81408) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:08:16.138430 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b08014b-5ded-40b9-b621-14cf663356e3 None admin] REQUEST_METHOD: `POST` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:08:16.138715 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b08014b-5ded-40b9-b621-14cf663356e3 None admin] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:08:16.138992 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b08014b-5ded-40b9-b621-14cf663356e3 None admin] PATH_INFO: `/v3/users` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:08:16.139647 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1b08014b-5ded-40b9-b621-14cf663356e3 None admin] RBAC: Authorizing `identity:create_user()` {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:08:16.141097 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1b08014b-5ded-40b9-b621-14cf663356e3 None admin] RBAC: Authorization granted {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:08:16.144479 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0ac7c7b6-db0c-4ff8-b29f-40af25556c9b None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 13 04:08:16.149839 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application [None req-0ac7c7b6-db0c-4ff8-b29f-40af25556c9b None admin] Could not find role: reader.: keystone.exception.RoleNotFound: Could not find role: reader. Oct 13 04:08:16.149839 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 13 04:08:16.149839 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 13 04:08:16.149839 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 13 04:08:16.149839 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 13 04:08:16.149839 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 13 04:08:16.149839 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 13 04:08:16.149839 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 13 04:08:16.149839 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 13 04:08:16.149839 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 13 04:08:16.149839 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 13 04:08:16.149839 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 13 04:08:16.149839 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 13 04:08:16.149839 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 13 04:08:16.149839 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 13 04:08:16.149839 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 13 04:08:16.150992 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 13 04:08:16.150992 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 13 04:08:16.150992 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 13 04:08:16.150992 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 13 04:08:16.150992 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 13 04:08:16.150992 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 13 04:08:16.150992 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 13 04:08:16.150992 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 13 04:08:16.150992 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 13 04:08:16.150992 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application return self.get_or_create( Oct 13 04:08:16.150992 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 13 04:08:16.150992 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application with Lock( Oct 13 04:08:16.150992 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 13 04:08:16.150992 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application return self._enter() Oct 13 04:08:16.150992 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 13 04:08:16.150992 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 13 04:08:16.151936 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 13 04:08:16.151936 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application return self.creator() Oct 13 04:08:16.151936 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 13 04:08:16.151936 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application created_value = creator( Oct 13 04:08:16.151936 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 13 04:08:16.151936 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 13 04:08:16.151936 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 13 04:08:16.151936 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 13 04:08:16.151936 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 13 04:08:16.151936 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 13 04:08:16.151936 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: reader. Oct 13 04:08:16.151936 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application Oct 13 04:08:16.151936 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 69/135] 10.222.0.27 () {56 vars in 1099 bytes} [Thu Oct 13 04:08:15 2022] GET /identity/v3/roles/reader => generated 84 bytes in 167 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 13 04:08:16.157247 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0ac7c7b6-db0c-4ff8-b29f-40af25556c9b None admin] Authenticating user token {{(pid=81409) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:08:16.165643 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 67/136] 10.222.0.27 () {60 vars in 1132 bytes} [Thu Oct 13 04:08:15 2022] POST /identity/v3/users => generated 283 bytes in 183 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 13 04:08:16.265310 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0ac7c7b6-db0c-4ff8-b29f-40af25556c9b None admin] Validating token access rules against request {{(pid=81409) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:08:16.301708 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-21fc563c-49a0-48de-845f-b798af49c6c3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81409) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:08:16.302446 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21fc563c-49a0-48de-845f-b798af49c6c3 None admin] REQUEST_METHOD: `GET` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:08:16.302732 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21fc563c-49a0-48de-845f-b798af49c6c3 None admin] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:08:16.303010 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21fc563c-49a0-48de-845f-b798af49c6c3 None admin] PATH_INFO: `/v3/roles` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:08:16.303687 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-21fc563c-49a0-48de-845f-b798af49c6c3 None admin] RBAC: Adding query filter params (name=reader) {{(pid=81409) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 13 04:08:16.303992 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-21fc563c-49a0-48de-845f-b798af49c6c3 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:08:16.305486 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-21fc563c-49a0-48de-845f-b798af49c6c3 None admin] RBAC: Authorization granted {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:08:16.314669 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 70/137] 10.222.0.27 () {56 vars in 1101 bytes} [Thu Oct 13 04:08:16 2022] GET /identity/v3/roles?name=reader => generated 341 bytes in 158 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 13 04:08:16.320413 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1b08014b-5ded-40b9-b621-14cf663356e3 None admin] Authenticating user token {{(pid=81408) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:08:16.404001 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG dbcounter [-] [81408] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81408) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:08:16.428955 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1b08014b-5ded-40b9-b621-14cf663356e3 None admin] Validating token access rules against request {{(pid=81408) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:08:16.465402 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1c517ae5-df63-485a-a62a-0f179475f041 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81408) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:08:16.466144 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c517ae5-df63-485a-a62a-0f179475f041 None admin] REQUEST_METHOD: `GET` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:08:16.466571 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c517ae5-df63-485a-a62a-0f179475f041 None admin] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:08:16.466853 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c517ae5-df63-485a-a62a-0f179475f041 None admin] PATH_INFO: `/v3/users/4cba1177c8534417a4db1df4fe654898` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:08:16.467996 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1c517ae5-df63-485a-a62a-0f179475f041 None admin] RBAC: Authorizing `identity:get_user(user_id=4cba1177c8534417a4db1df4fe654898)` {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:08:16.469473 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1c517ae5-df63-485a-a62a-0f179475f041 None admin] RBAC: Authorization granted {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:08:16.471245 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 68/138] 10.222.0.27 () {56 vars in 1177 bytes} [Thu Oct 13 04:08:16 2022] GET /identity/v3/users/4cba1177c8534417a4db1df4fe654898 => generated 289 bytes in 151 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 13 04:08:16.476859 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-21fc563c-49a0-48de-845f-b798af49c6c3 None admin] Authenticating user token {{(pid=81409) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:08:16.531978 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG dbcounter [-] [81409] Writing DB stats keystone:SELECT=100 {{(pid=81409) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:08:16.589376 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-21fc563c-49a0-48de-845f-b798af49c6c3 None admin] Validating token access rules against request {{(pid=81409) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:08:16.626846 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4a045814-1fc5-4446-bb1f-ffaf8bbc31d5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81409) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:08:16.627526 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a045814-1fc5-4446-bb1f-ffaf8bbc31d5 None admin] REQUEST_METHOD: `GET` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:08:16.627807 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a045814-1fc5-4446-bb1f-ffaf8bbc31d5 None admin] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:08:16.628081 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a045814-1fc5-4446-bb1f-ffaf8bbc31d5 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:08:16.708207 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4a045814-1fc5-4446-bb1f-ffaf8bbc31d5 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:08:16.710040 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4a045814-1fc5-4446-bb1f-ffaf8bbc31d5 None admin] RBAC: Authorization granted {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:08:16.763586 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG dbcounter [-] [81409] Writing DB stats keystone:SELECT=100 {{(pid=81409) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:08:16.818453 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 71/139] 10.222.0.27 () {58 vars in 1282 bytes} [Thu Oct 13 04:08:16 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 342 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Oct 13 04:08:16.824368 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1c517ae5-df63-485a-a62a-0f179475f041 None admin] Authenticating user token {{(pid=81408) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:08:16.933611 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1c517ae5-df63-485a-a62a-0f179475f041 None admin] Validating token access rules against request {{(pid=81408) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:08:16.969803 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-41301fa6-af7e-4cc2-aff9-8241a1effefb None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81408) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:08:16.970803 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41301fa6-af7e-4cc2-aff9-8241a1effefb None admin] REQUEST_METHOD: `GET` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:08:16.971090 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41301fa6-af7e-4cc2-aff9-8241a1effefb None admin] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:08:16.971372 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41301fa6-af7e-4cc2-aff9-8241a1effefb None admin] PATH_INFO: `/v3/projects/eeec419e34e44c1f881179933c828d97` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:08:16.974309 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-41301fa6-af7e-4cc2-aff9-8241a1effefb None admin] RBAC: Authorizing `identity:get_project(project_id=eeec419e34e44c1f881179933c828d97)` {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:08:16.975790 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-41301fa6-af7e-4cc2-aff9-8241a1effefb None admin] RBAC: Authorization granted {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:08:16.977952 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 69/140] 10.222.0.27 () {56 vars in 1186 bytes} [Thu Oct 13 04:08:16 2022] GET /identity/v3/projects/eeec419e34e44c1f881179933c828d97 => generated 298 bytes in 154 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 13 04:08:16.983934 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4a045814-1fc5-4446-bb1f-ffaf8bbc31d5 None admin] Authenticating user token {{(pid=81409) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:08:17.088651 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4a045814-1fc5-4446-bb1f-ffaf8bbc31d5 None admin] Validating token access rules against request {{(pid=81409) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:08:17.124098 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-72788a0d-7ca3-4dc4-adec-411a2d82c26e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81409) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:08:17.124713 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72788a0d-7ca3-4dc4-adec-411a2d82c26e None admin] REQUEST_METHOD: `GET` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:08:17.124999 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72788a0d-7ca3-4dc4-adec-411a2d82c26e None admin] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:08:17.125279 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72788a0d-7ca3-4dc4-adec-411a2d82c26e None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:08:17.126009 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-72788a0d-7ca3-4dc4-adec-411a2d82c26e None admin] RBAC: Adding query filter params (role.id=2306a441792b49f19af2d869acf5b3b6, scope.project.id=eeec419e34e44c1f881179933c828d97, user.id=4cba1177c8534417a4db1df4fe654898) {{(pid=81409) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 13 04:08:17.126351 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-72788a0d-7ca3-4dc4-adec-411a2d82c26e None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:08:17.127816 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-72788a0d-7ca3-4dc4-adec-411a2d82c26e None admin] RBAC: Authorization granted {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:08:17.138332 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 72/141] 10.222.0.27 () {56 vars in 1374 bytes} [Thu Oct 13 04:08:16 2022] GET /identity/v3/role_assignments?user.id=4cba1177c8534417a4db1df4fe654898&scope.project.id=eeec419e34e44c1f881179933c828d97&role.id=2306a441792b49f19af2d869acf5b3b6 => generated 608 bytes in 151 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 13 04:08:18.458543 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f9e0653-8ef1-49fb-82ab-1879921ab1cc None None] REQUEST_METHOD: `GET` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:08:18.459086 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f9e0653-8ef1-49fb-82ab-1879921ab1cc None None] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:08:18.459086 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f9e0653-8ef1-49fb-82ab-1879921ab1cc None None] PATH_INFO: `/` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:08:18.460443 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 70/142] 10.222.0.27 () {54 vars in 927 bytes} [Thu Oct 13 04:08:18 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 13 04:08:18.466553 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56c93738-2d86-47ba-a030-25bd372f3ec4 None None] REQUEST_METHOD: `POST` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:08:18.467324 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56c93738-2d86-47ba-a030-25bd372f3ec4 None None] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:08:18.467753 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56c93738-2d86-47ba-a030-25bd372f3ec4 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:08:18.488614 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.auth.core [None req-56c93738-2d86-47ba-a030-25bd372f3ec4 None None] MFA Rules not processed for user `0e2d979563d6421995451a0e86bf3022`. Rule list: `[]` (Enabled: `True`). {{(pid=81409) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 13 04:08:18.490107 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.fernet_utils [None req-56c93738-2d86-47ba-a030-25bd372f3ec4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81409) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:08:18.499935 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG dbcounter [-] [81409] Writing DB stats keystone:SELECT=100 {{(pid=81409) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:08:18.563072 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 73/143] 10.222.0.27 () {58 vars in 1024 bytes} [Thu Oct 13 04:08:18 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 97 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 13 04:08:18.601328 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b04e33c-863f-4fa7-8afc-ed105a06b0de None None] REQUEST_METHOD: `GET` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:08:18.601609 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b04e33c-863f-4fa7-8afc-ed105a06b0de None None] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:08:18.602061 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b04e33c-863f-4fa7-8afc-ed105a06b0de None None] PATH_INFO: `/` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:08:18.603983 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 71/144] 10.222.0.27 () {54 vars in 927 bytes} [Thu Oct 13 04:08:18 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 13 04:08:18.609516 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f32c9596-4ff4-4e3e-9b6e-125a530be9d1 None None] REQUEST_METHOD: `POST` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:08:18.609803 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f32c9596-4ff4-4e3e-9b6e-125a530be9d1 None None] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:08:18.610099 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f32c9596-4ff4-4e3e-9b6e-125a530be9d1 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:08:18.628987 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.auth.core [None req-f32c9596-4ff4-4e3e-9b6e-125a530be9d1 None None] MFA Rules not processed for user `0e2d979563d6421995451a0e86bf3022`. Rule list: `[]` (Enabled: `True`). {{(pid=81409) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 13 04:08:18.630160 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.fernet_utils [None req-f32c9596-4ff4-4e3e-9b6e-125a530be9d1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81409) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:08:18.691158 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e7e3809-6b8e-4127-a90a-8176a8bf8427 None None] REQUEST_METHOD: `GET` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:08:18.691668 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e7e3809-6b8e-4127-a90a-8176a8bf8427 None None] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:08:18.691933 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e7e3809-6b8e-4127-a90a-8176a8bf8427 None None] PATH_INFO: `/` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:08:18.693434 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 72/145] 10.222.0.27 () {54 vars in 927 bytes} [Thu Oct 13 04:08:18 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Oct 13 04:08:18.698761 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-234a8d64-7804-4e68-8e08-addd19cb5dc3 None None] REQUEST_METHOD: `POST` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:08:18.699138 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-234a8d64-7804-4e68-8e08-addd19cb5dc3 None None] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:08:18.704958 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 74/146] 10.222.0.27 () {58 vars in 1024 bytes} [Thu Oct 13 04:08:18 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 91 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 13 04:08:18.705100 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-234a8d64-7804-4e68-8e08-addd19cb5dc3 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:08:18.717923 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.auth.core [None req-234a8d64-7804-4e68-8e08-addd19cb5dc3 None None] MFA Rules not processed for user `0e2d979563d6421995451a0e86bf3022`. Rule list: `[]` (Enabled: `True`). {{(pid=81408) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 13 04:08:18.719129 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.fernet_utils [None req-234a8d64-7804-4e68-8e08-addd19cb5dc3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81408) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:08:18.731604 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG dbcounter [-] [81408] Writing DB stats keystone:SELECT=100 {{(pid=81408) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:08:18.788770 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 73/147] 10.222.0.27 () {58 vars in 1024 bytes} [Thu Oct 13 04:08:18 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 90 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Oct 13 04:08:18.849484 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f32c9596-4ff4-4e3e-9b6e-125a530be9d1 None None] Authenticating user token {{(pid=81409) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:08:18.921032 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG dbcounter [-] [81409] Writing DB stats keystone:SELECT=100 {{(pid=81409) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:08:18.951579 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f32c9596-4ff4-4e3e-9b6e-125a530be9d1 None None] Validating token access rules against request {{(pid=81409) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:08:18.971611 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-234a8d64-7804-4e68-8e08-addd19cb5dc3 None None] Authenticating user token {{(pid=81408) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:08:18.989472 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1bb48001-36ab-4610-84fa-49d8c1bffdd7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81409) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:08:18.990120 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1bb48001-36ab-4610-84fa-49d8c1bffdd7 None admin] REQUEST_METHOD: `GET` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:08:18.990566 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1bb48001-36ab-4610-84fa-49d8c1bffdd7 None admin] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:08:18.990836 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1bb48001-36ab-4610-84fa-49d8c1bffdd7 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:08:19.005040 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1bb48001-36ab-4610-84fa-49d8c1bffdd7 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 13 04:08:19.011363 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application [None req-1bb48001-36ab-4610-84fa-49d8c1bffdd7 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Oct 13 04:08:19.011363 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 13 04:08:19.011363 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 13 04:08:19.011363 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 13 04:08:19.011363 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 13 04:08:19.011363 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 13 04:08:19.011363 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 13 04:08:19.011363 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 13 04:08:19.011363 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 13 04:08:19.011363 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 13 04:08:19.011363 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 13 04:08:19.011363 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 13 04:08:19.011363 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 13 04:08:19.011363 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 13 04:08:19.011363 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 13 04:08:19.011363 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 13 04:08:19.012241 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 13 04:08:19.012241 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 13 04:08:19.012241 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 13 04:08:19.012241 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 13 04:08:19.012241 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 13 04:08:19.012241 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 13 04:08:19.012241 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 13 04:08:19.012241 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 13 04:08:19.012241 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 13 04:08:19.012241 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application return self.get_or_create( Oct 13 04:08:19.012241 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 13 04:08:19.012241 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application with Lock( Oct 13 04:08:19.012241 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 13 04:08:19.012241 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application return self._enter() Oct 13 04:08:19.012241 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 13 04:08:19.012241 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 13 04:08:19.013174 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 13 04:08:19.013174 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application return self.creator() Oct 13 04:08:19.013174 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 13 04:08:19.013174 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application created_value = creator( Oct 13 04:08:19.013174 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 13 04:08:19.013174 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 13 04:08:19.013174 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 13 04:08:19.013174 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 13 04:08:19.013174 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 13 04:08:19.013174 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 13 04:08:19.013174 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Oct 13 04:08:19.013174 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application Oct 13 04:08:19.013174 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 75/148] 10.222.0.27 () {56 vars in 1096 bytes} [Thu Oct 13 04:08:18 2022] GET /identity/v3/roles/admin => generated 83 bytes in 164 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 13 04:08:19.018546 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1bb48001-36ab-4610-84fa-49d8c1bffdd7 None admin] Authenticating user token {{(pid=81409) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:08:19.079401 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-234a8d64-7804-4e68-8e08-addd19cb5dc3 None None] Validating token access rules against request {{(pid=81408) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:08:19.113403 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7926d037-a8b8-425a-a491-4263c8061509 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81408) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:08:19.114098 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7926d037-a8b8-425a-a491-4263c8061509 None admin] REQUEST_METHOD: `GET` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:08:19.114411 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7926d037-a8b8-425a-a491-4263c8061509 None admin] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:08:19.114684 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7926d037-a8b8-425a-a491-4263c8061509 None admin] PATH_INFO: `/v3/roles/anotherrole` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:08:19.120332 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1bb48001-36ab-4610-84fa-49d8c1bffdd7 None admin] Validating token access rules against request {{(pid=81409) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:08:19.124855 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7926d037-a8b8-425a-a491-4263c8061509 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 13 04:08:19.129621 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application [None req-7926d037-a8b8-425a-a491-4263c8061509 None admin] Could not find role: anotherrole.: keystone.exception.RoleNotFound: Could not find role: anotherrole. Oct 13 04:08:19.129621 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 13 04:08:19.129621 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 13 04:08:19.129621 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 13 04:08:19.129621 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 13 04:08:19.129621 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 13 04:08:19.129621 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 13 04:08:19.129621 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 13 04:08:19.129621 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 13 04:08:19.129621 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 13 04:08:19.129621 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 13 04:08:19.129621 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 13 04:08:19.129621 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 13 04:08:19.129621 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 13 04:08:19.129621 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 13 04:08:19.129621 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 13 04:08:19.130536 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 13 04:08:19.130536 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 13 04:08:19.130536 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 13 04:08:19.130536 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 13 04:08:19.130536 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 13 04:08:19.130536 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 13 04:08:19.130536 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 13 04:08:19.130536 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 13 04:08:19.130536 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 13 04:08:19.130536 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application return self.get_or_create( Oct 13 04:08:19.130536 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 13 04:08:19.130536 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application with Lock( Oct 13 04:08:19.130536 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 13 04:08:19.130536 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application return self._enter() Oct 13 04:08:19.130536 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 13 04:08:19.130536 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 13 04:08:19.131300 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 13 04:08:19.131300 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application return self.creator() Oct 13 04:08:19.131300 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 13 04:08:19.131300 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application created_value = creator( Oct 13 04:08:19.131300 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 13 04:08:19.131300 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 13 04:08:19.131300 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 13 04:08:19.131300 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 13 04:08:19.131300 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 13 04:08:19.131300 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 13 04:08:19.131300 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: anotherrole. Oct 13 04:08:19.131300 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application Oct 13 04:08:19.131300 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 74/149] 10.222.0.27 () {56 vars in 1114 bytes} [Thu Oct 13 04:08:18 2022] GET /identity/v3/roles/anotherrole => generated 89 bytes in 160 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Oct 13 04:08:19.133064 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7926d037-a8b8-425a-a491-4263c8061509 None admin] Authenticating user token {{(pid=81408) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:08:19.142191 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG dbcounter [-] [81408] Writing DB stats keystone:SELECT=100 {{(pid=81408) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:08:19.149894 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG dbcounter [-] [81409] Writing DB stats keystone:SELECT=100 {{(pid=81409) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:08:19.155230 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6ceb356f-febc-45e3-a6a6-9b2cccf53aa2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81409) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:08:19.155833 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ceb356f-febc-45e3-a6a6-9b2cccf53aa2 None admin] REQUEST_METHOD: `GET` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:08:19.156108 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ceb356f-febc-45e3-a6a6-9b2cccf53aa2 None admin] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:08:19.156380 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ceb356f-febc-45e3-a6a6-9b2cccf53aa2 None admin] PATH_INFO: `/v3/roles` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:08:19.157043 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6ceb356f-febc-45e3-a6a6-9b2cccf53aa2 None admin] RBAC: Adding query filter params (name=admin) {{(pid=81409) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 13 04:08:19.157343 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6ceb356f-febc-45e3-a6a6-9b2cccf53aa2 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:08:19.158769 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6ceb356f-febc-45e3-a6a6-9b2cccf53aa2 None admin] RBAC: Authorization granted {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:08:19.168166 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 76/150] 10.222.0.27 () {56 vars in 1099 bytes} [Thu Oct 13 04:08:19 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 150 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 13 04:08:19.168983 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6ceb356f-febc-45e3-a6a6-9b2cccf53aa2 None admin] Authenticating user token {{(pid=81409) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:08:19.237594 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7926d037-a8b8-425a-a491-4263c8061509 None admin] Validating token access rules against request {{(pid=81408) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:08:19.267205 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6ceb356f-febc-45e3-a6a6-9b2cccf53aa2 None admin] Validating token access rules against request {{(pid=81409) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:08:19.272414 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8be1c05b-ec66-4b7f-9213-d10cc9fc086a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81408) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:08:19.273044 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8be1c05b-ec66-4b7f-9213-d10cc9fc086a None admin] REQUEST_METHOD: `GET` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:08:19.273323 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8be1c05b-ec66-4b7f-9213-d10cc9fc086a None admin] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:08:19.273601 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8be1c05b-ec66-4b7f-9213-d10cc9fc086a None admin] PATH_INFO: `/v3/domains/default` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:08:19.274767 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8be1c05b-ec66-4b7f-9213-d10cc9fc086a None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:08:19.276214 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8be1c05b-ec66-4b7f-9213-d10cc9fc086a None admin] RBAC: Authorization granted {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:08:19.277814 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 75/151] 10.222.0.27 () {56 vars in 1108 bytes} [Thu Oct 13 04:08:19 2022] GET /identity/v3/domains/default => generated 199 bytes in 146 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 13 04:08:19.278679 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8be1c05b-ec66-4b7f-9213-d10cc9fc086a None admin] Authenticating user token {{(pid=81408) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:08:19.300994 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-77621fc8-dfe9-46b0-8a73-6eb1c7b82565 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81409) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:08:19.301604 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77621fc8-dfe9-46b0-8a73-6eb1c7b82565 None admin] REQUEST_METHOD: `GET` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:08:19.301879 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77621fc8-dfe9-46b0-8a73-6eb1c7b82565 None admin] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:08:19.302154 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77621fc8-dfe9-46b0-8a73-6eb1c7b82565 None admin] PATH_INFO: `/v3/roles` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:08:19.302843 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-77621fc8-dfe9-46b0-8a73-6eb1c7b82565 None admin] RBAC: Adding query filter params (name=anotherrole) {{(pid=81409) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 13 04:08:19.303152 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-77621fc8-dfe9-46b0-8a73-6eb1c7b82565 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:08:19.304534 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-77621fc8-dfe9-46b0-8a73-6eb1c7b82565 None admin] RBAC: Authorization granted {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:08:19.313067 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 77/152] 10.222.0.27 () {56 vars in 1111 bytes} [Thu Oct 13 04:08:19 2022] GET /identity/v3/roles?name=anotherrole => generated 334 bytes in 144 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 13 04:08:19.313829 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-77621fc8-dfe9-46b0-8a73-6eb1c7b82565 None admin] Authenticating user token {{(pid=81409) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:08:19.356669 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG dbcounter [-] [81408] Writing DB stats keystone:SELECT=100 {{(pid=81408) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:08:19.368861 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG dbcounter [-] [81409] Writing DB stats keystone:SELECT=100 {{(pid=81409) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:08:19.386887 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8be1c05b-ec66-4b7f-9213-d10cc9fc086a None admin] Validating token access rules against request {{(pid=81408) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:08:19.418123 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-77621fc8-dfe9-46b0-8a73-6eb1c7b82565 None admin] Validating token access rules against request {{(pid=81409) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:08:19.425740 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-06691f83-1f58-48d7-82e2-82e027039d95 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81408) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:08:19.426402 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06691f83-1f58-48d7-82e2-82e027039d95 None admin] REQUEST_METHOD: `GET` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:08:19.426681 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06691f83-1f58-48d7-82e2-82e027039d95 None admin] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:08:19.426957 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06691f83-1f58-48d7-82e2-82e027039d95 None admin] PATH_INFO: `/v3/users/9e60b30c2fdc42bc9fa9ff5ad792f214` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:08:19.439919 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-06691f83-1f58-48d7-82e2-82e027039d95 None admin] RBAC: Authorizing `identity:get_user(user_id=9e60b30c2fdc42bc9fa9ff5ad792f214)` {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:08:19.441380 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-06691f83-1f58-48d7-82e2-82e027039d95 None admin] RBAC: Authorization granted {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:08:19.443019 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 76/153] 10.222.0.27 () {56 vars in 1177 bytes} [Thu Oct 13 04:08:19 2022] GET /identity/v3/users/9e60b30c2fdc42bc9fa9ff5ad792f214 => generated 283 bytes in 165 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 13 04:08:19.443793 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-06691f83-1f58-48d7-82e2-82e027039d95 None admin] Authenticating user token {{(pid=81408) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:08:19.452011 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-611e913c-0513-4971-9f37-51f88ed8a89e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81409) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:08:19.452624 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-611e913c-0513-4971-9f37-51f88ed8a89e None admin] REQUEST_METHOD: `POST` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:08:19.452903 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-611e913c-0513-4971-9f37-51f88ed8a89e None admin] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:08:19.453180 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-611e913c-0513-4971-9f37-51f88ed8a89e None admin] PATH_INFO: `/v3/users` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:08:19.453849 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-611e913c-0513-4971-9f37-51f88ed8a89e None admin] RBAC: Authorizing `identity:create_user()` {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:08:19.455304 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-611e913c-0513-4971-9f37-51f88ed8a89e None admin] RBAC: Authorization granted {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:08:19.482281 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 78/154] 10.222.0.27 () {60 vars in 1132 bytes} [Thu Oct 13 04:08:19 2022] POST /identity/v3/users => generated 297 bytes in 169 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 13 04:08:19.483106 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-611e913c-0513-4971-9f37-51f88ed8a89e None admin] Authenticating user token {{(pid=81409) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:08:19.557657 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-06691f83-1f58-48d7-82e2-82e027039d95 None admin] Validating token access rules against request {{(pid=81408) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:08:19.593233 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG dbcounter [-] [81408] Writing DB stats keystone:SELECT=100 {{(pid=81408) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:08:19.597958 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-611e913c-0513-4971-9f37-51f88ed8a89e None admin] Validating token access rules against request {{(pid=81409) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:08:19.599433 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e963f72e-6d21-4cde-a097-4e78ab290657 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81408) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:08:19.600081 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e963f72e-6d21-4cde-a097-4e78ab290657 None admin] REQUEST_METHOD: `GET` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:08:19.600362 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e963f72e-6d21-4cde-a097-4e78ab290657 None admin] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:08:19.600637 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e963f72e-6d21-4cde-a097-4e78ab290657 None admin] PATH_INFO: `/v3/users/9e60b30c2fdc42bc9fa9ff5ad792f214` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:08:19.601823 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e963f72e-6d21-4cde-a097-4e78ab290657 None admin] RBAC: Authorizing `identity:get_user(user_id=9e60b30c2fdc42bc9fa9ff5ad792f214)` {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:08:19.603906 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e963f72e-6d21-4cde-a097-4e78ab290657 None admin] RBAC: Authorization granted {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:08:19.605566 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 77/155] 10.222.0.27 () {56 vars in 1177 bytes} [Thu Oct 13 04:08:19 2022] GET /identity/v3/users/9e60b30c2fdc42bc9fa9ff5ad792f214 => generated 283 bytes in 162 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 13 04:08:19.611472 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e963f72e-6d21-4cde-a097-4e78ab290657 None admin] Authenticating user token {{(pid=81408) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:08:19.614442 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG dbcounter [-] [81409] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81409) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:08:19.642545 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cfcddcb3-8ae3-46c4-b93c-0071e54c212a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81409) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:08:19.643209 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cfcddcb3-8ae3-46c4-b93c-0071e54c212a None admin] REQUEST_METHOD: `GET` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:08:19.643482 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cfcddcb3-8ae3-46c4-b93c-0071e54c212a None admin] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:08:19.643756 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cfcddcb3-8ae3-46c4-b93c-0071e54c212a None admin] PATH_INFO: `/v3/projects/0114058aecc645ea8b47ae6d1301c17d` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:08:19.644971 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cfcddcb3-8ae3-46c4-b93c-0071e54c212a None admin] RBAC: Authorizing `identity:get_project(project_id=0114058aecc645ea8b47ae6d1301c17d)` {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:08:19.646491 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cfcddcb3-8ae3-46c4-b93c-0071e54c212a None admin] RBAC: Authorization granted {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:08:19.648217 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 79/156] 10.222.0.27 () {56 vars in 1186 bytes} [Thu Oct 13 04:08:19 2022] GET /identity/v3/projects/0114058aecc645ea8b47ae6d1301c17d => generated 302 bytes in 165 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 13 04:08:19.655743 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cfcddcb3-8ae3-46c4-b93c-0071e54c212a None admin] Authenticating user token {{(pid=81409) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:08:19.739353 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e963f72e-6d21-4cde-a097-4e78ab290657 None admin] Validating token access rules against request {{(pid=81408) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:08:19.793272 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f1260307-cc06-4e2a-ae7e-53bfd8905714 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81408) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:08:19.795257 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1260307-cc06-4e2a-ae7e-53bfd8905714 None admin] REQUEST_METHOD: `GET` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:08:19.795257 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1260307-cc06-4e2a-ae7e-53bfd8905714 None admin] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:08:19.795257 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1260307-cc06-4e2a-ae7e-53bfd8905714 None admin] PATH_INFO: `/v3/projects/0114058aecc645ea8b47ae6d1301c17d` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:08:19.797483 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cfcddcb3-8ae3-46c4-b93c-0071e54c212a None admin] Validating token access rules against request {{(pid=81409) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:08:19.797663 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f1260307-cc06-4e2a-ae7e-53bfd8905714 None admin] RBAC: Authorizing `identity:get_project(project_id=0114058aecc645ea8b47ae6d1301c17d)` {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:08:19.799010 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f1260307-cc06-4e2a-ae7e-53bfd8905714 None admin] RBAC: Authorization granted {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:08:19.800874 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 78/157] 10.222.0.27 () {56 vars in 1186 bytes} [Thu Oct 13 04:08:19 2022] GET /identity/v3/projects/0114058aecc645ea8b47ae6d1301c17d => generated 302 bytes in 190 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 13 04:08:19.807214 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f1260307-cc06-4e2a-ae7e-53bfd8905714 None admin] Authenticating user token {{(pid=81408) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:08:19.840563 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-305d6dec-dbff-4ec6-b759-d30cdb9a84da None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81409) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:08:19.842033 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-305d6dec-dbff-4ec6-b759-d30cdb9a84da None admin] REQUEST_METHOD: `PUT` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:08:19.842503 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-305d6dec-dbff-4ec6-b759-d30cdb9a84da None admin] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:08:19.842916 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-305d6dec-dbff-4ec6-b759-d30cdb9a84da None admin] PATH_INFO: `/v3/projects/0114058aecc645ea8b47ae6d1301c17d/users/9e60b30c2fdc42bc9fa9ff5ad792f214/roles/ef55fbd94e1040be83695754075a7ed5` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:08:19.845194 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-305d6dec-dbff-4ec6-b759-d30cdb9a84da None admin] RBAC: Authorizing `identity:create_grant(project_id=0114058aecc645ea8b47ae6d1301c17d, user_id=9e60b30c2fdc42bc9fa9ff5ad792f214, role_id=ef55fbd94e1040be83695754075a7ed5)` {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:08:19.847746 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-305d6dec-dbff-4ec6-b759-d30cdb9a84da None admin] RBAC: Authorization granted {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:08:19.864392 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 80/158] 10.222.0.27 () {58 vars in 1439 bytes} [Thu Oct 13 04:08:19 2022] PUT /identity/v3/projects/0114058aecc645ea8b47ae6d1301c17d/users/9e60b30c2fdc42bc9fa9ff5ad792f214/roles/ef55fbd94e1040be83695754075a7ed5 => generated 0 bytes in 209 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 13 04:08:19.892070 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG dbcounter [-] [81408] Writing DB stats keystone:SELECT=100 {{(pid=81408) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:08:19.938052 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f1260307-cc06-4e2a-ae7e-53bfd8905714 None admin] Validating token access rules against request {{(pid=81408) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:08:19.980192 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a20fa84c-6eb7-4a9d-816f-cd632b2bdec2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81408) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:08:19.981259 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a20fa84c-6eb7-4a9d-816f-cd632b2bdec2 None admin] REQUEST_METHOD: `PUT` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:08:19.981819 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a20fa84c-6eb7-4a9d-816f-cd632b2bdec2 None admin] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:08:19.982299 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a20fa84c-6eb7-4a9d-816f-cd632b2bdec2 None admin] PATH_INFO: `/v3/projects/0114058aecc645ea8b47ae6d1301c17d/users/9e60b30c2fdc42bc9fa9ff5ad792f214/roles/743995e93fef47fd91204c3d701054b0` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:08:19.984728 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a20fa84c-6eb7-4a9d-816f-cd632b2bdec2 None admin] RBAC: Authorizing `identity:create_grant(project_id=0114058aecc645ea8b47ae6d1301c17d, user_id=9e60b30c2fdc42bc9fa9ff5ad792f214, role_id=743995e93fef47fd91204c3d701054b0)` {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:08:19.986436 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a20fa84c-6eb7-4a9d-816f-cd632b2bdec2 None admin] RBAC: Authorization granted {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:08:20.005393 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 79/159] 10.222.0.27 () {58 vars in 1439 bytes} [Thu Oct 13 04:08:19 2022] PUT /identity/v3/projects/0114058aecc645ea8b47ae6d1301c17d/users/9e60b30c2fdc42bc9fa9ff5ad792f214/roles/743995e93fef47fd91204c3d701054b0 => generated 0 bytes in 199 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 13 04:08:21.819417 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6beeb472-65e7-4fbb-b113-7551c2e75e3e None None] REQUEST_METHOD: `GET` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:08:21.819957 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6beeb472-65e7-4fbb-b113-7551c2e75e3e None None] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:08:21.819957 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6beeb472-65e7-4fbb-b113-7551c2e75e3e None None] PATH_INFO: `/` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:08:21.821383 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 81/160] 10.222.0.27 () {54 vars in 927 bytes} [Thu Oct 13 04:08:21 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 13 04:08:21.826954 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3ead2a6-ef17-4e73-8095-74f0e5059c3b None None] REQUEST_METHOD: `POST` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:08:21.827476 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3ead2a6-ef17-4e73-8095-74f0e5059c3b None None] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:08:21.827627 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3ead2a6-ef17-4e73-8095-74f0e5059c3b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:08:21.846629 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.auth.core [None req-e3ead2a6-ef17-4e73-8095-74f0e5059c3b None None] MFA Rules not processed for user `0e2d979563d6421995451a0e86bf3022`. Rule list: `[]` (Enabled: `True`). {{(pid=81408) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 13 04:08:21.847833 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.fernet_utils [None req-e3ead2a6-ef17-4e73-8095-74f0e5059c3b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81408) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:08:21.898093 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-718fe9af-416e-4e16-9adc-d260fcd4ccec None None] REQUEST_METHOD: `GET` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:08:21.898629 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-718fe9af-416e-4e16-9adc-d260fcd4ccec None None] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:08:21.898979 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-718fe9af-416e-4e16-9adc-d260fcd4ccec None None] PATH_INFO: `/` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:08:21.901423 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 82/161] 10.222.0.27 () {54 vars in 927 bytes} [Thu Oct 13 04:08:21 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Oct 13 04:08:21.907009 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c51e7f52-8f85-4487-bf44-cc01421ab6e4 None None] REQUEST_METHOD: `POST` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:08:21.907284 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c51e7f52-8f85-4487-bf44-cc01421ab6e4 None None] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:08:21.907554 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c51e7f52-8f85-4487-bf44-cc01421ab6e4 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:08:21.923689 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 80/162] 10.222.0.27 () {58 vars in 1024 bytes} [Thu Oct 13 04:08:21 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 98 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 13 04:08:21.927785 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.auth.core [None req-c51e7f52-8f85-4487-bf44-cc01421ab6e4 None None] MFA Rules not processed for user `0e2d979563d6421995451a0e86bf3022`. Rule list: `[]` (Enabled: `True`). {{(pid=81409) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 13 04:08:21.928944 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.fernet_utils [None req-c51e7f52-8f85-4487-bf44-cc01421ab6e4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81409) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:08:21.966305 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG dbcounter [-] [81409] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81409) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:08:22.005462 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 83/163] 10.222.0.27 () {58 vars in 1024 bytes} [Thu Oct 13 04:08:21 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 99 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Oct 13 04:08:22.115212 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc9164d0-bbf1-4341-988f-579ec25d0765 None None] REQUEST_METHOD: `GET` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:08:22.115812 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc9164d0-bbf1-4341-988f-579ec25d0765 None None] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:08:22.116286 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc9164d0-bbf1-4341-988f-579ec25d0765 None None] PATH_INFO: `/` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:08:22.118491 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 81/164] 10.222.0.27 () {54 vars in 927 bytes} [Thu Oct 13 04:08:22 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 13 04:08:22.123813 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eee097ed-4bfd-4a70-aa7a-19a4f42a1b8a None None] REQUEST_METHOD: `POST` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:08:22.130227 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eee097ed-4bfd-4a70-aa7a-19a4f42a1b8a None None] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:08:22.130227 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eee097ed-4bfd-4a70-aa7a-19a4f42a1b8a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:08:22.143244 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.auth.core [None req-eee097ed-4bfd-4a70-aa7a-19a4f42a1b8a None None] MFA Rules not processed for user `0e2d979563d6421995451a0e86bf3022`. Rule list: `[]` (Enabled: `True`). {{(pid=81409) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 13 04:08:22.144418 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.fernet_utils [None req-eee097ed-4bfd-4a70-aa7a-19a4f42a1b8a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81409) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:08:22.186321 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a540eeb8-69e9-42f0-b13f-ca969a2d52ce None None] REQUEST_METHOD: `GET` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:08:22.186637 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a540eeb8-69e9-42f0-b13f-ca969a2d52ce None None] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:08:22.186877 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a540eeb8-69e9-42f0-b13f-ca969a2d52ce None None] PATH_INFO: `/` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:08:22.188358 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 82/165] 10.222.0.27 () {54 vars in 927 bytes} [Thu Oct 13 04:08:22 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 13 04:08:22.193129 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d289c372-d49a-4750-9424-5b11588cbb30 None None] REQUEST_METHOD: `POST` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:08:22.193412 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d289c372-d49a-4750-9424-5b11588cbb30 None None] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:08:22.193689 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d289c372-d49a-4750-9424-5b11588cbb30 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:08:22.212844 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.auth.core [None req-d289c372-d49a-4750-9424-5b11588cbb30 None None] MFA Rules not processed for user `0e2d979563d6421995451a0e86bf3022`. Rule list: `[]` (Enabled: `True`). {{(pid=81408) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 13 04:08:22.214070 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.fernet_utils [None req-d289c372-d49a-4750-9424-5b11588cbb30 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81408) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:08:22.216873 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 84/166] 10.222.0.27 () {58 vars in 1024 bytes} [Thu Oct 13 04:08:22 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 94 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 13 04:08:22.217710 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eee097ed-4bfd-4a70-aa7a-19a4f42a1b8a None None] Authenticating user token {{(pid=81409) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:08:22.277192 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG dbcounter [-] [81408] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81408) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:08:22.296080 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 83/167] 10.222.0.27 () {58 vars in 1024 bytes} [Thu Oct 13 04:08:22 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 104 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Oct 13 04:08:22.296080 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d289c372-d49a-4750-9424-5b11588cbb30 None None] Authenticating user token {{(pid=81408) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:08:22.332118 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG dbcounter [-] [81409] Writing DB stats keystone:SELECT=100 {{(pid=81409) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:08:22.335448 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eee097ed-4bfd-4a70-aa7a-19a4f42a1b8a None None] Validating token access rules against request {{(pid=81409) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:08:22.373621 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6780b85e-a3a6-461c-a51b-9def41cbc9f6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81409) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:08:22.374332 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6780b85e-a3a6-461c-a51b-9def41cbc9f6 None admin] REQUEST_METHOD: `GET` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:08:22.374700 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6780b85e-a3a6-461c-a51b-9def41cbc9f6 None admin] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:08:22.375080 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6780b85e-a3a6-461c-a51b-9def41cbc9f6 None admin] PATH_INFO: `/v3/roles/member` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:08:22.386147 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6780b85e-a3a6-461c-a51b-9def41cbc9f6 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 13 04:08:22.391190 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application [None req-6780b85e-a3a6-461c-a51b-9def41cbc9f6 None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Oct 13 04:08:22.391190 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 13 04:08:22.391190 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 13 04:08:22.391190 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 13 04:08:22.391190 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 13 04:08:22.391190 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 13 04:08:22.391190 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 13 04:08:22.391190 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 13 04:08:22.391190 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 13 04:08:22.391190 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 13 04:08:22.391190 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 13 04:08:22.391190 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 13 04:08:22.391190 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 13 04:08:22.391190 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 13 04:08:22.391190 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 13 04:08:22.391190 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 13 04:08:22.392303 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 13 04:08:22.392303 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 13 04:08:22.392303 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 13 04:08:22.392303 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 13 04:08:22.392303 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 13 04:08:22.392303 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 13 04:08:22.392303 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 13 04:08:22.392303 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 13 04:08:22.392303 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 13 04:08:22.392303 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application return self.get_or_create( Oct 13 04:08:22.392303 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 13 04:08:22.392303 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application with Lock( Oct 13 04:08:22.392303 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 13 04:08:22.392303 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application return self._enter() Oct 13 04:08:22.392303 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 13 04:08:22.392303 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 13 04:08:22.393303 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 13 04:08:22.393303 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application return self.creator() Oct 13 04:08:22.393303 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 13 04:08:22.393303 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application created_value = creator( Oct 13 04:08:22.393303 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 13 04:08:22.393303 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 13 04:08:22.393303 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 13 04:08:22.393303 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 13 04:08:22.393303 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 13 04:08:22.393303 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 13 04:08:22.393303 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Oct 13 04:08:22.393303 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application Oct 13 04:08:22.394757 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 85/168] 10.222.0.27 () {56 vars in 1099 bytes} [Thu Oct 13 04:08:22 2022] GET /identity/v3/roles/member => generated 84 bytes in 177 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 13 04:08:22.400887 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6780b85e-a3a6-461c-a51b-9def41cbc9f6 None admin] Authenticating user token {{(pid=81409) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:08:22.436125 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d289c372-d49a-4750-9424-5b11588cbb30 None None] Validating token access rules against request {{(pid=81408) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:08:22.470981 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d85bdc16-c554-4c31-a5f9-4c7a9f700b7d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81408) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:08:22.471940 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d85bdc16-c554-4c31-a5f9-4c7a9f700b7d None admin] REQUEST_METHOD: `GET` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:08:22.472429 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d85bdc16-c554-4c31-a5f9-4c7a9f700b7d None admin] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:08:22.472905 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d85bdc16-c554-4c31-a5f9-4c7a9f700b7d None admin] PATH_INFO: `/v3/domains/default` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:08:22.474480 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d85bdc16-c554-4c31-a5f9-4c7a9f700b7d None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:08:22.476122 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d85bdc16-c554-4c31-a5f9-4c7a9f700b7d None admin] RBAC: Authorization granted {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:08:22.478859 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 84/169] 10.222.0.27 () {56 vars in 1108 bytes} [Thu Oct 13 04:08:22 2022] GET /identity/v3/domains/default => generated 199 bytes in 180 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 13 04:08:22.490369 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d85bdc16-c554-4c31-a5f9-4c7a9f700b7d None admin] Authenticating user token {{(pid=81408) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:08:22.519334 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6780b85e-a3a6-461c-a51b-9def41cbc9f6 None admin] Validating token access rules against request {{(pid=81409) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:08:22.550068 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG dbcounter [-] [81408] Writing DB stats keystone:SELECT=100 {{(pid=81408) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:08:22.558338 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-953898a9-96f3-4b98-83ab-282d9a55d3cf None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81409) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:08:22.558980 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-953898a9-96f3-4b98-83ab-282d9a55d3cf None admin] REQUEST_METHOD: `GET` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:08:22.559232 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-953898a9-96f3-4b98-83ab-282d9a55d3cf None admin] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:08:22.559502 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-953898a9-96f3-4b98-83ab-282d9a55d3cf None admin] PATH_INFO: `/v3/roles` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:08:22.560169 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-953898a9-96f3-4b98-83ab-282d9a55d3cf None admin] RBAC: Adding query filter params (name=member) {{(pid=81409) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 13 04:08:22.560473 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-953898a9-96f3-4b98-83ab-282d9a55d3cf None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:08:22.561872 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-953898a9-96f3-4b98-83ab-282d9a55d3cf None admin] RBAC: Authorization granted {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:08:22.570529 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 86/170] 10.222.0.27 () {56 vars in 1101 bytes} [Thu Oct 13 04:08:22 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 170 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 13 04:08:22.571223 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-953898a9-96f3-4b98-83ab-282d9a55d3cf None admin] Authenticating user token {{(pid=81409) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:08:22.595853 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG dbcounter [-] [81409] Writing DB stats keystone:SELECT=100 {{(pid=81409) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:08:22.597703 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d85bdc16-c554-4c31-a5f9-4c7a9f700b7d None admin] Validating token access rules against request {{(pid=81408) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:08:22.632629 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fad1e8b2-a168-4a67-8a6f-95e2a1df7f96 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81408) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:08:22.633161 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fad1e8b2-a168-4a67-8a6f-95e2a1df7f96 None admin] REQUEST_METHOD: `POST` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:08:22.633437 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fad1e8b2-a168-4a67-8a6f-95e2a1df7f96 None admin] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:08:22.633846 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fad1e8b2-a168-4a67-8a6f-95e2a1df7f96 None admin] PATH_INFO: `/v3/users` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:08:22.634674 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fad1e8b2-a168-4a67-8a6f-95e2a1df7f96 None admin] RBAC: Authorizing `identity:create_user()` {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:08:22.636125 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fad1e8b2-a168-4a67-8a6f-95e2a1df7f96 None admin] RBAC: Authorization granted {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:08:22.656796 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 85/171] 10.222.0.27 () {60 vars in 1132 bytes} [Thu Oct 13 04:08:22 2022] POST /identity/v3/users => generated 297 bytes in 173 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 13 04:08:22.657608 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fad1e8b2-a168-4a67-8a6f-95e2a1df7f96 None admin] Authenticating user token {{(pid=81408) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:08:22.675023 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-953898a9-96f3-4b98-83ab-282d9a55d3cf None admin] Validating token access rules against request {{(pid=81409) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:08:22.717323 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b1730ef4-d5f2-40fa-bd16-59d099edcc1f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81409) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:08:22.718050 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1730ef4-d5f2-40fa-bd16-59d099edcc1f None admin] REQUEST_METHOD: `GET` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:08:22.718368 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1730ef4-d5f2-40fa-bd16-59d099edcc1f None admin] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:08:22.718662 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1730ef4-d5f2-40fa-bd16-59d099edcc1f None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:08:22.732305 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b1730ef4-d5f2-40fa-bd16-59d099edcc1f None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 13 04:08:22.737879 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application [None req-b1730ef4-d5f2-40fa-bd16-59d099edcc1f None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Oct 13 04:08:22.737879 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 13 04:08:22.737879 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 13 04:08:22.737879 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 13 04:08:22.737879 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 13 04:08:22.737879 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 13 04:08:22.737879 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 13 04:08:22.737879 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 13 04:08:22.737879 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 13 04:08:22.737879 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 13 04:08:22.737879 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 13 04:08:22.737879 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 13 04:08:22.737879 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 13 04:08:22.737879 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 13 04:08:22.737879 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 13 04:08:22.737879 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 13 04:08:22.738994 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 13 04:08:22.738994 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 13 04:08:22.738994 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 13 04:08:22.738994 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 13 04:08:22.738994 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 13 04:08:22.738994 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 13 04:08:22.738994 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 13 04:08:22.738994 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 13 04:08:22.738994 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 13 04:08:22.738994 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application return self.get_or_create( Oct 13 04:08:22.738994 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 13 04:08:22.738994 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application with Lock( Oct 13 04:08:22.738994 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 13 04:08:22.738994 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application return self._enter() Oct 13 04:08:22.738994 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 13 04:08:22.738994 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 13 04:08:22.739960 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 13 04:08:22.739960 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application return self.creator() Oct 13 04:08:22.739960 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 13 04:08:22.739960 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application created_value = creator( Oct 13 04:08:22.739960 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 13 04:08:22.739960 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 13 04:08:22.739960 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 13 04:08:22.739960 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 13 04:08:22.739960 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 13 04:08:22.739960 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 13 04:08:22.739960 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Oct 13 04:08:22.739960 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application Oct 13 04:08:22.739960 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 87/172] 10.222.0.27 () {56 vars in 1096 bytes} [Thu Oct 13 04:08:22 2022] GET /identity/v3/roles/admin => generated 83 bytes in 168 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 13 04:08:22.740718 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b1730ef4-d5f2-40fa-bd16-59d099edcc1f None admin] Authenticating user token {{(pid=81409) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:08:22.787821 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fad1e8b2-a168-4a67-8a6f-95e2a1df7f96 None admin] Validating token access rules against request {{(pid=81408) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:08:22.802580 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG dbcounter [-] [81408] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81408) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:08:22.832594 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cd630653-216c-4b5e-b43b-9a950fac9c26 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81408) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:08:22.833272 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd630653-216c-4b5e-b43b-9a950fac9c26 None admin] REQUEST_METHOD: `GET` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:08:22.833513 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd630653-216c-4b5e-b43b-9a950fac9c26 None admin] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:08:22.833789 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd630653-216c-4b5e-b43b-9a950fac9c26 None admin] PATH_INFO: `/v3/roles/anotherrole` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:08:22.836928 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG dbcounter [-] [81409] Writing DB stats keystone:SELECT=100 {{(pid=81409) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:08:22.847718 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cd630653-216c-4b5e-b43b-9a950fac9c26 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 13 04:08:22.853053 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application [None req-cd630653-216c-4b5e-b43b-9a950fac9c26 None admin] Could not find role: anotherrole.: keystone.exception.RoleNotFound: Could not find role: anotherrole. Oct 13 04:08:22.853053 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 13 04:08:22.853053 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 13 04:08:22.853053 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 13 04:08:22.853053 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 13 04:08:22.853053 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 13 04:08:22.853053 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 13 04:08:22.853053 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 13 04:08:22.853053 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 13 04:08:22.853053 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 13 04:08:22.853053 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 13 04:08:22.853053 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 13 04:08:22.853053 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 13 04:08:22.853053 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 13 04:08:22.853053 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 13 04:08:22.853053 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 13 04:08:22.854144 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 13 04:08:22.854144 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 13 04:08:22.854144 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 13 04:08:22.854144 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 13 04:08:22.854144 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 13 04:08:22.854144 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 13 04:08:22.854144 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 13 04:08:22.854144 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 13 04:08:22.854144 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 13 04:08:22.854144 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application return self.get_or_create( Oct 13 04:08:22.854144 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 13 04:08:22.854144 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application with Lock( Oct 13 04:08:22.854144 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 13 04:08:22.854144 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application return self._enter() Oct 13 04:08:22.854144 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 13 04:08:22.854144 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 13 04:08:22.855410 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 13 04:08:22.855410 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application return self.creator() Oct 13 04:08:22.855410 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 13 04:08:22.855410 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application created_value = creator( Oct 13 04:08:22.855410 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 13 04:08:22.855410 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 13 04:08:22.855410 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 13 04:08:22.855410 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 13 04:08:22.855410 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 13 04:08:22.855410 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 13 04:08:22.855410 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: anotherrole. Oct 13 04:08:22.855410 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application Oct 13 04:08:22.855410 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 86/173] 10.222.0.27 () {56 vars in 1114 bytes} [Thu Oct 13 04:08:22 2022] GET /identity/v3/roles/anotherrole => generated 89 bytes in 197 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 13 04:08:22.855410 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cd630653-216c-4b5e-b43b-9a950fac9c26 None admin] Authenticating user token {{(pid=81408) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:08:22.869228 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b1730ef4-d5f2-40fa-bd16-59d099edcc1f None admin] Validating token access rules against request {{(pid=81409) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:08:22.908574 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-14be76e7-427e-4d24-973f-9b19d2a22e3b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81409) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:08:22.909139 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14be76e7-427e-4d24-973f-9b19d2a22e3b None admin] REQUEST_METHOD: `GET` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:08:22.909466 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14be76e7-427e-4d24-973f-9b19d2a22e3b None admin] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:08:22.909744 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14be76e7-427e-4d24-973f-9b19d2a22e3b None admin] PATH_INFO: `/v3/users/5e337c24b5934eb88fbc565d2c0dc71d` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:08:22.926736 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-14be76e7-427e-4d24-973f-9b19d2a22e3b None admin] RBAC: Authorizing `identity:get_user(user_id=5e337c24b5934eb88fbc565d2c0dc71d)` {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:08:22.928216 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-14be76e7-427e-4d24-973f-9b19d2a22e3b None admin] RBAC: Authorization granted {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:08:22.929905 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 88/174] 10.222.0.27 () {56 vars in 1177 bytes} [Thu Oct 13 04:08:22 2022] GET /identity/v3/users/5e337c24b5934eb88fbc565d2c0dc71d => generated 297 bytes in 190 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 13 04:08:22.931489 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-14be76e7-427e-4d24-973f-9b19d2a22e3b None admin] Authenticating user token {{(pid=81409) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:08:22.969449 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cd630653-216c-4b5e-b43b-9a950fac9c26 None admin] Validating token access rules against request {{(pid=81408) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:08:23.008043 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9f5480fe-7568-43f7-aac4-6cbbc11a451f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81408) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:08:23.009577 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f5480fe-7568-43f7-aac4-6cbbc11a451f None admin] REQUEST_METHOD: `GET` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:08:23.010571 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f5480fe-7568-43f7-aac4-6cbbc11a451f None admin] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:08:23.011300 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f5480fe-7568-43f7-aac4-6cbbc11a451f None admin] PATH_INFO: `/v3/roles` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:08:23.012494 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9f5480fe-7568-43f7-aac4-6cbbc11a451f None admin] RBAC: Adding query filter params (name=admin) {{(pid=81408) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 13 04:08:23.013394 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9f5480fe-7568-43f7-aac4-6cbbc11a451f None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:08:23.015065 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9f5480fe-7568-43f7-aac4-6cbbc11a451f None admin] RBAC: Authorization granted {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:08:23.025072 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 87/175] 10.222.0.27 () {56 vars in 1099 bytes} [Thu Oct 13 04:08:22 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 170 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 13 04:08:23.026043 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9f5480fe-7568-43f7-aac4-6cbbc11a451f None admin] Authenticating user token {{(pid=81408) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:08:23.062186 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-14be76e7-427e-4d24-973f-9b19d2a22e3b None admin] Validating token access rules against request {{(pid=81409) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:08:23.069458 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG dbcounter [-] [81408] Writing DB stats keystone:SELECT=100 {{(pid=81408) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:08:23.098934 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-78f59a26-1eaf-4534-92c9-9d705ba54a98 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81409) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:08:23.099557 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78f59a26-1eaf-4534-92c9-9d705ba54a98 None admin] REQUEST_METHOD: `GET` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:08:23.099832 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78f59a26-1eaf-4534-92c9-9d705ba54a98 None admin] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:08:23.100107 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78f59a26-1eaf-4534-92c9-9d705ba54a98 None admin] PATH_INFO: `/v3/roles` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:08:23.100774 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-78f59a26-1eaf-4534-92c9-9d705ba54a98 None admin] RBAC: Adding query filter params (name=anotherrole) {{(pid=81409) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 13 04:08:23.101075 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-78f59a26-1eaf-4534-92c9-9d705ba54a98 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:08:23.103036 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-78f59a26-1eaf-4534-92c9-9d705ba54a98 None admin] RBAC: Authorization granted {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:08:23.111692 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 89/176] 10.222.0.27 () {56 vars in 1111 bytes} [Thu Oct 13 04:08:22 2022] GET /identity/v3/roles?name=anotherrole => generated 334 bytes in 181 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 13 04:08:23.112578 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-78f59a26-1eaf-4534-92c9-9d705ba54a98 None admin] Authenticating user token {{(pid=81409) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:08:23.115404 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG dbcounter [-] [81409] Writing DB stats keystone:SELECT=100 {{(pid=81409) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:08:23.158958 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9f5480fe-7568-43f7-aac4-6cbbc11a451f None admin] Validating token access rules against request {{(pid=81408) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:08:23.209032 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0e886738-2df4-4b8f-87d3-a702a90c368b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81408) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:08:23.209921 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e886738-2df4-4b8f-87d3-a702a90c368b None admin] REQUEST_METHOD: `GET` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:08:23.210589 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e886738-2df4-4b8f-87d3-a702a90c368b None admin] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:08:23.210868 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e886738-2df4-4b8f-87d3-a702a90c368b None admin] PATH_INFO: `/v3/projects/0114058aecc645ea8b47ae6d1301c17d` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:08:23.212117 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0e886738-2df4-4b8f-87d3-a702a90c368b None admin] RBAC: Authorizing `identity:get_project(project_id=0114058aecc645ea8b47ae6d1301c17d)` {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:08:23.214542 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0e886738-2df4-4b8f-87d3-a702a90c368b None admin] RBAC: Authorization granted {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:08:23.217463 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 88/177] 10.222.0.27 () {56 vars in 1186 bytes} [Thu Oct 13 04:08:23 2022] GET /identity/v3/projects/0114058aecc645ea8b47ae6d1301c17d => generated 302 bytes in 192 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 13 04:08:23.218993 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0e886738-2df4-4b8f-87d3-a702a90c368b None admin] Authenticating user token {{(pid=81408) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:08:23.224928 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-78f59a26-1eaf-4534-92c9-9d705ba54a98 None admin] Validating token access rules against request {{(pid=81409) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:08:23.262399 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2f8ed9cc-5e72-475c-866a-b3d8f9173ee6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81409) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:08:23.263012 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f8ed9cc-5e72-475c-866a-b3d8f9173ee6 None admin] REQUEST_METHOD: `GET` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:08:23.263293 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f8ed9cc-5e72-475c-866a-b3d8f9173ee6 None admin] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:08:23.263566 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f8ed9cc-5e72-475c-866a-b3d8f9173ee6 None admin] PATH_INFO: `/v3/users/9e60b30c2fdc42bc9fa9ff5ad792f214` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:08:23.264721 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2f8ed9cc-5e72-475c-866a-b3d8f9173ee6 None admin] RBAC: Authorizing `identity:get_user(user_id=9e60b30c2fdc42bc9fa9ff5ad792f214)` {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:08:23.266363 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2f8ed9cc-5e72-475c-866a-b3d8f9173ee6 None admin] RBAC: Authorization granted {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:08:23.267978 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 90/178] 10.222.0.27 () {56 vars in 1177 bytes} [Thu Oct 13 04:08:23 2022] GET /identity/v3/users/9e60b30c2fdc42bc9fa9ff5ad792f214 => generated 283 bytes in 156 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 13 04:08:23.268720 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2f8ed9cc-5e72-475c-866a-b3d8f9173ee6 None admin] Authenticating user token {{(pid=81409) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:08:23.342459 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0e886738-2df4-4b8f-87d3-a702a90c368b None admin] Validating token access rules against request {{(pid=81408) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:08:23.344146 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG dbcounter [-] [81408] Writing DB stats keystone:SELECT=100 {{(pid=81408) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:08:23.358115 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG dbcounter [-] [81409] Writing DB stats keystone:SELECT=100 {{(pid=81409) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:08:23.392329 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dbd42ac2-0157-4529-aca5-b96e95403c24 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81408) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:08:23.393938 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dbd42ac2-0157-4529-aca5-b96e95403c24 None admin] REQUEST_METHOD: `GET` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:08:23.394591 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dbd42ac2-0157-4529-aca5-b96e95403c24 None admin] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:08:23.396321 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dbd42ac2-0157-4529-aca5-b96e95403c24 None admin] PATH_INFO: `/v3/users/9e60b30c2fdc42bc9fa9ff5ad792f214` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:08:23.398160 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dbd42ac2-0157-4529-aca5-b96e95403c24 None admin] RBAC: Authorizing `identity:get_user(user_id=9e60b30c2fdc42bc9fa9ff5ad792f214)` {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:08:23.400466 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dbd42ac2-0157-4529-aca5-b96e95403c24 None admin] RBAC: Authorization granted {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:08:23.401286 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2f8ed9cc-5e72-475c-866a-b3d8f9173ee6 None admin] Validating token access rules against request {{(pid=81409) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:08:23.401993 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 89/179] 10.222.0.27 () {56 vars in 1177 bytes} [Thu Oct 13 04:08:23 2022] GET /identity/v3/users/9e60b30c2fdc42bc9fa9ff5ad792f214 => generated 283 bytes in 184 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 13 04:08:23.402928 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dbd42ac2-0157-4529-aca5-b96e95403c24 None admin] Authenticating user token {{(pid=81408) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:08:23.450746 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5a3c6bc5-df38-4401-8758-55d69cf2c8a5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81409) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:08:23.451417 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a3c6bc5-df38-4401-8758-55d69cf2c8a5 None admin] REQUEST_METHOD: `PUT` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:08:23.451694 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a3c6bc5-df38-4401-8758-55d69cf2c8a5 None admin] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:08:23.451968 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a3c6bc5-df38-4401-8758-55d69cf2c8a5 None admin] PATH_INFO: `/v3/projects/0114058aecc645ea8b47ae6d1301c17d/users/5e337c24b5934eb88fbc565d2c0dc71d/roles/cc56c4ae13a7424ea771c15da4b153c1` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:08:23.455546 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5a3c6bc5-df38-4401-8758-55d69cf2c8a5 None admin] RBAC: Authorizing `identity:create_grant(project_id=0114058aecc645ea8b47ae6d1301c17d, user_id=5e337c24b5934eb88fbc565d2c0dc71d, role_id=cc56c4ae13a7424ea771c15da4b153c1)` {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:08:23.457197 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5a3c6bc5-df38-4401-8758-55d69cf2c8a5 None admin] RBAC: Authorization granted {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:08:23.477523 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 91/180] 10.222.0.27 () {58 vars in 1439 bytes} [Thu Oct 13 04:08:23 2022] PUT /identity/v3/projects/0114058aecc645ea8b47ae6d1301c17d/users/5e337c24b5934eb88fbc565d2c0dc71d/roles/cc56c4ae13a7424ea771c15da4b153c1 => generated 0 bytes in 209 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 13 04:08:23.478361 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5a3c6bc5-df38-4401-8758-55d69cf2c8a5 None admin] Authenticating user token {{(pid=81409) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:08:23.534241 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dbd42ac2-0157-4529-aca5-b96e95403c24 None admin] Validating token access rules against request {{(pid=81408) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:08:23.574754 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-07fc257e-63d9-4dda-a371-68d425f254e5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81408) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:08:23.575470 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07fc257e-63d9-4dda-a371-68d425f254e5 None admin] REQUEST_METHOD: `GET` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:08:23.575842 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07fc257e-63d9-4dda-a371-68d425f254e5 None admin] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:08:23.576181 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07fc257e-63d9-4dda-a371-68d425f254e5 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:08:23.617504 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5a3c6bc5-df38-4401-8758-55d69cf2c8a5 None admin] Validating token access rules against request {{(pid=81409) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:08:23.622747 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG dbcounter [-] [81408] Writing DB stats keystone:SELECT=100 {{(pid=81408) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:08:23.654098 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG dbcounter [-] [81409] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81409) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:08:23.664265 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-225393b5-862b-48f0-a7fe-95e2948649c3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81409) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:08:23.665135 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-225393b5-862b-48f0-a7fe-95e2948649c3 None admin] REQUEST_METHOD: `GET` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:08:23.665463 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-225393b5-862b-48f0-a7fe-95e2948649c3 None admin] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:08:23.665742 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-225393b5-862b-48f0-a7fe-95e2948649c3 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:08:23.688512 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-07fc257e-63d9-4dda-a371-68d425f254e5 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:08:23.689956 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-07fc257e-63d9-4dda-a371-68d425f254e5 None admin] RBAC: Authorization granted {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:08:23.783690 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-225393b5-862b-48f0-a7fe-95e2948649c3 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:08:23.785181 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-225393b5-862b-48f0-a7fe-95e2948649c3 None admin] RBAC: Authorization granted {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:08:23.857085 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 90/181] 10.222.0.27 () {58 vars in 1282 bytes} [Thu Oct 13 04:08:23 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 455 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Oct 13 04:08:23.862730 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-07fc257e-63d9-4dda-a371-68d425f254e5 None admin] Authenticating user token {{(pid=81408) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:08:23.895651 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 92/182] 10.222.0.27 () {58 vars in 1282 bytes} [Thu Oct 13 04:08:23 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 418 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Oct 13 04:08:23.901346 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-225393b5-862b-48f0-a7fe-95e2948649c3 None admin] Authenticating user token {{(pid=81409) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:08:23.934643 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG dbcounter [-] [81409] Writing DB stats keystone:SELECT=100 {{(pid=81409) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:08:23.986770 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG dbcounter [-] [81408] Writing DB stats keystone:SELECT=100 {{(pid=81408) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:08:24.011536 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-225393b5-862b-48f0-a7fe-95e2948649c3 None admin] Validating token access rules against request {{(pid=81409) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:08:24.041589 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-07fc257e-63d9-4dda-a371-68d425f254e5 None admin] Validating token access rules against request {{(pid=81408) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:08:24.056742 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff3af43f-a9b7-4d8a-af94-d71a9784a15a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81409) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:08:24.057301 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff3af43f-a9b7-4d8a-af94-d71a9784a15a None admin] REQUEST_METHOD: `GET` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:08:24.057576 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff3af43f-a9b7-4d8a-af94-d71a9784a15a None admin] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:08:24.057845 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff3af43f-a9b7-4d8a-af94-d71a9784a15a None admin] PATH_INFO: `/v3/projects/0114058aecc645ea8b47ae6d1301c17d` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:08:24.060246 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ff3af43f-a9b7-4d8a-af94-d71a9784a15a None admin] RBAC: Authorizing `identity:get_project(project_id=0114058aecc645ea8b47ae6d1301c17d)` {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:08:24.061651 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ff3af43f-a9b7-4d8a-af94-d71a9784a15a None admin] RBAC: Authorization granted {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:08:24.068371 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 93/183] 10.222.0.27 () {56 vars in 1186 bytes} [Thu Oct 13 04:08:23 2022] GET /identity/v3/projects/0114058aecc645ea8b47ae6d1301c17d => generated 302 bytes in 163 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 13 04:08:24.069607 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff3af43f-a9b7-4d8a-af94-d71a9784a15a None admin] Authenticating user token {{(pid=81409) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:08:24.112903 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-48859a51-00aa-4ee0-9b5c-3c90034a427b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81408) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:08:24.113544 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48859a51-00aa-4ee0-9b5c-3c90034a427b None admin] REQUEST_METHOD: `GET` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:08:24.113820 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48859a51-00aa-4ee0-9b5c-3c90034a427b None admin] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:08:24.114121 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48859a51-00aa-4ee0-9b5c-3c90034a427b None admin] PATH_INFO: `/v3/projects/0114058aecc645ea8b47ae6d1301c17d` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:08:24.116473 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-48859a51-00aa-4ee0-9b5c-3c90034a427b None admin] RBAC: Authorizing `identity:get_project(project_id=0114058aecc645ea8b47ae6d1301c17d)` {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:08:24.118003 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-48859a51-00aa-4ee0-9b5c-3c90034a427b None admin] RBAC: Authorization granted {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:08:24.119717 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 91/184] 10.222.0.27 () {56 vars in 1186 bytes} [Thu Oct 13 04:08:23 2022] GET /identity/v3/projects/0114058aecc645ea8b47ae6d1301c17d => generated 302 bytes in 258 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 13 04:08:24.125706 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-48859a51-00aa-4ee0-9b5c-3c90034a427b None admin] Authenticating user token {{(pid=81408) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:08:24.224706 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG dbcounter [-] [81409] Writing DB stats keystone:SELECT=100 {{(pid=81409) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:08:24.231299 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff3af43f-a9b7-4d8a-af94-d71a9784a15a None admin] Validating token access rules against request {{(pid=81409) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:08:24.284520 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-84e0d411-d5cd-414b-b2d6-061a68121fad None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81409) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:08:24.285129 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84e0d411-d5cd-414b-b2d6-061a68121fad None admin] REQUEST_METHOD: `GET` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:08:24.285399 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84e0d411-d5cd-414b-b2d6-061a68121fad None admin] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:08:24.285664 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84e0d411-d5cd-414b-b2d6-061a68121fad None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:08:24.286422 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-84e0d411-d5cd-414b-b2d6-061a68121fad None admin] RBAC: Adding query filter params (role.id=743995e93fef47fd91204c3d701054b0, scope.project.id=0114058aecc645ea8b47ae6d1301c17d, user.id=9e60b30c2fdc42bc9fa9ff5ad792f214) {{(pid=81409) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 13 04:08:24.286728 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-84e0d411-d5cd-414b-b2d6-061a68121fad None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:08:24.288154 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-84e0d411-d5cd-414b-b2d6-061a68121fad None admin] RBAC: Authorization granted {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:08:24.300826 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 94/185] 10.222.0.27 () {56 vars in 1374 bytes} [Thu Oct 13 04:08:24 2022] GET /identity/v3/role_assignments?user.id=9e60b30c2fdc42bc9fa9ff5ad792f214&scope.project.id=0114058aecc645ea8b47ae6d1301c17d&role.id=743995e93fef47fd91204c3d701054b0 => generated 608 bytes in 232 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 13 04:08:24.341761 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-48859a51-00aa-4ee0-9b5c-3c90034a427b None admin] Validating token access rules against request {{(pid=81408) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:08:24.408572 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f1aa9903-2cb1-4c97-8552-612c53792d8e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81408) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:08:24.409198 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1aa9903-2cb1-4c97-8552-612c53792d8e None admin] REQUEST_METHOD: `GET` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:08:24.409472 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1aa9903-2cb1-4c97-8552-612c53792d8e None admin] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:08:24.409738 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1aa9903-2cb1-4c97-8552-612c53792d8e None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:08:24.410509 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f1aa9903-2cb1-4c97-8552-612c53792d8e None admin] RBAC: Adding query filter params (role.id=ef55fbd94e1040be83695754075a7ed5, scope.project.id=0114058aecc645ea8b47ae6d1301c17d, user.id=9e60b30c2fdc42bc9fa9ff5ad792f214) {{(pid=81408) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 13 04:08:24.410816 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f1aa9903-2cb1-4c97-8552-612c53792d8e None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:08:24.412266 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f1aa9903-2cb1-4c97-8552-612c53792d8e None admin] RBAC: Authorization granted {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:08:24.421829 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 92/186] 10.222.0.27 () {56 vars in 1374 bytes} [Thu Oct 13 04:08:24 2022] GET /identity/v3/role_assignments?user.id=9e60b30c2fdc42bc9fa9ff5ad792f214&scope.project.id=0114058aecc645ea8b47ae6d1301c17d&role.id=ef55fbd94e1040be83695754075a7ed5 => generated 608 bytes in 296 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 13 04:08:25.079612 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b6cabc8-3a15-40ac-b7b0-f11353cd5c5c None None] REQUEST_METHOD: `GET` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:08:25.080147 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b6cabc8-3a15-40ac-b7b0-f11353cd5c5c None None] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:08:25.080147 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b6cabc8-3a15-40ac-b7b0-f11353cd5c5c None None] PATH_INFO: `/` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:08:25.081655 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 95/187] 10.222.0.27 () {54 vars in 927 bytes} [Thu Oct 13 04:08:25 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 13 04:08:25.086592 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-989e74af-fde8-40a0-8d1b-308e9344d6e6 None None] REQUEST_METHOD: `POST` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:08:25.087514 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-989e74af-fde8-40a0-8d1b-308e9344d6e6 None None] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:08:25.087673 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-989e74af-fde8-40a0-8d1b-308e9344d6e6 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:08:25.094096 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG dbcounter [-] [81408] Writing DB stats keystone:SELECT=100 {{(pid=81408) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:08:25.107170 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.auth.core [None req-989e74af-fde8-40a0-8d1b-308e9344d6e6 None None] MFA Rules not processed for user `0e2d979563d6421995451a0e86bf3022`. Rule list: `[]` (Enabled: `True`). {{(pid=81408) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 13 04:08:25.108374 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.fernet_utils [None req-989e74af-fde8-40a0-8d1b-308e9344d6e6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81408) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:08:25.176462 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 93/188] 10.222.0.27 () {58 vars in 1024 bytes} [Thu Oct 13 04:08:25 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 91 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 13 04:08:25.325507 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-396289f5-f00f-467c-9bc4-8fa5541d4082 None None] REQUEST_METHOD: `GET` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:08:25.325882 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-396289f5-f00f-467c-9bc4-8fa5541d4082 None None] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:08:25.326057 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-396289f5-f00f-467c-9bc4-8fa5541d4082 None None] PATH_INFO: `/` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:08:25.327566 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 96/189] 10.222.0.27 () {54 vars in 927 bytes} [Thu Oct 13 04:08:25 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 13 04:08:25.332674 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d357915-ebc4-4116-b022-0d10c610dc89 None None] REQUEST_METHOD: `POST` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:08:25.332906 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d357915-ebc4-4116-b022-0d10c610dc89 None None] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:08:25.333174 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d357915-ebc4-4116-b022-0d10c610dc89 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:08:25.351751 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.auth.core [None req-8d357915-ebc4-4116-b022-0d10c610dc89 None None] MFA Rules not processed for user `0e2d979563d6421995451a0e86bf3022`. Rule list: `[]` (Enabled: `True`). {{(pid=81408) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 13 04:08:25.352988 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.fernet_utils [None req-8d357915-ebc4-4116-b022-0d10c610dc89 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81408) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:08:25.422258 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 94/190] 10.222.0.27 () {58 vars in 1024 bytes} [Thu Oct 13 04:08:25 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 90 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 13 04:08:25.451088 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-396289f5-f00f-467c-9bc4-8fa5541d4082 None None] Authenticating user token {{(pid=81409) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:08:25.551971 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-396289f5-f00f-467c-9bc4-8fa5541d4082 None None] Validating token access rules against request {{(pid=81409) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:08:25.584605 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b90b49c4-e876-4f72-aae3-1ae8556daa62 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81409) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:08:25.585160 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b90b49c4-e876-4f72-aae3-1ae8556daa62 None admin] REQUEST_METHOD: `GET` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:08:25.585433 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b90b49c4-e876-4f72-aae3-1ae8556daa62 None admin] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:08:25.585699 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b90b49c4-e876-4f72-aae3-1ae8556daa62 None admin] PATH_INFO: `/v3/roles/reader` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:08:25.595872 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b90b49c4-e876-4f72-aae3-1ae8556daa62 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 13 04:08:25.601376 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application [None req-b90b49c4-e876-4f72-aae3-1ae8556daa62 None admin] Could not find role: reader.: keystone.exception.RoleNotFound: Could not find role: reader. Oct 13 04:08:25.601376 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 13 04:08:25.601376 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 13 04:08:25.601376 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 13 04:08:25.601376 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 13 04:08:25.601376 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 13 04:08:25.601376 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 13 04:08:25.601376 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 13 04:08:25.601376 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 13 04:08:25.601376 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 13 04:08:25.601376 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 13 04:08:25.601376 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 13 04:08:25.601376 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 13 04:08:25.601376 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 13 04:08:25.601376 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 13 04:08:25.601376 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 13 04:08:25.602221 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 13 04:08:25.602221 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 13 04:08:25.602221 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 13 04:08:25.602221 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 13 04:08:25.602221 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 13 04:08:25.602221 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 13 04:08:25.602221 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 13 04:08:25.602221 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 13 04:08:25.602221 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 13 04:08:25.602221 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application return self.get_or_create( Oct 13 04:08:25.602221 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 13 04:08:25.602221 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application with Lock( Oct 13 04:08:25.602221 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 13 04:08:25.602221 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application return self._enter() Oct 13 04:08:25.602221 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 13 04:08:25.602221 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 13 04:08:25.603871 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 13 04:08:25.603871 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application return self.creator() Oct 13 04:08:25.603871 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 13 04:08:25.603871 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application created_value = creator( Oct 13 04:08:25.603871 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 13 04:08:25.603871 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 13 04:08:25.603871 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 13 04:08:25.603871 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 13 04:08:25.603871 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 13 04:08:25.603871 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 13 04:08:25.603871 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: reader. Oct 13 04:08:25.603871 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application Oct 13 04:08:25.606540 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 97/191] 10.222.0.27 () {56 vars in 1099 bytes} [Thu Oct 13 04:08:25 2022] GET /identity/v3/roles/reader => generated 84 bytes in 156 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Oct 13 04:08:25.608738 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8d357915-ebc4-4116-b022-0d10c610dc89 None None] Authenticating user token {{(pid=81408) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:08:25.672970 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG dbcounter [-] [81408] Writing DB stats keystone:SELECT=100 {{(pid=81408) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:08:25.706999 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b90b49c4-e876-4f72-aae3-1ae8556daa62 None admin] Authenticating user token {{(pid=81409) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:08:25.713849 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8d357915-ebc4-4116-b022-0d10c610dc89 None None] Validating token access rules against request {{(pid=81408) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:08:25.732955 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG dbcounter [-] [81409] Writing DB stats keystone:SELECT=100 {{(pid=81409) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:08:25.752292 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-efe7857e-2af6-47b3-aafb-77b57f54b2c2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81408) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:08:25.752927 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-efe7857e-2af6-47b3-aafb-77b57f54b2c2 None admin] REQUEST_METHOD: `GET` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:08:25.753327 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-efe7857e-2af6-47b3-aafb-77b57f54b2c2 None admin] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:08:25.753598 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-efe7857e-2af6-47b3-aafb-77b57f54b2c2 None admin] PATH_INFO: `/v3/roles` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:08:25.754395 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-efe7857e-2af6-47b3-aafb-77b57f54b2c2 None admin] RBAC: Adding query filter params (name=reader) {{(pid=81408) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 13 04:08:25.754705 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-efe7857e-2af6-47b3-aafb-77b57f54b2c2 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:08:25.756216 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-efe7857e-2af6-47b3-aafb-77b57f54b2c2 None admin] RBAC: Authorization granted {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:08:25.766237 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 95/192] 10.222.0.27 () {56 vars in 1101 bytes} [Thu Oct 13 04:08:25 2022] GET /identity/v3/roles?name=reader => generated 341 bytes in 158 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 13 04:08:25.771664 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-efe7857e-2af6-47b3-aafb-77b57f54b2c2 None admin] Authenticating user token {{(pid=81408) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:08:25.812632 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b90b49c4-e876-4f72-aae3-1ae8556daa62 None admin] Validating token access rules against request {{(pid=81409) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:08:25.848145 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-982cd978-7dd3-4e4d-9c2c-a8e69dd599d6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81409) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:08:25.848959 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-982cd978-7dd3-4e4d-9c2c-a8e69dd599d6 None admin] REQUEST_METHOD: `GET` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:08:25.849234 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-982cd978-7dd3-4e4d-9c2c-a8e69dd599d6 None admin] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:08:25.849502 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-982cd978-7dd3-4e4d-9c2c-a8e69dd599d6 None admin] PATH_INFO: `/v3/domains/default` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:08:25.850649 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-982cd978-7dd3-4e4d-9c2c-a8e69dd599d6 None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:08:25.852247 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-982cd978-7dd3-4e4d-9c2c-a8e69dd599d6 None admin] RBAC: Authorization granted {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:08:25.858543 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 98/193] 10.222.0.27 () {56 vars in 1108 bytes} [Thu Oct 13 04:08:25 2022] GET /identity/v3/domains/default => generated 199 bytes in 148 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 13 04:08:25.859946 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-982cd978-7dd3-4e4d-9c2c-a8e69dd599d6 None admin] Authenticating user token {{(pid=81409) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:08:25.882564 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-efe7857e-2af6-47b3-aafb-77b57f54b2c2 None admin] Validating token access rules against request {{(pid=81408) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:08:25.910232 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG dbcounter [-] [81408] Writing DB stats keystone:SELECT=100 {{(pid=81408) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:08:25.923383 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ed862f40-cc1c-4902-8d59-562f75b4a6d8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81408) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:08:25.924229 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed862f40-cc1c-4902-8d59-562f75b4a6d8 None admin] REQUEST_METHOD: `GET` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:08:25.924568 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed862f40-cc1c-4902-8d59-562f75b4a6d8 None admin] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:08:25.924909 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed862f40-cc1c-4902-8d59-562f75b4a6d8 None admin] PATH_INFO: `/v3/users/8ad7a8e540384e4688c9dccfd1dcc631` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:08:25.941379 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ed862f40-cc1c-4902-8d59-562f75b4a6d8 None admin] RBAC: Authorizing `identity:get_user(user_id=8ad7a8e540384e4688c9dccfd1dcc631)` {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:08:25.943014 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ed862f40-cc1c-4902-8d59-562f75b4a6d8 None admin] RBAC: Authorization granted {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:08:25.949436 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 96/194] 10.222.0.27 () {56 vars in 1177 bytes} [Thu Oct 13 04:08:25 2022] GET /identity/v3/users/8ad7a8e540384e4688c9dccfd1dcc631 => generated 297 bytes in 174 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 13 04:08:25.950498 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ed862f40-cc1c-4902-8d59-562f75b4a6d8 None admin] Authenticating user token {{(pid=81408) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:08:25.966115 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG dbcounter [-] [81409] Writing DB stats keystone:SELECT=100 {{(pid=81409) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:08:25.977243 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-982cd978-7dd3-4e4d-9c2c-a8e69dd599d6 None admin] Validating token access rules against request {{(pid=81409) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:08:26.014089 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ade80b49-e0d6-4b82-9e3a-8bf4a995e7be None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81409) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:08:26.014726 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ade80b49-e0d6-4b82-9e3a-8bf4a995e7be None admin] REQUEST_METHOD: `POST` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:08:26.015003 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ade80b49-e0d6-4b82-9e3a-8bf4a995e7be None admin] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:08:26.015265 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ade80b49-e0d6-4b82-9e3a-8bf4a995e7be None admin] PATH_INFO: `/v3/users` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:08:26.015907 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ade80b49-e0d6-4b82-9e3a-8bf4a995e7be None admin] RBAC: Authorizing `identity:create_user()` {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:08:26.017326 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ade80b49-e0d6-4b82-9e3a-8bf4a995e7be None admin] RBAC: Authorization granted {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:08:26.050236 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 99/195] 10.222.0.27 () {60 vars in 1132 bytes} [Thu Oct 13 04:08:25 2022] POST /identity/v3/users => generated 293 bytes in 180 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 13 04:08:26.050236 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-978c76d7-1c18-40d5-87de-ad43ce84099a None None] REQUEST_METHOD: `GET` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:08:26.050236 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-978c76d7-1c18-40d5-87de-ad43ce84099a None None] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:08:26.050236 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-978c76d7-1c18-40d5-87de-ad43ce84099a None None] PATH_INFO: `/` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:08:26.050236 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 100/196] 10.222.0.27 () {54 vars in 927 bytes} [Thu Oct 13 04:08:26 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 13 04:08:26.050236 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f5431ff-9120-4bbe-be68-fb95726ecd76 None None] REQUEST_METHOD: `POST` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:08:26.050236 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f5431ff-9120-4bbe-be68-fb95726ecd76 None None] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:08:26.050803 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f5431ff-9120-4bbe-be68-fb95726ecd76 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:08:26.070478 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ed862f40-cc1c-4902-8d59-562f75b4a6d8 None admin] Validating token access rules against request {{(pid=81408) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:08:26.071283 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.auth.core [None req-5f5431ff-9120-4bbe-be68-fb95726ecd76 None None] MFA Rules not processed for user `0e2d979563d6421995451a0e86bf3022`. Rule list: `[]` (Enabled: `True`). {{(pid=81409) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 13 04:08:26.072541 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.fernet_utils [None req-5f5431ff-9120-4bbe-be68-fb95726ecd76 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81409) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:08:26.111553 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e50b347f-eeb3-492e-9be0-d3b52609d66a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81408) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:08:26.112118 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e50b347f-eeb3-492e-9be0-d3b52609d66a None admin] REQUEST_METHOD: `GET` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:08:26.112396 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e50b347f-eeb3-492e-9be0-d3b52609d66a None admin] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:08:26.112669 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e50b347f-eeb3-492e-9be0-d3b52609d66a None admin] PATH_INFO: `/v3/projects/0114058aecc645ea8b47ae6d1301c17d` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:08:26.113881 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e50b347f-eeb3-492e-9be0-d3b52609d66a None admin] RBAC: Authorizing `identity:get_project(project_id=0114058aecc645ea8b47ae6d1301c17d)` {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:08:26.115487 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e50b347f-eeb3-492e-9be0-d3b52609d66a None admin] RBAC: Authorization granted {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:08:26.117433 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 97/197] 10.222.0.27 () {56 vars in 1186 bytes} [Thu Oct 13 04:08:25 2022] GET /identity/v3/projects/0114058aecc645ea8b47ae6d1301c17d => generated 302 bytes in 167 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 13 04:08:26.123366 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e50b347f-eeb3-492e-9be0-d3b52609d66a None admin] Authenticating user token {{(pid=81408) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:08:26.151579 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 101/198] 10.222.0.27 () {58 vars in 1024 bytes} [Thu Oct 13 04:08:26 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 103 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Oct 13 04:08:26.183198 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG dbcounter [-] [81408] Writing DB stats keystone:SELECT=100 {{(pid=81408) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:08:26.238491 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e50b347f-eeb3-492e-9be0-d3b52609d66a None admin] Validating token access rules against request {{(pid=81408) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:08:26.275577 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bfbab003-cc4f-4733-8232-498282229d14 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81408) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:08:26.276241 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bfbab003-cc4f-4733-8232-498282229d14 None admin] REQUEST_METHOD: `PUT` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:08:26.276518 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bfbab003-cc4f-4733-8232-498282229d14 None admin] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:08:26.276785 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bfbab003-cc4f-4733-8232-498282229d14 None admin] PATH_INFO: `/v3/projects/0114058aecc645ea8b47ae6d1301c17d/users/8ad7a8e540384e4688c9dccfd1dcc631/roles/2306a441792b49f19af2d869acf5b3b6` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:08:26.278745 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bfbab003-cc4f-4733-8232-498282229d14 None admin] RBAC: Authorizing `identity:create_grant(project_id=0114058aecc645ea8b47ae6d1301c17d, user_id=8ad7a8e540384e4688c9dccfd1dcc631, role_id=2306a441792b49f19af2d869acf5b3b6)` {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:08:26.280196 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bfbab003-cc4f-4733-8232-498282229d14 None admin] RBAC: Authorization granted {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:08:26.294675 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 98/199] 10.222.0.27 () {58 vars in 1439 bytes} [Thu Oct 13 04:08:26 2022] PUT /identity/v3/projects/0114058aecc645ea8b47ae6d1301c17d/users/8ad7a8e540384e4688c9dccfd1dcc631/roles/2306a441792b49f19af2d869acf5b3b6 => generated 0 bytes in 172 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 13 04:08:26.439867 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f5431ff-9120-4bbe-be68-fb95726ecd76 None None] Authenticating user token {{(pid=81409) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:08:26.527429 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG dbcounter [-] [81409] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81409) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:08:26.553222 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f5431ff-9120-4bbe-be68-fb95726ecd76 None None] Validating token access rules against request {{(pid=81409) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:08:26.591227 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-90d6d12c-2b84-48e2-a019-d39e3a2a66fe None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81409) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:08:26.591855 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90d6d12c-2b84-48e2-a019-d39e3a2a66fe None admin] REQUEST_METHOD: `GET` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:08:26.592125 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90d6d12c-2b84-48e2-a019-d39e3a2a66fe None admin] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:08:26.592390 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90d6d12c-2b84-48e2-a019-d39e3a2a66fe None admin] PATH_INFO: `/v3/roles/member` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:08:26.603615 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-90d6d12c-2b84-48e2-a019-d39e3a2a66fe None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 13 04:08:26.608535 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application [None req-90d6d12c-2b84-48e2-a019-d39e3a2a66fe None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Oct 13 04:08:26.608535 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 13 04:08:26.608535 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 13 04:08:26.608535 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 13 04:08:26.608535 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 13 04:08:26.608535 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 13 04:08:26.608535 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 13 04:08:26.608535 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 13 04:08:26.608535 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 13 04:08:26.608535 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 13 04:08:26.608535 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 13 04:08:26.608535 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 13 04:08:26.608535 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 13 04:08:26.608535 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 13 04:08:26.608535 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 13 04:08:26.608535 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 13 04:08:26.609433 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 13 04:08:26.609433 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 13 04:08:26.609433 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 13 04:08:26.609433 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 13 04:08:26.609433 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 13 04:08:26.609433 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 13 04:08:26.609433 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 13 04:08:26.609433 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 13 04:08:26.609433 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 13 04:08:26.609433 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application return self.get_or_create( Oct 13 04:08:26.609433 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 13 04:08:26.609433 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application with Lock( Oct 13 04:08:26.609433 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 13 04:08:26.609433 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application return self._enter() Oct 13 04:08:26.609433 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 13 04:08:26.609433 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 13 04:08:26.610243 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 13 04:08:26.610243 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application return self.creator() Oct 13 04:08:26.610243 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 13 04:08:26.610243 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application created_value = creator( Oct 13 04:08:26.610243 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 13 04:08:26.610243 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 13 04:08:26.610243 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 13 04:08:26.610243 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 13 04:08:26.610243 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 13 04:08:26.610243 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 13 04:08:26.610243 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Oct 13 04:08:26.610243 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application Oct 13 04:08:26.610243 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 102/200] 10.222.0.27 () {56 vars in 1099 bytes} [Thu Oct 13 04:08:26 2022] GET /identity/v3/roles/member => generated 84 bytes in 170 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 13 04:08:26.615655 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bfbab003-cc4f-4733-8232-498282229d14 None admin] Authenticating user token {{(pid=81408) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:08:26.731588 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bfbab003-cc4f-4733-8232-498282229d14 None admin] Validating token access rules against request {{(pid=81408) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:08:26.743785 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG dbcounter [-] [81408] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81408) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:08:26.769478 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b26308b3-6cdd-4da0-b4aa-a3fb2a7c9e22 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81408) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:08:26.770092 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b26308b3-6cdd-4da0-b4aa-a3fb2a7c9e22 None admin] REQUEST_METHOD: `GET` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:08:26.770401 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b26308b3-6cdd-4da0-b4aa-a3fb2a7c9e22 None admin] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:08:26.770681 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b26308b3-6cdd-4da0-b4aa-a3fb2a7c9e22 None admin] PATH_INFO: `/v3/roles` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:08:26.771355 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b26308b3-6cdd-4da0-b4aa-a3fb2a7c9e22 None admin] RBAC: Adding query filter params (name=member) {{(pid=81408) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 13 04:08:26.771658 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b26308b3-6cdd-4da0-b4aa-a3fb2a7c9e22 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:08:26.773113 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b26308b3-6cdd-4da0-b4aa-a3fb2a7c9e22 None admin] RBAC: Authorization granted {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:08:26.781719 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 99/201] 10.222.0.27 () {56 vars in 1101 bytes} [Thu Oct 13 04:08:26 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 167 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 13 04:08:26.787457 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-90d6d12c-2b84-48e2-a019-d39e3a2a66fe None admin] Authenticating user token {{(pid=81409) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:08:26.892048 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-90d6d12c-2b84-48e2-a019-d39e3a2a66fe None admin] Validating token access rules against request {{(pid=81409) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:08:26.933493 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-615a38f3-cd1b-4826-8eb4-ae21c8d8b586 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81409) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:08:26.934129 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-615a38f3-cd1b-4826-8eb4-ae21c8d8b586 None admin] REQUEST_METHOD: `GET` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:08:26.934430 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-615a38f3-cd1b-4826-8eb4-ae21c8d8b586 None admin] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:08:26.934697 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-615a38f3-cd1b-4826-8eb4-ae21c8d8b586 None admin] PATH_INFO: `/v3/users/5e337c24b5934eb88fbc565d2c0dc71d` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:08:26.935854 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-615a38f3-cd1b-4826-8eb4-ae21c8d8b586 None admin] RBAC: Authorizing `identity:get_user(user_id=5e337c24b5934eb88fbc565d2c0dc71d)` {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:08:26.937299 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-615a38f3-cd1b-4826-8eb4-ae21c8d8b586 None admin] RBAC: Authorization granted {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:08:26.938933 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 103/202] 10.222.0.27 () {56 vars in 1177 bytes} [Thu Oct 13 04:08:26 2022] GET /identity/v3/users/5e337c24b5934eb88fbc565d2c0dc71d => generated 297 bytes in 152 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 13 04:08:26.944503 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b26308b3-6cdd-4da0-b4aa-a3fb2a7c9e22 None admin] Authenticating user token {{(pid=81408) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:08:27.050138 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b26308b3-6cdd-4da0-b4aa-a3fb2a7c9e22 None admin] Validating token access rules against request {{(pid=81408) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:08:27.087693 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ef57bba7-2806-4f5f-8416-ebd049f7c5db None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81408) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:08:27.088551 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef57bba7-2806-4f5f-8416-ebd049f7c5db None admin] REQUEST_METHOD: `GET` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:08:27.089058 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef57bba7-2806-4f5f-8416-ebd049f7c5db None admin] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:08:27.089476 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef57bba7-2806-4f5f-8416-ebd049f7c5db None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:08:27.135079 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG dbcounter [-] [81408] Writing DB stats keystone:SELECT=100 {{(pid=81408) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:08:27.164015 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ef57bba7-2806-4f5f-8416-ebd049f7c5db None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:08:27.165796 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ef57bba7-2806-4f5f-8416-ebd049f7c5db None admin] RBAC: Authorization granted {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:08:27.267696 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 100/203] 10.222.0.27 () {58 vars in 1282 bytes} [Thu Oct 13 04:08:26 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 324 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Oct 13 04:08:27.273540 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-615a38f3-cd1b-4826-8eb4-ae21c8d8b586 None admin] Authenticating user token {{(pid=81409) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:08:27.278571 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG dbcounter [-] [81409] Writing DB stats keystone:SELECT=100 {{(pid=81409) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:08:27.380337 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-615a38f3-cd1b-4826-8eb4-ae21c8d8b586 None admin] Validating token access rules against request {{(pid=81409) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:08:27.424783 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-16eccdf1-5161-4323-bd7e-719bfded557e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81409) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:08:27.425433 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16eccdf1-5161-4323-bd7e-719bfded557e None admin] REQUEST_METHOD: `GET` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:08:27.425713 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16eccdf1-5161-4323-bd7e-719bfded557e None admin] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:08:27.425984 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16eccdf1-5161-4323-bd7e-719bfded557e None admin] PATH_INFO: `/v3/projects/0114058aecc645ea8b47ae6d1301c17d` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:08:27.427155 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-16eccdf1-5161-4323-bd7e-719bfded557e None admin] RBAC: Authorizing `identity:get_project(project_id=0114058aecc645ea8b47ae6d1301c17d)` {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:08:27.428601 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-16eccdf1-5161-4323-bd7e-719bfded557e None admin] RBAC: Authorization granted {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:08:27.431016 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 104/204] 10.222.0.27 () {56 vars in 1186 bytes} [Thu Oct 13 04:08:27 2022] GET /identity/v3/projects/0114058aecc645ea8b47ae6d1301c17d => generated 302 bytes in 157 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 13 04:08:27.436867 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ef57bba7-2806-4f5f-8416-ebd049f7c5db None admin] Authenticating user token {{(pid=81408) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:08:27.511349 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG dbcounter [-] [81408] Writing DB stats keystone:SELECT=100 {{(pid=81408) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:08:27.543379 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ef57bba7-2806-4f5f-8416-ebd049f7c5db None admin] Validating token access rules against request {{(pid=81408) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:08:27.583380 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4f9775e4-aa4f-4a66-843a-cabfa0e280b9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81408) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:08:27.584326 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f9775e4-aa4f-4a66-843a-cabfa0e280b9 None admin] REQUEST_METHOD: `GET` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:08:27.584828 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f9775e4-aa4f-4a66-843a-cabfa0e280b9 None admin] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:08:27.585292 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f9775e4-aa4f-4a66-843a-cabfa0e280b9 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:08:27.586246 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4f9775e4-aa4f-4a66-843a-cabfa0e280b9 None admin] RBAC: Adding query filter params (role.id=cc56c4ae13a7424ea771c15da4b153c1, scope.project.id=0114058aecc645ea8b47ae6d1301c17d, user.id=5e337c24b5934eb88fbc565d2c0dc71d) {{(pid=81408) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 13 04:08:27.586908 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4f9775e4-aa4f-4a66-843a-cabfa0e280b9 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:08:27.588630 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4f9775e4-aa4f-4a66-843a-cabfa0e280b9 None admin] RBAC: Authorization granted {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:08:27.595461 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 101/205] 10.222.0.27 () {56 vars in 1374 bytes} [Thu Oct 13 04:08:27 2022] GET /identity/v3/role_assignments?user.id=5e337c24b5934eb88fbc565d2c0dc71d&scope.project.id=0114058aecc645ea8b47ae6d1301c17d&role.id=cc56c4ae13a7424ea771c15da4b153c1 => generated 608 bytes in 159 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 13 04:08:28.430264 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e2f0050-fae4-418a-b4aa-7e7f3ded15e9 None None] REQUEST_METHOD: `GET` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:08:28.430781 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e2f0050-fae4-418a-b4aa-7e7f3ded15e9 None None] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:08:28.430976 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e2f0050-fae4-418a-b4aa-7e7f3ded15e9 None None] PATH_INFO: `/` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:08:28.432347 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 105/206] 10.222.0.27 () {54 vars in 927 bytes} [Thu Oct 13 04:08:28 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 13 04:08:28.437572 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ba1a0a7-37ae-490a-97af-fef73111c539 None None] REQUEST_METHOD: `POST` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:08:28.437949 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ba1a0a7-37ae-490a-97af-fef73111c539 None None] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:08:28.438118 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ba1a0a7-37ae-490a-97af-fef73111c539 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:08:28.456698 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.auth.core [None req-1ba1a0a7-37ae-490a-97af-fef73111c539 None None] MFA Rules not processed for user `0e2d979563d6421995451a0e86bf3022`. Rule list: `[]` (Enabled: `True`). {{(pid=81408) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 13 04:08:28.457903 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.fernet_utils [None req-1ba1a0a7-37ae-490a-97af-fef73111c539 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81408) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:08:28.478643 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-895ee151-3d58-473e-b212-9d5516a73c12 None None] REQUEST_METHOD: `GET` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:08:28.478927 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-895ee151-3d58-473e-b212-9d5516a73c12 None None] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:08:28.479191 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-895ee151-3d58-473e-b212-9d5516a73c12 None None] PATH_INFO: `/` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:08:28.480559 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 106/207] 10.222.0.27 () {54 vars in 927 bytes} [Thu Oct 13 04:08:28 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Oct 13 04:08:28.485738 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7b2cc44-3416-422b-99e3-6fa88f44939a None None] REQUEST_METHOD: `POST` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:08:28.486095 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7b2cc44-3416-422b-99e3-6fa88f44939a None None] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:08:28.486485 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7b2cc44-3416-422b-99e3-6fa88f44939a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:08:28.505600 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.auth.core [None req-c7b2cc44-3416-422b-99e3-6fa88f44939a None None] MFA Rules not processed for user `0e2d979563d6421995451a0e86bf3022`. Rule list: `[]` (Enabled: `True`). {{(pid=81409) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 13 04:08:28.506824 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.fernet_utils [None req-c7b2cc44-3416-422b-99e3-6fa88f44939a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81409) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:08:28.530779 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 102/208] 10.222.0.27 () {58 vars in 1024 bytes} [Thu Oct 13 04:08:28 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 94 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 13 04:08:28.579601 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 107/209] 10.222.0.27 () {58 vars in 1024 bytes} [Thu Oct 13 04:08:28 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 95 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Oct 13 04:08:28.654996 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8494fd24-2633-4704-81b5-4abe089ab2da None None] REQUEST_METHOD: `GET` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:08:28.655302 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8494fd24-2633-4704-81b5-4abe089ab2da None None] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:08:28.655546 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8494fd24-2633-4704-81b5-4abe089ab2da None None] PATH_INFO: `/` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:08:28.657033 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 103/210] 10.222.0.27 () {54 vars in 927 bytes} [Thu Oct 13 04:08:28 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 13 04:08:28.662123 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85a19e95-a9b4-45dc-88c1-fe28748106f0 None None] REQUEST_METHOD: `POST` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:08:28.662606 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85a19e95-a9b4-45dc-88c1-fe28748106f0 None None] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:08:28.662893 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85a19e95-a9b4-45dc-88c1-fe28748106f0 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:08:28.669956 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG dbcounter [-] [81409] Writing DB stats keystone:SELECT=100 {{(pid=81409) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:08:28.682358 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.auth.core [None req-85a19e95-a9b4-45dc-88c1-fe28748106f0 None None] MFA Rules not processed for user `0e2d979563d6421995451a0e86bf3022`. Rule list: `[]` (Enabled: `True`). {{(pid=81409) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 13 04:08:28.683550 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.fernet_utils [None req-85a19e95-a9b4-45dc-88c1-fe28748106f0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81409) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:08:28.751774 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 108/211] 10.222.0.27 () {58 vars in 1024 bytes} [Thu Oct 13 04:08:28 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 90 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Oct 13 04:08:28.802087 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8494fd24-2633-4704-81b5-4abe089ab2da None None] Authenticating user token {{(pid=81408) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:08:28.847561 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-85a19e95-a9b4-45dc-88c1-fe28748106f0 None None] Authenticating user token {{(pid=81409) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:08:28.872230 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG dbcounter [-] [81408] Writing DB stats keystone:SELECT=100 {{(pid=81408) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:08:28.914417 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8494fd24-2633-4704-81b5-4abe089ab2da None None] Validating token access rules against request {{(pid=81408) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:08:28.948839 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a4279d8f-aa4a-4350-ad2f-f0cba993c2cd None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81408) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:08:28.949709 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4279d8f-aa4a-4350-ad2f-f0cba993c2cd None admin] REQUEST_METHOD: `GET` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:08:28.950172 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4279d8f-aa4a-4350-ad2f-f0cba993c2cd None admin] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:08:28.950480 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4279d8f-aa4a-4350-ad2f-f0cba993c2cd None admin] PATH_INFO: `/v3/roles/member` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:08:28.959220 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-85a19e95-a9b4-45dc-88c1-fe28748106f0 None None] Validating token access rules against request {{(pid=81409) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:08:28.960393 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a4279d8f-aa4a-4350-ad2f-f0cba993c2cd None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 13 04:08:28.965906 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application [None req-a4279d8f-aa4a-4350-ad2f-f0cba993c2cd None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Oct 13 04:08:28.965906 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 13 04:08:28.965906 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 13 04:08:28.965906 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 13 04:08:28.965906 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 13 04:08:28.965906 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 13 04:08:28.965906 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 13 04:08:28.965906 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 13 04:08:28.965906 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 13 04:08:28.965906 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 13 04:08:28.965906 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 13 04:08:28.965906 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 13 04:08:28.965906 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 13 04:08:28.965906 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 13 04:08:28.965906 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 13 04:08:28.965906 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 13 04:08:28.966968 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 13 04:08:28.966968 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 13 04:08:28.966968 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 13 04:08:28.966968 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 13 04:08:28.966968 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 13 04:08:28.966968 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 13 04:08:28.966968 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 13 04:08:28.966968 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 13 04:08:28.966968 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 13 04:08:28.966968 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application return self.get_or_create( Oct 13 04:08:28.966968 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 13 04:08:28.966968 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application with Lock( Oct 13 04:08:28.966968 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 13 04:08:28.966968 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application return self._enter() Oct 13 04:08:28.966968 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 13 04:08:28.966968 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 13 04:08:28.967760 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 13 04:08:28.967760 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application return self.creator() Oct 13 04:08:28.967760 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 13 04:08:28.967760 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application created_value = creator( Oct 13 04:08:28.967760 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 13 04:08:28.967760 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 13 04:08:28.967760 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 13 04:08:28.967760 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 13 04:08:28.967760 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 13 04:08:28.967760 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 13 04:08:28.967760 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Oct 13 04:08:28.967760 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application Oct 13 04:08:28.967760 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 104/212] 10.222.0.27 () {56 vars in 1099 bytes} [Thu Oct 13 04:08:28 2022] GET /identity/v3/roles/member => generated 84 bytes in 165 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 13 04:08:28.972650 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a4279d8f-aa4a-4350-ad2f-f0cba993c2cd None admin] Authenticating user token {{(pid=81408) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:08:28.993999 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG dbcounter [-] [81409] Writing DB stats keystone:SELECT=100 {{(pid=81409) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:08:29.001295 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d510cf0e-b9f1-44ce-a042-60756d144bff None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81409) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:08:29.001926 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d510cf0e-b9f1-44ce-a042-60756d144bff None admin] REQUEST_METHOD: `GET` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:08:29.002312 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d510cf0e-b9f1-44ce-a042-60756d144bff None admin] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:08:29.002489 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d510cf0e-b9f1-44ce-a042-60756d144bff None admin] PATH_INFO: `/v3/domains/default` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:08:29.003686 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d510cf0e-b9f1-44ce-a042-60756d144bff None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:08:29.005246 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d510cf0e-b9f1-44ce-a042-60756d144bff None admin] RBAC: Authorization granted {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:08:29.007156 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 109/213] 10.222.0.27 () {56 vars in 1108 bytes} [Thu Oct 13 04:08:28 2022] GET /identity/v3/domains/default => generated 199 bytes in 160 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 13 04:08:29.012809 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d510cf0e-b9f1-44ce-a042-60756d144bff None admin] Authenticating user token {{(pid=81409) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:08:29.078818 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a4279d8f-aa4a-4350-ad2f-f0cba993c2cd None admin] Validating token access rules against request {{(pid=81408) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:08:29.104957 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG dbcounter [-] [81408] Writing DB stats keystone:SELECT=100 {{(pid=81408) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:08:29.112535 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d510cf0e-b9f1-44ce-a042-60756d144bff None admin] Validating token access rules against request {{(pid=81409) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:08:29.114376 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-46a64be0-7b4d-4d91-9da8-f2841478318b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81408) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:08:29.114988 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46a64be0-7b4d-4d91-9da8-f2841478318b None admin] REQUEST_METHOD: `GET` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:08:29.115266 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46a64be0-7b4d-4d91-9da8-f2841478318b None admin] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:08:29.115536 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46a64be0-7b4d-4d91-9da8-f2841478318b None admin] PATH_INFO: `/v3/roles` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:08:29.116200 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-46a64be0-7b4d-4d91-9da8-f2841478318b None admin] RBAC: Adding query filter params (name=member) {{(pid=81408) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 13 04:08:29.116500 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-46a64be0-7b4d-4d91-9da8-f2841478318b None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:08:29.117886 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-46a64be0-7b4d-4d91-9da8-f2841478318b None admin] RBAC: Authorization granted {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:08:29.126702 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 105/214] 10.222.0.27 () {56 vars in 1101 bytes} [Thu Oct 13 04:08:28 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 154 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 13 04:08:29.130542 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-46a64be0-7b4d-4d91-9da8-f2841478318b None admin] Authenticating user token {{(pid=81408) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:08:29.146303 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6da7ceba-61b4-4b19-ab72-22f291afe5c2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81409) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:08:29.146906 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6da7ceba-61b4-4b19-ab72-22f291afe5c2 None admin] REQUEST_METHOD: `POST` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:08:29.147175 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6da7ceba-61b4-4b19-ab72-22f291afe5c2 None admin] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:08:29.147439 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6da7ceba-61b4-4b19-ab72-22f291afe5c2 None admin] PATH_INFO: `/v3/users` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:08:29.148080 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6da7ceba-61b4-4b19-ab72-22f291afe5c2 None admin] RBAC: Authorizing `identity:create_user()` {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:08:29.149599 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6da7ceba-61b4-4b19-ab72-22f291afe5c2 None admin] RBAC: Authorization granted {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:08:29.170864 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 110/215] 10.222.0.27 () {60 vars in 1132 bytes} [Thu Oct 13 04:08:29 2022] POST /identity/v3/users => generated 293 bytes in 158 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 13 04:08:29.171746 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6da7ceba-61b4-4b19-ab72-22f291afe5c2 None admin] Authenticating user token {{(pid=81409) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:08:29.235817 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG dbcounter [-] [81409] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81409) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:08:29.251905 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-46a64be0-7b4d-4d91-9da8-f2841478318b None admin] Validating token access rules against request {{(pid=81408) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:08:29.288630 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-acfe229c-baa3-4c8d-96e7-18c93378c342 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81408) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:08:29.289272 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-acfe229c-baa3-4c8d-96e7-18c93378c342 None admin] REQUEST_METHOD: `GET` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:08:29.289545 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-acfe229c-baa3-4c8d-96e7-18c93378c342 None admin] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:08:29.290384 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-acfe229c-baa3-4c8d-96e7-18c93378c342 None admin] PATH_INFO: `/v3/roles/reader` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:08:29.295964 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6da7ceba-61b4-4b19-ab72-22f291afe5c2 None admin] Validating token access rules against request {{(pid=81409) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:08:29.302470 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-acfe229c-baa3-4c8d-96e7-18c93378c342 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 13 04:08:29.308564 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application [None req-acfe229c-baa3-4c8d-96e7-18c93378c342 None admin] Could not find role: reader.: keystone.exception.RoleNotFound: Could not find role: reader. Oct 13 04:08:29.308564 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 13 04:08:29.308564 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 13 04:08:29.308564 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 13 04:08:29.308564 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 13 04:08:29.308564 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 13 04:08:29.308564 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 13 04:08:29.308564 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 13 04:08:29.308564 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 13 04:08:29.308564 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 13 04:08:29.308564 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 13 04:08:29.308564 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 13 04:08:29.308564 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 13 04:08:29.308564 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 13 04:08:29.308564 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 13 04:08:29.308564 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 13 04:08:29.309471 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 13 04:08:29.309471 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 13 04:08:29.309471 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 13 04:08:29.309471 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 13 04:08:29.309471 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 13 04:08:29.309471 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 13 04:08:29.309471 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 13 04:08:29.309471 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 13 04:08:29.309471 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 13 04:08:29.309471 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application return self.get_or_create( Oct 13 04:08:29.309471 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 13 04:08:29.309471 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application with Lock( Oct 13 04:08:29.309471 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 13 04:08:29.309471 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application return self._enter() Oct 13 04:08:29.309471 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 13 04:08:29.309471 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 13 04:08:29.310322 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 13 04:08:29.310322 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application return self.creator() Oct 13 04:08:29.310322 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 13 04:08:29.310322 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application created_value = creator( Oct 13 04:08:29.310322 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 13 04:08:29.310322 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 13 04:08:29.310322 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 13 04:08:29.310322 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 13 04:08:29.310322 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 13 04:08:29.310322 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 13 04:08:29.310322 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: reader. Oct 13 04:08:29.310322 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application Oct 13 04:08:29.310322 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 106/216] 10.222.0.27 () {56 vars in 1099 bytes} [Thu Oct 13 04:08:29 2022] GET /identity/v3/roles/reader => generated 84 bytes in 182 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 13 04:08:29.315757 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-acfe229c-baa3-4c8d-96e7-18c93378c342 None admin] Authenticating user token {{(pid=81408) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:08:29.338580 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f020d345-48ec-4dcf-a2cb-6bc7848365c6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81409) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:08:29.339270 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f020d345-48ec-4dcf-a2cb-6bc7848365c6 None admin] REQUEST_METHOD: `GET` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:08:29.339543 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f020d345-48ec-4dcf-a2cb-6bc7848365c6 None admin] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:08:29.339808 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f020d345-48ec-4dcf-a2cb-6bc7848365c6 None admin] PATH_INFO: `/v3/users/6f6fd03072dd4a7caefb1086e9def656` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:08:29.358675 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f020d345-48ec-4dcf-a2cb-6bc7848365c6 None admin] RBAC: Authorizing `identity:get_user(user_id=6f6fd03072dd4a7caefb1086e9def656)` {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:08:29.360137 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f020d345-48ec-4dcf-a2cb-6bc7848365c6 None admin] RBAC: Authorization granted {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:08:29.362233 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 111/217] 10.222.0.27 () {56 vars in 1177 bytes} [Thu Oct 13 04:08:29 2022] GET /identity/v3/users/6f6fd03072dd4a7caefb1086e9def656 => generated 293 bytes in 190 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 13 04:08:29.368955 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f020d345-48ec-4dcf-a2cb-6bc7848365c6 None admin] Authenticating user token {{(pid=81409) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:08:29.374744 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG dbcounter [-] [81408] Writing DB stats keystone:SELECT=100 {{(pid=81408) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:08:29.437408 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-acfe229c-baa3-4c8d-96e7-18c93378c342 None admin] Validating token access rules against request {{(pid=81408) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:08:29.476572 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c270ee8c-afb3-420f-ae5d-140a66be7709 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81408) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:08:29.477234 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c270ee8c-afb3-420f-ae5d-140a66be7709 None admin] REQUEST_METHOD: `GET` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:08:29.478173 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c270ee8c-afb3-420f-ae5d-140a66be7709 None admin] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:08:29.478493 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c270ee8c-afb3-420f-ae5d-140a66be7709 None admin] PATH_INFO: `/v3/roles` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:08:29.479357 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c270ee8c-afb3-420f-ae5d-140a66be7709 None admin] RBAC: Adding query filter params (name=reader) {{(pid=81408) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 13 04:08:29.479668 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c270ee8c-afb3-420f-ae5d-140a66be7709 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:08:29.481234 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c270ee8c-afb3-420f-ae5d-140a66be7709 None admin] RBAC: Authorization granted {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:08:29.491075 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 107/218] 10.222.0.27 () {56 vars in 1101 bytes} [Thu Oct 13 04:08:29 2022] GET /identity/v3/roles?name=reader => generated 341 bytes in 176 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 13 04:08:29.497121 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c270ee8c-afb3-420f-ae5d-140a66be7709 None admin] Authenticating user token {{(pid=81408) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:08:29.512661 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f020d345-48ec-4dcf-a2cb-6bc7848365c6 None admin] Validating token access rules against request {{(pid=81409) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:08:29.531708 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG dbcounter [-] [81409] Writing DB stats keystone:SELECT=100 {{(pid=81409) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:08:29.561962 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-82d0936c-56e6-4be0-aa3f-fda0676e0d7d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81409) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:08:29.562658 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82d0936c-56e6-4be0-aa3f-fda0676e0d7d None admin] REQUEST_METHOD: `GET` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:08:29.562983 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82d0936c-56e6-4be0-aa3f-fda0676e0d7d None admin] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:08:29.563264 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82d0936c-56e6-4be0-aa3f-fda0676e0d7d None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:08:29.564060 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-82d0936c-56e6-4be0-aa3f-fda0676e0d7d None admin] RBAC: Adding query filter params (role.id=cc56c4ae13a7424ea771c15da4b153c1, user.id=6f6fd03072dd4a7caefb1086e9def656, scope.system=all) {{(pid=81409) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 13 04:08:29.564374 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-82d0936c-56e6-4be0-aa3f-fda0676e0d7d None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:08:29.565820 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-82d0936c-56e6-4be0-aa3f-fda0676e0d7d None admin] RBAC: Authorization granted {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:08:29.571447 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 112/219] 10.222.0.27 () {56 vars in 1308 bytes} [Thu Oct 13 04:08:29 2022] GET /identity/v3/role_assignments?user.id=6f6fd03072dd4a7caefb1086e9def656&scope.system=all&role.id=cc56c4ae13a7424ea771c15da4b153c1 => generated 226 bytes in 204 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 13 04:08:29.627074 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c270ee8c-afb3-420f-ae5d-140a66be7709 None admin] Validating token access rules against request {{(pid=81408) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:08:29.639919 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG dbcounter [-] [81408] Writing DB stats keystone:SELECT=100 {{(pid=81408) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:08:29.667931 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ed0d64b3-6334-45d6-8060-0d946c560fc8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81408) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:08:29.668647 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed0d64b3-6334-45d6-8060-0d946c560fc8 None admin] REQUEST_METHOD: `GET` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:08:29.668925 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed0d64b3-6334-45d6-8060-0d946c560fc8 None admin] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:08:29.669199 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed0d64b3-6334-45d6-8060-0d946c560fc8 None admin] PATH_INFO: `/v3/users/8ad7a8e540384e4688c9dccfd1dcc631` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:08:29.670545 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ed0d64b3-6334-45d6-8060-0d946c560fc8 None admin] RBAC: Authorizing `identity:get_user(user_id=8ad7a8e540384e4688c9dccfd1dcc631)` {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:08:29.672047 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ed0d64b3-6334-45d6-8060-0d946c560fc8 None admin] RBAC: Authorization granted {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:08:29.674071 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 108/220] 10.222.0.27 () {56 vars in 1177 bytes} [Thu Oct 13 04:08:29 2022] GET /identity/v3/users/8ad7a8e540384e4688c9dccfd1dcc631 => generated 297 bytes in 177 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 13 04:08:29.679417 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-82d0936c-56e6-4be0-aa3f-fda0676e0d7d None admin] Authenticating user token {{(pid=81409) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:08:29.793672 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-82d0936c-56e6-4be0-aa3f-fda0676e0d7d None admin] Validating token access rules against request {{(pid=81409) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:08:29.830172 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e6e7839-9cd7-4920-b880-eefbdfb1dbe3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81409) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:08:29.830808 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e6e7839-9cd7-4920-b880-eefbdfb1dbe3 None admin] REQUEST_METHOD: `GET` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:08:29.831085 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e6e7839-9cd7-4920-b880-eefbdfb1dbe3 None admin] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:08:29.831355 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e6e7839-9cd7-4920-b880-eefbdfb1dbe3 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:08:29.883301 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG dbcounter [-] [81409] Writing DB stats keystone:SELECT=100 {{(pid=81409) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:08:29.906142 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1e6e7839-9cd7-4920-b880-eefbdfb1dbe3 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:08:29.907635 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1e6e7839-9cd7-4920-b880-eefbdfb1dbe3 None admin] RBAC: Authorization granted {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:08:30.016273 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 113/221] 10.222.0.27 () {58 vars in 1282 bytes} [Thu Oct 13 04:08:29 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 337 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Oct 13 04:08:30.023282 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ed0d64b3-6334-45d6-8060-0d946c560fc8 None admin] Authenticating user token {{(pid=81408) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:08:30.135633 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ed0d64b3-6334-45d6-8060-0d946c560fc8 None admin] Validating token access rules against request {{(pid=81408) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:08:30.175860 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-59806e43-712f-428a-b968-6d4716bbe98c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81408) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:08:30.176513 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59806e43-712f-428a-b968-6d4716bbe98c None admin] REQUEST_METHOD: `GET` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:08:30.176798 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59806e43-712f-428a-b968-6d4716bbe98c None admin] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:08:30.177071 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59806e43-712f-428a-b968-6d4716bbe98c None admin] PATH_INFO: `/v3/projects/0114058aecc645ea8b47ae6d1301c17d` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:08:30.178230 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-59806e43-712f-428a-b968-6d4716bbe98c None admin] RBAC: Authorizing `identity:get_project(project_id=0114058aecc645ea8b47ae6d1301c17d)` {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:08:30.179716 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-59806e43-712f-428a-b968-6d4716bbe98c None admin] RBAC: Authorization granted {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:08:30.182191 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 109/222] 10.222.0.27 () {56 vars in 1186 bytes} [Thu Oct 13 04:08:30 2022] GET /identity/v3/projects/0114058aecc645ea8b47ae6d1301c17d => generated 302 bytes in 159 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 13 04:08:30.188097 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e6e7839-9cd7-4920-b880-eefbdfb1dbe3 None admin] Authenticating user token {{(pid=81409) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:08:30.274106 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG dbcounter [-] [81409] Writing DB stats keystone:SELECT=100 {{(pid=81409) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:08:30.298166 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e6e7839-9cd7-4920-b880-eefbdfb1dbe3 None admin] Validating token access rules against request {{(pid=81409) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:08:30.335823 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0140be8f-e219-451f-aa24-24d8f5f6e0d3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81409) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:08:30.336444 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0140be8f-e219-451f-aa24-24d8f5f6e0d3 None admin] REQUEST_METHOD: `GET` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:08:30.337249 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0140be8f-e219-451f-aa24-24d8f5f6e0d3 None admin] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:08:30.337523 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0140be8f-e219-451f-aa24-24d8f5f6e0d3 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:08:30.338335 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0140be8f-e219-451f-aa24-24d8f5f6e0d3 None admin] RBAC: Adding query filter params (role.id=2306a441792b49f19af2d869acf5b3b6, scope.project.id=0114058aecc645ea8b47ae6d1301c17d, user.id=8ad7a8e540384e4688c9dccfd1dcc631) {{(pid=81409) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 13 04:08:30.338647 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0140be8f-e219-451f-aa24-24d8f5f6e0d3 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:08:30.340116 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0140be8f-e219-451f-aa24-24d8f5f6e0d3 None admin] RBAC: Authorization granted {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:08:30.346567 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 114/223] 10.222.0.27 () {56 vars in 1374 bytes} [Thu Oct 13 04:08:30 2022] GET /identity/v3/role_assignments?user.id=8ad7a8e540384e4688c9dccfd1dcc631&scope.project.id=0114058aecc645ea8b47ae6d1301c17d&role.id=2306a441792b49f19af2d869acf5b3b6 => generated 608 bytes in 159 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 13 04:08:31.518948 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca9d0a45-02fa-4fdc-855e-9ea2e1ccaf1e None None] REQUEST_METHOD: `GET` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:08:31.519504 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca9d0a45-02fa-4fdc-855e-9ea2e1ccaf1e None None] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:08:31.519504 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca9d0a45-02fa-4fdc-855e-9ea2e1ccaf1e None None] PATH_INFO: `/` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:08:31.520913 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 110/224] 10.222.0.27 () {54 vars in 927 bytes} [Thu Oct 13 04:08:31 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 13 04:08:31.527078 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb75a896-30dc-4afa-be08-336b7ee9e9e8 None None] REQUEST_METHOD: `POST` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:08:31.527609 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb75a896-30dc-4afa-be08-336b7ee9e9e8 None None] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:08:31.527609 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb75a896-30dc-4afa-be08-336b7ee9e9e8 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:08:31.546964 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.auth.core [None req-eb75a896-30dc-4afa-be08-336b7ee9e9e8 None None] MFA Rules not processed for user `0e2d979563d6421995451a0e86bf3022`. Rule list: `[]` (Enabled: `True`). {{(pid=81409) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 13 04:08:31.548160 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.fernet_utils [None req-eb75a896-30dc-4afa-be08-336b7ee9e9e8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81409) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:08:31.616501 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 115/225] 10.222.0.27 () {58 vars in 1024 bytes} [Thu Oct 13 04:08:31 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 90 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 13 04:08:31.630349 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c3f3ee8-6d32-4e2e-8a69-70f882d0d60a None None] REQUEST_METHOD: `GET` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:08:31.630640 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c3f3ee8-6d32-4e2e-8a69-70f882d0d60a None None] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:08:31.630897 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c3f3ee8-6d32-4e2e-8a69-70f882d0d60a None None] PATH_INFO: `/` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:08:31.632362 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 111/226] 10.222.0.27 () {54 vars in 927 bytes} [Thu Oct 13 04:08:31 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 13 04:08:31.637741 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a25e6b46-c224-42cb-b2a7-5b90c51deff9 None None] REQUEST_METHOD: `POST` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:08:31.638023 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a25e6b46-c224-42cb-b2a7-5b90c51deff9 None None] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:08:31.638325 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a25e6b46-c224-42cb-b2a7-5b90c51deff9 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:08:31.656868 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.auth.core [None req-a25e6b46-c224-42cb-b2a7-5b90c51deff9 None None] MFA Rules not processed for user `0e2d979563d6421995451a0e86bf3022`. Rule list: `[]` (Enabled: `True`). {{(pid=81409) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 13 04:08:31.658030 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.fernet_utils [None req-a25e6b46-c224-42cb-b2a7-5b90c51deff9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81409) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:08:31.729928 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 116/227] 10.222.0.27 () {58 vars in 1024 bytes} [Thu Oct 13 04:08:31 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 93 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 13 04:08:31.799410 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ecf4e344-f3eb-4ffa-9f4a-3e93b1029943 None None] REQUEST_METHOD: `GET` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:08:31.799722 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ecf4e344-f3eb-4ffa-9f4a-3e93b1029943 None None] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:08:31.799966 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ecf4e344-f3eb-4ffa-9f4a-3e93b1029943 None None] PATH_INFO: `/` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:08:31.801459 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 112/228] 10.222.0.27 () {54 vars in 927 bytes} [Thu Oct 13 04:08:31 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 13 04:08:31.806952 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1bb7ea6c-39d0-49ea-bcb6-0fa172764ceb None None] REQUEST_METHOD: `POST` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:08:31.807241 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1bb7ea6c-39d0-49ea-bcb6-0fa172764ceb None None] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:08:31.807517 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1bb7ea6c-39d0-49ea-bcb6-0fa172764ceb None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:08:31.813715 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG dbcounter [-] [81409] Writing DB stats keystone:SELECT=100 {{(pid=81409) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:08:31.826729 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.auth.core [None req-1bb7ea6c-39d0-49ea-bcb6-0fa172764ceb None None] MFA Rules not processed for user `0e2d979563d6421995451a0e86bf3022`. Rule list: `[]` (Enabled: `True`). {{(pid=81409) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 13 04:08:31.827933 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.fernet_utils [None req-1bb7ea6c-39d0-49ea-bcb6-0fa172764ceb None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81409) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:08:31.882994 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ecf4e344-f3eb-4ffa-9f4a-3e93b1029943 None None] Authenticating user token {{(pid=81408) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:08:31.896559 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 117/229] 10.222.0.27 () {58 vars in 1024 bytes} [Thu Oct 13 04:08:31 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 91 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 13 04:08:31.932231 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG dbcounter [-] [81408] Writing DB stats keystone:SELECT=100 {{(pid=81408) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:08:31.989394 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ecf4e344-f3eb-4ffa-9f4a-3e93b1029943 None None] Validating token access rules against request {{(pid=81408) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:08:32.006246 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1bb7ea6c-39d0-49ea-bcb6-0fa172764ceb None None] Authenticating user token {{(pid=81409) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:08:32.025311 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7f15d405-4083-47c5-8fec-f14281d88cd5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81408) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:08:32.025948 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f15d405-4083-47c5-8fec-f14281d88cd5 None admin] REQUEST_METHOD: `GET` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:08:32.026243 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f15d405-4083-47c5-8fec-f14281d88cd5 None admin] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:08:32.026519 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f15d405-4083-47c5-8fec-f14281d88cd5 None admin] PATH_INFO: `/v3/roles/reader` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:08:32.038146 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7f15d405-4083-47c5-8fec-f14281d88cd5 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 13 04:08:32.043060 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application [None req-7f15d405-4083-47c5-8fec-f14281d88cd5 None admin] Could not find role: reader.: keystone.exception.RoleNotFound: Could not find role: reader. Oct 13 04:08:32.043060 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 13 04:08:32.043060 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 13 04:08:32.043060 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 13 04:08:32.043060 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 13 04:08:32.043060 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 13 04:08:32.043060 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 13 04:08:32.043060 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 13 04:08:32.043060 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 13 04:08:32.043060 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 13 04:08:32.043060 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 13 04:08:32.043060 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 13 04:08:32.043060 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 13 04:08:32.043060 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 13 04:08:32.043060 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 13 04:08:32.043060 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 13 04:08:32.043940 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 13 04:08:32.043940 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 13 04:08:32.043940 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 13 04:08:32.043940 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 13 04:08:32.043940 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 13 04:08:32.043940 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 13 04:08:32.043940 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 13 04:08:32.043940 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 13 04:08:32.043940 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 13 04:08:32.043940 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application return self.get_or_create( Oct 13 04:08:32.043940 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 13 04:08:32.043940 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application with Lock( Oct 13 04:08:32.043940 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 13 04:08:32.043940 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application return self._enter() Oct 13 04:08:32.043940 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 13 04:08:32.043940 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 13 04:08:32.044712 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 13 04:08:32.044712 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application return self.creator() Oct 13 04:08:32.044712 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 13 04:08:32.044712 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application created_value = creator( Oct 13 04:08:32.044712 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 13 04:08:32.044712 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 13 04:08:32.044712 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 13 04:08:32.044712 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 13 04:08:32.044712 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 13 04:08:32.044712 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 13 04:08:32.044712 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: reader. Oct 13 04:08:32.044712 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application Oct 13 04:08:32.044712 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 113/230] 10.222.0.27 () {56 vars in 1099 bytes} [Thu Oct 13 04:08:31 2022] GET /identity/v3/roles/reader => generated 84 bytes in 162 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Oct 13 04:08:32.049761 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7f15d405-4083-47c5-8fec-f14281d88cd5 None admin] Authenticating user token {{(pid=81408) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:08:32.127728 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1bb7ea6c-39d0-49ea-bcb6-0fa172764ceb None None] Validating token access rules against request {{(pid=81409) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:08:32.151927 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7f15d405-4083-47c5-8fec-f14281d88cd5 None admin] Validating token access rules against request {{(pid=81408) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:08:32.159999 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG dbcounter [-] [81408] Writing DB stats keystone:SELECT=100 {{(pid=81408) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:08:32.160595 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG dbcounter [-] [81409] Writing DB stats keystone:SELECT=100 {{(pid=81409) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:08:32.167350 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-36039f54-37fc-4e12-aab7-022abaad8197 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81409) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:08:32.167986 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36039f54-37fc-4e12-aab7-022abaad8197 None admin] REQUEST_METHOD: `GET` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:08:32.168264 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36039f54-37fc-4e12-aab7-022abaad8197 None admin] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:08:32.168539 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36039f54-37fc-4e12-aab7-022abaad8197 None admin] PATH_INFO: `/v3/domains/default` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:08:32.169919 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-36039f54-37fc-4e12-aab7-022abaad8197 None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:08:32.171254 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-36039f54-37fc-4e12-aab7-022abaad8197 None admin] RBAC: Authorization granted {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:08:32.172846 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 118/231] 10.222.0.27 () {56 vars in 1108 bytes} [Thu Oct 13 04:08:32 2022] GET /identity/v3/domains/default => generated 199 bytes in 167 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 13 04:08:32.173603 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-36039f54-37fc-4e12-aab7-022abaad8197 None admin] Authenticating user token {{(pid=81409) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:08:32.191247 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0545fd75-62f0-4ad8-9923-b5a690fad87f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81408) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:08:32.191856 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0545fd75-62f0-4ad8-9923-b5a690fad87f None admin] REQUEST_METHOD: `GET` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:08:32.192135 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0545fd75-62f0-4ad8-9923-b5a690fad87f None admin] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:08:32.192410 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0545fd75-62f0-4ad8-9923-b5a690fad87f None admin] PATH_INFO: `/v3/roles` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:08:32.193072 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0545fd75-62f0-4ad8-9923-b5a690fad87f None admin] RBAC: Adding query filter params (name=reader) {{(pid=81408) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 13 04:08:32.193374 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0545fd75-62f0-4ad8-9923-b5a690fad87f None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:08:32.194817 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0545fd75-62f0-4ad8-9923-b5a690fad87f None admin] RBAC: Authorization granted {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:08:32.203334 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 114/232] 10.222.0.27 () {56 vars in 1101 bytes} [Thu Oct 13 04:08:32 2022] GET /identity/v3/roles?name=reader => generated 341 bytes in 154 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 13 04:08:32.204097 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0545fd75-62f0-4ad8-9923-b5a690fad87f None admin] Authenticating user token {{(pid=81408) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:08:32.272701 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-36039f54-37fc-4e12-aab7-022abaad8197 None admin] Validating token access rules against request {{(pid=81409) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:08:32.305458 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0545fd75-62f0-4ad8-9923-b5a690fad87f None admin] Validating token access rules against request {{(pid=81408) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:08:32.308075 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-41968e17-cb27-4b2f-89bc-9cb2612d1e1b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81409) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:08:32.308700 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41968e17-cb27-4b2f-89bc-9cb2612d1e1b None admin] REQUEST_METHOD: `GET` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:08:32.308979 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41968e17-cb27-4b2f-89bc-9cb2612d1e1b None admin] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:08:32.309249 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41968e17-cb27-4b2f-89bc-9cb2612d1e1b None admin] PATH_INFO: `/v3/roles/member` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:08:32.319283 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-41968e17-cb27-4b2f-89bc-9cb2612d1e1b None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 13 04:08:32.324287 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application [None req-41968e17-cb27-4b2f-89bc-9cb2612d1e1b None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Oct 13 04:08:32.324287 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 13 04:08:32.324287 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 13 04:08:32.324287 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 13 04:08:32.324287 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 13 04:08:32.324287 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 13 04:08:32.324287 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 13 04:08:32.324287 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 13 04:08:32.324287 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 13 04:08:32.324287 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 13 04:08:32.324287 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 13 04:08:32.324287 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 13 04:08:32.324287 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 13 04:08:32.324287 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 13 04:08:32.324287 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 13 04:08:32.324287 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 13 04:08:32.325154 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 13 04:08:32.325154 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 13 04:08:32.325154 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 13 04:08:32.325154 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 13 04:08:32.325154 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 13 04:08:32.325154 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 13 04:08:32.325154 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 13 04:08:32.325154 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 13 04:08:32.325154 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 13 04:08:32.325154 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application return self.get_or_create( Oct 13 04:08:32.325154 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 13 04:08:32.325154 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application with Lock( Oct 13 04:08:32.325154 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 13 04:08:32.325154 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application return self._enter() Oct 13 04:08:32.325154 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 13 04:08:32.325154 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 13 04:08:32.325923 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 13 04:08:32.325923 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application return self.creator() Oct 13 04:08:32.325923 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 13 04:08:32.325923 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application created_value = creator( Oct 13 04:08:32.325923 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 13 04:08:32.325923 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 13 04:08:32.325923 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 13 04:08:32.325923 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 13 04:08:32.325923 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 13 04:08:32.325923 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 13 04:08:32.325923 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Oct 13 04:08:32.325923 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application Oct 13 04:08:32.325923 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 119/233] 10.222.0.27 () {56 vars in 1099 bytes} [Thu Oct 13 04:08:32 2022] GET /identity/v3/roles/member => generated 84 bytes in 152 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 13 04:08:32.326628 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-41968e17-cb27-4b2f-89bc-9cb2612d1e1b None admin] Authenticating user token {{(pid=81409) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:08:32.340586 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9bff0122-f94f-4439-aeba-4373f844e4e0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81408) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:08:32.341196 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9bff0122-f94f-4439-aeba-4373f844e4e0 None admin] REQUEST_METHOD: `POST` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:08:32.341474 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9bff0122-f94f-4439-aeba-4373f844e4e0 None admin] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:08:32.341747 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9bff0122-f94f-4439-aeba-4373f844e4e0 None admin] PATH_INFO: `/v3/groups` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:08:32.342407 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9bff0122-f94f-4439-aeba-4373f844e4e0 None admin] RBAC: Authorizing `identity:create_group()` {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:08:32.343847 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9bff0122-f94f-4439-aeba-4373f844e4e0 None admin] RBAC: Authorization granted {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:08:32.360136 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 115/234] 10.222.0.27 () {60 vars in 1134 bytes} [Thu Oct 13 04:08:32 2022] POST /identity/v3/groups => generated 229 bytes in 156 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 13 04:08:32.360875 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9bff0122-f94f-4439-aeba-4373f844e4e0 None admin] Authenticating user token {{(pid=81408) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:08:32.397239 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG dbcounter [-] [81408] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81408) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:08:32.400212 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG dbcounter [-] [81409] Writing DB stats keystone:SELECT=100 {{(pid=81409) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:08:32.453041 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-41968e17-cb27-4b2f-89bc-9cb2612d1e1b None admin] Validating token access rules against request {{(pid=81409) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:08:32.491602 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f5de6bb1-afb1-476f-a356-6da5bab98309 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81409) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:08:32.492635 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5de6bb1-afb1-476f-a356-6da5bab98309 None admin] REQUEST_METHOD: `GET` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:08:32.493101 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5de6bb1-afb1-476f-a356-6da5bab98309 None admin] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:08:32.493372 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5de6bb1-afb1-476f-a356-6da5bab98309 None admin] PATH_INFO: `/v3/users/0bcd00d1131e4d8caebc3d3f8c6f8777` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:08:32.498582 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9bff0122-f94f-4439-aeba-4373f844e4e0 None admin] Validating token access rules against request {{(pid=81408) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:08:32.508686 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f5de6bb1-afb1-476f-a356-6da5bab98309 None admin] RBAC: Authorizing `identity:get_user(user_id=0bcd00d1131e4d8caebc3d3f8c6f8777)` {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:08:32.510219 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f5de6bb1-afb1-476f-a356-6da5bab98309 None admin] RBAC: Authorization granted {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:08:32.511964 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 120/235] 10.222.0.27 () {56 vars in 1177 bytes} [Thu Oct 13 04:08:32 2022] GET /identity/v3/users/0bcd00d1131e4d8caebc3d3f8c6f8777 => generated 293 bytes in 186 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 13 04:08:32.518146 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f5de6bb1-afb1-476f-a356-6da5bab98309 None admin] Authenticating user token {{(pid=81409) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:08:32.539314 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-949a237c-3bfa-472b-b7d5-9b42b619061a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81408) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:08:32.540679 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-949a237c-3bfa-472b-b7d5-9b42b619061a None admin] REQUEST_METHOD: `GET` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:08:32.542823 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-949a237c-3bfa-472b-b7d5-9b42b619061a None admin] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:08:32.542823 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-949a237c-3bfa-472b-b7d5-9b42b619061a None admin] PATH_INFO: `/v3/roles` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:08:32.542823 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-949a237c-3bfa-472b-b7d5-9b42b619061a None admin] RBAC: Adding query filter params (name=member) {{(pid=81408) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 13 04:08:32.542823 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-949a237c-3bfa-472b-b7d5-9b42b619061a None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:08:32.543921 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-949a237c-3bfa-472b-b7d5-9b42b619061a None admin] RBAC: Authorization granted {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:08:32.552720 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 116/236] 10.222.0.27 () {56 vars in 1101 bytes} [Thu Oct 13 04:08:32 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 192 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 13 04:08:32.558761 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-949a237c-3bfa-472b-b7d5-9b42b619061a None admin] Authenticating user token {{(pid=81408) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:08:32.638709 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f5de6bb1-afb1-476f-a356-6da5bab98309 None admin] Validating token access rules against request {{(pid=81409) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:08:32.656742 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG dbcounter [-] [81409] Writing DB stats keystone:SELECT=100 {{(pid=81409) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:08:32.661771 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG dbcounter [-] [81408] Writing DB stats keystone:SELECT=100 {{(pid=81408) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:08:32.672634 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-949a237c-3bfa-472b-b7d5-9b42b619061a None admin] Validating token access rules against request {{(pid=81408) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:08:32.677019 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-66a92343-fc6a-4358-a151-07f2a8ffa615 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81409) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:08:32.677631 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66a92343-fc6a-4358-a151-07f2a8ffa615 None admin] REQUEST_METHOD: `GET` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:08:32.677906 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66a92343-fc6a-4358-a151-07f2a8ffa615 None admin] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:08:32.678175 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66a92343-fc6a-4358-a151-07f2a8ffa615 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:08:32.679007 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-66a92343-fc6a-4358-a151-07f2a8ffa615 None admin] RBAC: Adding query filter params (role.id=2306a441792b49f19af2d869acf5b3b6, user.id=0bcd00d1131e4d8caebc3d3f8c6f8777, scope.system=all) {{(pid=81409) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 13 04:08:32.679318 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-66a92343-fc6a-4358-a151-07f2a8ffa615 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:08:32.681207 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-66a92343-fc6a-4358-a151-07f2a8ffa615 None admin] RBAC: Authorization granted {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:08:32.687327 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 121/237] 10.222.0.27 () {56 vars in 1308 bytes} [Thu Oct 13 04:08:32 2022] GET /identity/v3/role_assignments?user.id=0bcd00d1131e4d8caebc3d3f8c6f8777&scope.system=all&role.id=2306a441792b49f19af2d869acf5b3b6 => generated 226 bytes in 170 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 13 04:08:32.708838 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9ee85db3-a1e7-492f-9f17-a2ba9a59f0b4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81408) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:08:32.709491 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ee85db3-a1e7-492f-9f17-a2ba9a59f0b4 None admin] REQUEST_METHOD: `GET` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:08:32.709767 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ee85db3-a1e7-492f-9f17-a2ba9a59f0b4 None admin] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:08:32.710037 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ee85db3-a1e7-492f-9f17-a2ba9a59f0b4 None admin] PATH_INFO: `/v3/users/6f6fd03072dd4a7caefb1086e9def656` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:08:32.711277 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9ee85db3-a1e7-492f-9f17-a2ba9a59f0b4 None admin] RBAC: Authorizing `identity:get_user(user_id=6f6fd03072dd4a7caefb1086e9def656)` {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:08:32.713505 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9ee85db3-a1e7-492f-9f17-a2ba9a59f0b4 None admin] RBAC: Authorization granted {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:08:32.715657 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 117/238] 10.222.0.27 () {56 vars in 1177 bytes} [Thu Oct 13 04:08:32 2022] GET /identity/v3/users/6f6fd03072dd4a7caefb1086e9def656 => generated 293 bytes in 157 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 13 04:08:32.721260 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-66a92343-fc6a-4358-a151-07f2a8ffa615 None admin] Authenticating user token {{(pid=81409) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:08:32.831723 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-66a92343-fc6a-4358-a151-07f2a8ffa615 None admin] Validating token access rules against request {{(pid=81409) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:08:32.870963 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c3d41c27-67ec-426e-bfc9-e660b6530aff None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81409) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:08:32.871622 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c3d41c27-67ec-426e-bfc9-e660b6530aff None admin] REQUEST_METHOD: `PUT` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:08:32.871895 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c3d41c27-67ec-426e-bfc9-e660b6530aff None admin] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:08:32.872155 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c3d41c27-67ec-426e-bfc9-e660b6530aff None admin] PATH_INFO: `/v3/system/users/6f6fd03072dd4a7caefb1086e9def656/roles/cc56c4ae13a7424ea771c15da4b153c1` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:08:32.873562 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c3d41c27-67ec-426e-bfc9-e660b6530aff None admin] RBAC: Authorizing `identity:create_system_grant_for_user(user_id=6f6fd03072dd4a7caefb1086e9def656, role_id=cc56c4ae13a7424ea771c15da4b153c1)` {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:08:32.874970 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c3d41c27-67ec-426e-bfc9-e660b6530aff None admin] RBAC: Authorization granted {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:08:32.884289 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 122/239] 10.222.0.27 () {58 vars in 1334 bytes} [Thu Oct 13 04:08:32 2022] PUT /identity/v3/system/users/6f6fd03072dd4a7caefb1086e9def656/roles/cc56c4ae13a7424ea771c15da4b153c1 => generated 0 bytes in 164 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 13 04:08:34.543936 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9fc0c42-712a-4b8b-b718-8cfa5e801fcf None None] REQUEST_METHOD: `GET` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:08:34.544434 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9fc0c42-712a-4b8b-b718-8cfa5e801fcf None None] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:08:34.544505 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9fc0c42-712a-4b8b-b718-8cfa5e801fcf None None] PATH_INFO: `/` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:08:34.545956 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 118/240] 10.222.0.27 () {54 vars in 927 bytes} [Thu Oct 13 04:08:34 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Oct 13 04:08:34.550975 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65c133f8-ed2b-4bf0-a117-65b43cc51c7d None None] REQUEST_METHOD: `POST` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:08:34.551385 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65c133f8-ed2b-4bf0-a117-65b43cc51c7d None None] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:08:34.551561 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65c133f8-ed2b-4bf0-a117-65b43cc51c7d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:08:34.569517 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.auth.core [None req-65c133f8-ed2b-4bf0-a117-65b43cc51c7d None None] MFA Rules not processed for user `0e2d979563d6421995451a0e86bf3022`. Rule list: `[]` (Enabled: `True`). {{(pid=81409) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 13 04:08:34.570728 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.fernet_utils [None req-65c133f8-ed2b-4bf0-a117-65b43cc51c7d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81409) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:08:34.615787 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG dbcounter [-] [81409] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81409) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:08:34.642069 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 123/241] 10.222.0.27 () {58 vars in 1024 bytes} [Thu Oct 13 04:08:34 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 92 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 13 04:08:34.842366 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eeb8e2b0-8782-42c1-a199-77a7d528f223 None None] REQUEST_METHOD: `GET` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:08:34.842746 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eeb8e2b0-8782-42c1-a199-77a7d528f223 None None] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:08:34.842923 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eeb8e2b0-8782-42c1-a199-77a7d528f223 None None] PATH_INFO: `/` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:08:34.844405 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 119/242] 10.222.0.27 () {54 vars in 927 bytes} [Thu Oct 13 04:08:34 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 13 04:08:34.849753 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4decfbe-c9d7-4657-aa48-8182b9a3cbe4 None None] REQUEST_METHOD: `POST` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:08:34.850036 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4decfbe-c9d7-4657-aa48-8182b9a3cbe4 None None] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:08:34.850338 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4decfbe-c9d7-4657-aa48-8182b9a3cbe4 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:08:34.869348 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.auth.core [None req-a4decfbe-c9d7-4657-aa48-8182b9a3cbe4 None None] MFA Rules not processed for user `0e2d979563d6421995451a0e86bf3022`. Rule list: `[]` (Enabled: `True`). {{(pid=81409) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 13 04:08:34.870550 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.fernet_utils [None req-a4decfbe-c9d7-4657-aa48-8182b9a3cbe4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81409) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:08:34.907853 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eeb8e2b0-8782-42c1-a199-77a7d528f223 None None] Authenticating user token {{(pid=81408) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:08:34.944617 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 124/243] 10.222.0.27 () {58 vars in 1024 bytes} [Thu Oct 13 04:08:34 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 96 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 13 04:08:35.016720 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eeb8e2b0-8782-42c1-a199-77a7d528f223 None None] Validating token access rules against request {{(pid=81408) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:08:35.043951 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6bdb1a9-9bec-4691-935d-3760cd536057 None None] REQUEST_METHOD: `GET` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:08:35.044233 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6bdb1a9-9bec-4691-935d-3760cd536057 None None] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:08:35.044505 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6bdb1a9-9bec-4691-935d-3760cd536057 None None] PATH_INFO: `/` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:08:35.045861 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 125/244] 10.222.0.27 () {54 vars in 927 bytes} [Thu Oct 13 04:08:35 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Oct 13 04:08:35.050624 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5eea608d-ebd9-4f0a-8a3f-1bc27f7453a7 None None] REQUEST_METHOD: `POST` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:08:35.050897 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5eea608d-ebd9-4f0a-8a3f-1bc27f7453a7 None None] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:08:35.051160 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5eea608d-ebd9-4f0a-8a3f-1bc27f7453a7 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:08:35.052903 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c319b5b7-ac3c-4a7a-88d9-4fcd4b1de2bf None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81408) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:08:35.053736 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c319b5b7-ac3c-4a7a-88d9-4fcd4b1de2bf None admin] REQUEST_METHOD: `GET` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:08:35.054184 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c319b5b7-ac3c-4a7a-88d9-4fcd4b1de2bf None admin] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:08:35.054666 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c319b5b7-ac3c-4a7a-88d9-4fcd4b1de2bf None admin] PATH_INFO: `/v3/domains/default` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:08:35.055975 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c319b5b7-ac3c-4a7a-88d9-4fcd4b1de2bf None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:08:35.057685 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c319b5b7-ac3c-4a7a-88d9-4fcd4b1de2bf None admin] RBAC: Authorization granted {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:08:35.063779 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 120/245] 10.222.0.27 () {56 vars in 1108 bytes} [Thu Oct 13 04:08:34 2022] GET /identity/v3/domains/default => generated 199 bytes in 155 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 13 04:08:35.064908 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c319b5b7-ac3c-4a7a-88d9-4fcd4b1de2bf None admin] Authenticating user token {{(pid=81408) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:08:35.069513 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.auth.core [None req-5eea608d-ebd9-4f0a-8a3f-1bc27f7453a7 None None] MFA Rules not processed for user `0e2d979563d6421995451a0e86bf3022`. Rule list: `[]` (Enabled: `True`). {{(pid=81409) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 13 04:08:35.070924 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.fernet_utils [None req-5eea608d-ebd9-4f0a-8a3f-1bc27f7453a7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81409) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:08:35.095380 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG dbcounter [-] [81408] Writing DB stats keystone:SELECT=100 {{(pid=81408) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:08:35.146255 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 126/246] 10.222.0.27 () {58 vars in 1024 bytes} [Thu Oct 13 04:08:35 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 95 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Oct 13 04:08:35.176085 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c319b5b7-ac3c-4a7a-88d9-4fcd4b1de2bf None admin] Validating token access rules against request {{(pid=81408) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:08:35.211145 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-60f8a689-5574-42c1-9ae1-408613edb4a1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81408) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:08:35.211761 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60f8a689-5574-42c1-9ae1-408613edb4a1 None admin] REQUEST_METHOD: `POST` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:08:35.212034 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60f8a689-5574-42c1-9ae1-408613edb4a1 None admin] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:08:35.212305 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60f8a689-5574-42c1-9ae1-408613edb4a1 None admin] PATH_INFO: `/v3/groups` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:08:35.212932 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-60f8a689-5574-42c1-9ae1-408613edb4a1 None admin] RBAC: Authorizing `identity:create_group()` {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:08:35.214427 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-60f8a689-5574-42c1-9ae1-408613edb4a1 None admin] RBAC: Authorization granted {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:08:35.217198 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5eea608d-ebd9-4f0a-8a3f-1bc27f7453a7 None None] Authenticating user token {{(pid=81409) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:08:35.234706 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 121/247] 10.222.0.27 () {60 vars in 1134 bytes} [Thu Oct 13 04:08:35 2022] POST /identity/v3/groups => generated 226 bytes in 170 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 13 04:08:35.263826 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG dbcounter [-] [81409] Writing DB stats keystone:SELECT=100 {{(pid=81409) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:08:35.338487 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5eea608d-ebd9-4f0a-8a3f-1bc27f7453a7 None None] Validating token access rules against request {{(pid=81409) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:08:35.376736 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ab2832cd-c3ad-46dd-877f-9d3ae3538ed5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81409) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:08:35.377390 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab2832cd-c3ad-46dd-877f-9d3ae3538ed5 None admin] REQUEST_METHOD: `GET` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:08:35.377781 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab2832cd-c3ad-46dd-877f-9d3ae3538ed5 None admin] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:08:35.378184 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab2832cd-c3ad-46dd-877f-9d3ae3538ed5 None admin] PATH_INFO: `/v3/roles/reader` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:08:35.389881 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ab2832cd-c3ad-46dd-877f-9d3ae3538ed5 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 13 04:08:35.395232 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application [None req-ab2832cd-c3ad-46dd-877f-9d3ae3538ed5 None admin] Could not find role: reader.: keystone.exception.RoleNotFound: Could not find role: reader. Oct 13 04:08:35.395232 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 13 04:08:35.395232 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 13 04:08:35.395232 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 13 04:08:35.395232 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 13 04:08:35.395232 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 13 04:08:35.395232 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 13 04:08:35.395232 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 13 04:08:35.395232 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 13 04:08:35.395232 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 13 04:08:35.395232 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 13 04:08:35.395232 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 13 04:08:35.395232 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 13 04:08:35.395232 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 13 04:08:35.395232 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 13 04:08:35.395232 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 13 04:08:35.396311 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 13 04:08:35.396311 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 13 04:08:35.396311 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 13 04:08:35.396311 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 13 04:08:35.396311 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 13 04:08:35.396311 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 13 04:08:35.396311 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 13 04:08:35.396311 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 13 04:08:35.396311 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 13 04:08:35.396311 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application return self.get_or_create( Oct 13 04:08:35.396311 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 13 04:08:35.396311 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application with Lock( Oct 13 04:08:35.396311 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 13 04:08:35.396311 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application return self._enter() Oct 13 04:08:35.396311 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 13 04:08:35.396311 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 13 04:08:35.397273 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 13 04:08:35.397273 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application return self.creator() Oct 13 04:08:35.397273 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 13 04:08:35.397273 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application created_value = creator( Oct 13 04:08:35.397273 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 13 04:08:35.397273 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 13 04:08:35.397273 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 13 04:08:35.397273 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 13 04:08:35.397273 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 13 04:08:35.397273 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 13 04:08:35.397273 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: reader. Oct 13 04:08:35.397273 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application Oct 13 04:08:35.403314 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 127/248] 10.222.0.27 () {56 vars in 1099 bytes} [Thu Oct 13 04:08:35 2022] GET /identity/v3/roles/reader => generated 84 bytes in 181 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Oct 13 04:08:35.404853 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-60f8a689-5574-42c1-9ae1-408613edb4a1 None admin] Authenticating user token {{(pid=81408) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:08:35.424174 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ab2832cd-c3ad-46dd-877f-9d3ae3538ed5 None admin] Authenticating user token {{(pid=81409) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:08:35.505736 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG dbcounter [-] [81408] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81408) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:08:35.524017 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-60f8a689-5574-42c1-9ae1-408613edb4a1 None admin] Validating token access rules against request {{(pid=81408) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:08:35.554219 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG dbcounter [-] [81409] Writing DB stats keystone:SELECT=100 {{(pid=81409) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:08:35.563107 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ab2832cd-c3ad-46dd-877f-9d3ae3538ed5 None admin] Validating token access rules against request {{(pid=81409) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:08:35.570245 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d28099e7-3ecc-400c-b68b-4b523f090208 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81408) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:08:35.570997 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d28099e7-3ecc-400c-b68b-4b523f090208 None admin] REQUEST_METHOD: `GET` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:08:35.571434 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d28099e7-3ecc-400c-b68b-4b523f090208 None admin] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:08:35.571803 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d28099e7-3ecc-400c-b68b-4b523f090208 None admin] PATH_INFO: `/v3/roles` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:08:35.572584 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d28099e7-3ecc-400c-b68b-4b523f090208 None admin] RBAC: Adding query filter params (name=reader) {{(pid=81408) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 13 04:08:35.572894 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d28099e7-3ecc-400c-b68b-4b523f090208 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:08:35.574380 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d28099e7-3ecc-400c-b68b-4b523f090208 None admin] RBAC: Authorization granted {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:08:35.584488 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 122/249] 10.222.0.27 () {56 vars in 1101 bytes} [Thu Oct 13 04:08:35 2022] GET /identity/v3/roles?name=reader => generated 341 bytes in 180 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 13 04:08:35.591430 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d28099e7-3ecc-400c-b68b-4b523f090208 None admin] Authenticating user token {{(pid=81408) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:08:35.606558 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c34536f4-ed96-4296-a8ac-5cea07d74a75 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81409) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:08:35.607110 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c34536f4-ed96-4296-a8ac-5cea07d74a75 None admin] REQUEST_METHOD: `GET` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:08:35.607389 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c34536f4-ed96-4296-a8ac-5cea07d74a75 None admin] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:08:35.607658 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c34536f4-ed96-4296-a8ac-5cea07d74a75 None admin] PATH_INFO: `/v3/roles/member` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:08:35.620832 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c34536f4-ed96-4296-a8ac-5cea07d74a75 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 13 04:08:35.626615 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application [None req-c34536f4-ed96-4296-a8ac-5cea07d74a75 None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Oct 13 04:08:35.626615 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 13 04:08:35.626615 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 13 04:08:35.626615 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 13 04:08:35.626615 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 13 04:08:35.626615 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 13 04:08:35.626615 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 13 04:08:35.626615 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 13 04:08:35.626615 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 13 04:08:35.626615 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 13 04:08:35.626615 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 13 04:08:35.626615 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 13 04:08:35.626615 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 13 04:08:35.626615 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 13 04:08:35.626615 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 13 04:08:35.626615 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 13 04:08:35.627529 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 13 04:08:35.627529 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 13 04:08:35.627529 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 13 04:08:35.627529 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 13 04:08:35.627529 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 13 04:08:35.627529 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 13 04:08:35.627529 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 13 04:08:35.627529 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 13 04:08:35.627529 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 13 04:08:35.627529 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application return self.get_or_create( Oct 13 04:08:35.627529 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 13 04:08:35.627529 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application with Lock( Oct 13 04:08:35.627529 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 13 04:08:35.627529 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application return self._enter() Oct 13 04:08:35.627529 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 13 04:08:35.627529 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 13 04:08:35.628338 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 13 04:08:35.628338 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application return self.creator() Oct 13 04:08:35.628338 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 13 04:08:35.628338 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application created_value = creator( Oct 13 04:08:35.628338 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 13 04:08:35.628338 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 13 04:08:35.628338 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 13 04:08:35.628338 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 13 04:08:35.628338 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 13 04:08:35.628338 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 13 04:08:35.628338 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Oct 13 04:08:35.628338 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application Oct 13 04:08:35.631627 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 128/250] 10.222.0.27 () {56 vars in 1099 bytes} [Thu Oct 13 04:08:35 2022] GET /identity/v3/roles/member => generated 84 bytes in 208 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Oct 13 04:08:35.638604 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c34536f4-ed96-4296-a8ac-5cea07d74a75 None admin] Authenticating user token {{(pid=81409) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:08:35.725197 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d28099e7-3ecc-400c-b68b-4b523f090208 None admin] Validating token access rules against request {{(pid=81408) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:08:35.764948 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a2035d75-24c2-4fe4-9579-698e1ef231be None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81408) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:08:35.766172 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2035d75-24c2-4fe4-9579-698e1ef231be None admin] REQUEST_METHOD: `GET` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:08:35.768691 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2035d75-24c2-4fe4-9579-698e1ef231be None admin] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:08:35.769073 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2035d75-24c2-4fe4-9579-698e1ef231be None admin] PATH_INFO: `/v3/users/0bcd00d1131e4d8caebc3d3f8c6f8777` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:08:35.769319 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c34536f4-ed96-4296-a8ac-5cea07d74a75 None admin] Validating token access rules against request {{(pid=81409) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:08:35.771289 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a2035d75-24c2-4fe4-9579-698e1ef231be None admin] RBAC: Authorizing `identity:get_user(user_id=0bcd00d1131e4d8caebc3d3f8c6f8777)` {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:08:35.772227 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a2035d75-24c2-4fe4-9579-698e1ef231be None admin] RBAC: Authorization granted {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:08:35.774493 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 123/251] 10.222.0.27 () {56 vars in 1177 bytes} [Thu Oct 13 04:08:35 2022] GET /identity/v3/users/0bcd00d1131e4d8caebc3d3f8c6f8777 => generated 293 bytes in 183 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 13 04:08:35.783328 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a2035d75-24c2-4fe4-9579-698e1ef231be None admin] Authenticating user token {{(pid=81408) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:08:35.803784 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG dbcounter [-] [81408] Writing DB stats keystone:SELECT=100 {{(pid=81408) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:08:35.811775 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-14c60739-70b8-44e7-84d3-7f6dff0c93a7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81409) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:08:35.812543 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14c60739-70b8-44e7-84d3-7f6dff0c93a7 None admin] REQUEST_METHOD: `GET` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:08:35.812957 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14c60739-70b8-44e7-84d3-7f6dff0c93a7 None admin] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:08:35.813357 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14c60739-70b8-44e7-84d3-7f6dff0c93a7 None admin] PATH_INFO: `/v3/roles` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:08:35.814192 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-14c60739-70b8-44e7-84d3-7f6dff0c93a7 None admin] RBAC: Adding query filter params (name=member) {{(pid=81409) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 13 04:08:35.814596 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-14c60739-70b8-44e7-84d3-7f6dff0c93a7 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:08:35.816064 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-14c60739-70b8-44e7-84d3-7f6dff0c93a7 None admin] RBAC: Authorization granted {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:08:35.827088 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 129/252] 10.222.0.27 () {56 vars in 1101 bytes} [Thu Oct 13 04:08:35 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 189 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 13 04:08:35.833126 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-14c60739-70b8-44e7-84d3-7f6dff0c93a7 None admin] Authenticating user token {{(pid=81409) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:08:35.856490 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG dbcounter [-] [81409] Writing DB stats keystone:SELECT=100 {{(pid=81409) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:08:35.899237 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a2035d75-24c2-4fe4-9579-698e1ef231be None admin] Validating token access rules against request {{(pid=81408) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:08:35.939481 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-80f5efc7-bb50-4b6c-bf9a-67fef59f112a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81408) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:08:35.940462 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80f5efc7-bb50-4b6c-bf9a-67fef59f112a None admin] REQUEST_METHOD: `PUT` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:08:35.940909 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80f5efc7-bb50-4b6c-bf9a-67fef59f112a None admin] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:08:35.941348 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80f5efc7-bb50-4b6c-bf9a-67fef59f112a None admin] PATH_INFO: `/v3/system/users/0bcd00d1131e4d8caebc3d3f8c6f8777/roles/2306a441792b49f19af2d869acf5b3b6` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:08:35.942988 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-80f5efc7-bb50-4b6c-bf9a-67fef59f112a None admin] RBAC: Authorizing `identity:create_system_grant_for_user(user_id=0bcd00d1131e4d8caebc3d3f8c6f8777, role_id=2306a441792b49f19af2d869acf5b3b6)` {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:08:35.944619 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-80f5efc7-bb50-4b6c-bf9a-67fef59f112a None admin] RBAC: Authorization granted {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:08:35.955730 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 124/253] 10.222.0.27 () {58 vars in 1334 bytes} [Thu Oct 13 04:08:35 2022] PUT /identity/v3/system/users/0bcd00d1131e4d8caebc3d3f8c6f8777/roles/2306a441792b49f19af2d869acf5b3b6 => generated 0 bytes in 173 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 13 04:08:35.970747 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-14c60739-70b8-44e7-84d3-7f6dff0c93a7 None admin] Validating token access rules against request {{(pid=81409) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:08:36.018699 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8fae1764-e9ee-42ff-9114-ae55a9750383 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81409) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:08:36.019647 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8fae1764-e9ee-42ff-9114-ae55a9750383 None admin] REQUEST_METHOD: `GET` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:08:36.020207 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8fae1764-e9ee-42ff-9114-ae55a9750383 None admin] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:08:36.020650 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8fae1764-e9ee-42ff-9114-ae55a9750383 None admin] PATH_INFO: `/v3/users/6f6fd03072dd4a7caefb1086e9def656` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:08:36.022093 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8fae1764-e9ee-42ff-9114-ae55a9750383 None admin] RBAC: Authorizing `identity:get_user(user_id=6f6fd03072dd4a7caefb1086e9def656)` {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:08:36.023898 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8fae1764-e9ee-42ff-9114-ae55a9750383 None admin] RBAC: Authorization granted {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:08:36.026411 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 130/254] 10.222.0.27 () {56 vars in 1177 bytes} [Thu Oct 13 04:08:35 2022] GET /identity/v3/users/6f6fd03072dd4a7caefb1086e9def656 => generated 293 bytes in 194 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 13 04:08:36.033013 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-80f5efc7-bb50-4b6c-bf9a-67fef59f112a None admin] Authenticating user token {{(pid=81408) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:08:36.153865 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG dbcounter [-] [81408] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81408) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:08:36.177681 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-80f5efc7-bb50-4b6c-bf9a-67fef59f112a None admin] Validating token access rules against request {{(pid=81408) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:08:36.215706 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-48881faa-b82e-4dab-b5da-dee99e217ac4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81408) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:08:36.216324 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48881faa-b82e-4dab-b5da-dee99e217ac4 None admin] REQUEST_METHOD: `GET` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:08:36.216603 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48881faa-b82e-4dab-b5da-dee99e217ac4 None admin] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:08:36.216873 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48881faa-b82e-4dab-b5da-dee99e217ac4 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:08:36.217605 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-48881faa-b82e-4dab-b5da-dee99e217ac4 None admin] RBAC: Adding query filter params (role.id=cc56c4ae13a7424ea771c15da4b153c1, user.id=6f6fd03072dd4a7caefb1086e9def656, scope.system=all) {{(pid=81408) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 13 04:08:36.217912 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-48881faa-b82e-4dab-b5da-dee99e217ac4 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:08:36.219513 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-48881faa-b82e-4dab-b5da-dee99e217ac4 None admin] RBAC: Authorization granted {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:08:36.236366 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 125/255] 10.222.0.27 () {56 vars in 1308 bytes} [Thu Oct 13 04:08:36 2022] GET /identity/v3/role_assignments?user.id=6f6fd03072dd4a7caefb1086e9def656&scope.system=all&role.id=cc56c4ae13a7424ea771c15da4b153c1 => generated 510 bytes in 204 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 13 04:08:37.684506 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca5c4f3b-0141-4acd-b422-bcc3d93250c0 None None] REQUEST_METHOD: `GET` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:08:37.684972 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca5c4f3b-0141-4acd-b422-bcc3d93250c0 None None] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:08:37.684972 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca5c4f3b-0141-4acd-b422-bcc3d93250c0 None None] PATH_INFO: `/` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:08:37.686326 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 131/256] 10.222.0.27 () {54 vars in 927 bytes} [Thu Oct 13 04:08:37 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 13 04:08:37.691969 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25d575ff-7e84-4e35-9c2f-d4cbde8a5da2 None None] REQUEST_METHOD: `POST` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:08:37.692854 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25d575ff-7e84-4e35-9c2f-d4cbde8a5da2 None None] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:08:37.693336 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25d575ff-7e84-4e35-9c2f-d4cbde8a5da2 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:08:37.723180 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.auth.core [None req-25d575ff-7e84-4e35-9c2f-d4cbde8a5da2 None None] MFA Rules not processed for user `0e2d979563d6421995451a0e86bf3022`. Rule list: `[]` (Enabled: `True`). {{(pid=81408) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 13 04:08:37.724990 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.fernet_utils [None req-25d575ff-7e84-4e35-9c2f-d4cbde8a5da2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81408) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:08:37.799853 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 126/257] 10.222.0.27 () {58 vars in 1024 bytes} [Thu Oct 13 04:08:37 2022] POST /identity/v3/auth/tokens => generated 914 bytes in 109 msecs (HTTP/1.1 201) 6 headers in 384 bytes (1 switches on core 0) Oct 13 04:08:37.905834 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4cf853d5-8262-4d67-a160-5eb5529c1796 None None] REQUEST_METHOD: `GET` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:08:37.906152 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4cf853d5-8262-4d67-a160-5eb5529c1796 None None] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:08:37.906394 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4cf853d5-8262-4d67-a160-5eb5529c1796 None None] PATH_INFO: `/` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:08:37.907879 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 132/258] 10.222.0.27 () {54 vars in 927 bytes} [Thu Oct 13 04:08:37 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 13 04:08:37.913201 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f905fe90-cfce-4417-825d-90268ec42938 None None] REQUEST_METHOD: `POST` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:08:37.913491 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f905fe90-cfce-4417-825d-90268ec42938 None None] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:08:37.913771 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f905fe90-cfce-4417-825d-90268ec42938 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:08:37.934694 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.auth.core [None req-f905fe90-cfce-4417-825d-90268ec42938 None None] MFA Rules not processed for user `0e2d979563d6421995451a0e86bf3022`. Rule list: `[]` (Enabled: `True`). {{(pid=81408) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 13 04:08:37.936194 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.fernet_utils [None req-f905fe90-cfce-4417-825d-90268ec42938 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81408) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:08:37.944220 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 127/259] 10.222.0.27 () {58 vars in 1024 bytes} [Thu Oct 13 04:08:37 2022] POST /identity/v3/auth/tokens => generated 914 bytes in 32 msecs (HTTP/1.1 201) 6 headers in 384 bytes (1 switches on core 0) Oct 13 04:08:38.316626 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4cf853d5-8262-4d67-a160-5eb5529c1796 None None] Authenticating user token {{(pid=81409) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:08:38.331432 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4cf853d5-8262-4d67-a160-5eb5529c1796 None None] Validating token access rules against request {{(pid=81409) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:08:38.341269 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1681b4b9-c368-4da0-9fbf-c1133b227875 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8ba823f313144da3b683bacef8fd62e0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81409) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:08:38.342228 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1681b4b9-c368-4da0-9fbf-c1133b227875 admin admin] REQUEST_METHOD: `GET` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:08:38.342701 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1681b4b9-c368-4da0-9fbf-c1133b227875 admin admin] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:08:38.343167 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1681b4b9-c368-4da0-9fbf-c1133b227875 admin admin] PATH_INFO: `/v3/roles/anotherrole` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:08:38.358539 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f905fe90-cfce-4417-825d-90268ec42938 None None] Authenticating user token {{(pid=81408) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:08:38.372020 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1681b4b9-c368-4da0-9fbf-c1133b227875 admin admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 13 04:08:38.377447 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application [None req-1681b4b9-c368-4da0-9fbf-c1133b227875 admin admin] Could not find role: anotherrole.: keystone.exception.RoleNotFound: Could not find role: anotherrole. Oct 13 04:08:38.377447 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 13 04:08:38.377447 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 13 04:08:38.377447 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 13 04:08:38.377447 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 13 04:08:38.377447 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 13 04:08:38.377447 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 13 04:08:38.377447 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 13 04:08:38.377447 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 13 04:08:38.377447 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 13 04:08:38.377447 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 13 04:08:38.377447 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 13 04:08:38.377447 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 13 04:08:38.377447 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 13 04:08:38.377447 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 13 04:08:38.377447 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 13 04:08:38.378306 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 13 04:08:38.378306 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 13 04:08:38.378306 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 13 04:08:38.378306 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 13 04:08:38.378306 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 13 04:08:38.378306 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 13 04:08:38.378306 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 13 04:08:38.378306 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 13 04:08:38.378306 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 13 04:08:38.378306 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application return self.get_or_create( Oct 13 04:08:38.378306 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 13 04:08:38.378306 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application with Lock( Oct 13 04:08:38.378306 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 13 04:08:38.378306 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application return self._enter() Oct 13 04:08:38.378306 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 13 04:08:38.378306 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 13 04:08:38.379084 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 13 04:08:38.379084 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application return self.creator() Oct 13 04:08:38.379084 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 13 04:08:38.379084 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application created_value = creator( Oct 13 04:08:38.379084 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 13 04:08:38.379084 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 13 04:08:38.379084 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 13 04:08:38.379084 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 13 04:08:38.379084 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 13 04:08:38.379084 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 13 04:08:38.379084 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: anotherrole. Oct 13 04:08:38.379084 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application Oct 13 04:08:38.382360 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f905fe90-cfce-4417-825d-90268ec42938 None None] Validating token access rules against request {{(pid=81408) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:08:38.384796 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 133/260] 10.222.0.27 () {56 vars in 1135 bytes} [Thu Oct 13 04:08:38 2022] GET /identity/v3/roles/anotherrole => generated 89 bytes in 69 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 13 04:08:38.390593 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1681b4b9-c368-4da0-9fbf-c1133b227875 admin admin] Authenticating user token {{(pid=81409) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:08:38.397043 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-968f84db-e2e3-4b4c-ad4b-cd7543f17061 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8ba823f313144da3b683bacef8fd62e0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81408) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:08:38.397703 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-968f84db-e2e3-4b4c-ad4b-cd7543f17061 admin admin] REQUEST_METHOD: `GET` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:08:38.397976 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-968f84db-e2e3-4b4c-ad4b-cd7543f17061 admin admin] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:08:38.398277 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-968f84db-e2e3-4b4c-ad4b-cd7543f17061 admin admin] PATH_INFO: `/v3/roles/member` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:08:38.403865 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1681b4b9-c368-4da0-9fbf-c1133b227875 admin admin] Validating token access rules against request {{(pid=81409) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:08:38.412875 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e07a42c2-e32b-4173-b5c4-6b4f25e3d576 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8ba823f313144da3b683bacef8fd62e0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81409) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:08:38.413822 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e07a42c2-e32b-4173-b5c4-6b4f25e3d576 admin admin] REQUEST_METHOD: `GET` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:08:38.414318 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e07a42c2-e32b-4173-b5c4-6b4f25e3d576 admin admin] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:08:38.415135 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e07a42c2-e32b-4173-b5c4-6b4f25e3d576 admin admin] PATH_INFO: `/v3/roles` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:08:38.416013 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e07a42c2-e32b-4173-b5c4-6b4f25e3d576 admin admin] RBAC: Adding query filter params (name=anotherrole) {{(pid=81409) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 13 04:08:38.417042 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e07a42c2-e32b-4173-b5c4-6b4f25e3d576 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:08:38.419137 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: WARNING py.warnings [None req-e07a42c2-e32b-4173-b5c4-6b4f25e3d576 admin admin] /usr/local/lib/python3.8/dist-packages/oslo_policy/policy.py:1129: UserWarning: Policy "identity:list_roles": "role:reader and system_scope:all" failed scope check. The token used to make the request was project scoped but the policy requires ['system'] scope. This behavior may change in the future where using the intended scope is required Oct 13 04:08:38.419137 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: warnings.warn(msg) Oct 13 04:08:38.419137 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: Oct 13 04:08:38.420408 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e07a42c2-e32b-4173-b5c4-6b4f25e3d576 admin admin] RBAC: Authorization granted {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:08:38.430227 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-968f84db-e2e3-4b4c-ad4b-cd7543f17061 admin admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 13 04:08:38.430227 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application [None req-968f84db-e2e3-4b4c-ad4b-cd7543f17061 admin admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Oct 13 04:08:38.430227 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 13 04:08:38.430227 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 13 04:08:38.430227 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 13 04:08:38.430227 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 13 04:08:38.430227 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 13 04:08:38.430227 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 13 04:08:38.430227 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 13 04:08:38.430227 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 13 04:08:38.430227 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 13 04:08:38.430227 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 13 04:08:38.430227 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 13 04:08:38.430227 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 13 04:08:38.431577 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 13 04:08:38.431577 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 13 04:08:38.431577 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 13 04:08:38.431577 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 13 04:08:38.431577 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 13 04:08:38.431577 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 13 04:08:38.431577 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 13 04:08:38.431577 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 13 04:08:38.431577 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 13 04:08:38.431577 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 13 04:08:38.431577 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 13 04:08:38.431577 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 13 04:08:38.431577 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application return self.get_or_create( Oct 13 04:08:38.431577 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 13 04:08:38.431577 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application with Lock( Oct 13 04:08:38.431577 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 13 04:08:38.431577 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application return self._enter() Oct 13 04:08:38.432421 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 13 04:08:38.432421 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 13 04:08:38.432421 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 13 04:08:38.432421 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application return self.creator() Oct 13 04:08:38.432421 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 13 04:08:38.432421 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application created_value = creator( Oct 13 04:08:38.432421 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 13 04:08:38.432421 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 13 04:08:38.432421 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 13 04:08:38.432421 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 13 04:08:38.432421 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 13 04:08:38.432421 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 13 04:08:38.432421 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Oct 13 04:08:38.432421 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application Oct 13 04:08:38.432421 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 128/261] 10.222.0.27 () {56 vars in 1120 bytes} [Thu Oct 13 04:08:38 2022] GET /identity/v3/roles/member => generated 84 bytes in 72 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 13 04:08:38.440984 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-968f84db-e2e3-4b4c-ad4b-cd7543f17061 admin admin] Authenticating user token {{(pid=81408) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:08:38.457224 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 134/262] 10.222.0.27 () {56 vars in 1132 bytes} [Thu Oct 13 04:08:38 2022] GET /identity/v3/roles?name=anotherrole => generated 334 bytes in 67 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 13 04:08:38.460514 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-968f84db-e2e3-4b4c-ad4b-cd7543f17061 admin admin] Validating token access rules against request {{(pid=81408) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:08:38.464268 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e07a42c2-e32b-4173-b5c4-6b4f25e3d576 admin admin] Authenticating user token {{(pid=81409) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:08:38.473073 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e8590e12-de98-4c95-80d1-42896bd87726 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8ba823f313144da3b683bacef8fd62e0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81408) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:08:38.473697 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8590e12-de98-4c95-80d1-42896bd87726 admin admin] REQUEST_METHOD: `GET` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:08:38.473967 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8590e12-de98-4c95-80d1-42896bd87726 admin admin] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:08:38.474511 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8590e12-de98-4c95-80d1-42896bd87726 admin admin] PATH_INFO: `/v3/roles` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:08:38.475185 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e8590e12-de98-4c95-80d1-42896bd87726 admin admin] RBAC: Adding query filter params (name=member) {{(pid=81408) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 13 04:08:38.475481 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e8590e12-de98-4c95-80d1-42896bd87726 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:08:38.478776 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: WARNING py.warnings [None req-e8590e12-de98-4c95-80d1-42896bd87726 admin admin] /usr/local/lib/python3.8/dist-packages/oslo_policy/policy.py:1129: UserWarning: Policy "identity:list_roles": "role:reader and system_scope:all" failed scope check. The token used to make the request was project scoped but the policy requires ['system'] scope. This behavior may change in the future where using the intended scope is required Oct 13 04:08:38.478776 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: warnings.warn(msg) Oct 13 04:08:38.478776 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: Oct 13 04:08:38.479889 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e8590e12-de98-4c95-80d1-42896bd87726 admin admin] RBAC: Authorization granted {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:08:38.481827 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e07a42c2-e32b-4173-b5c4-6b4f25e3d576 admin admin] Validating token access rules against request {{(pid=81409) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:08:38.491521 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7be28b96-ae95-437f-87a4-449e43428809 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8ba823f313144da3b683bacef8fd62e0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81409) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:08:38.492530 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7be28b96-ae95-437f-87a4-449e43428809 admin admin] REQUEST_METHOD: `GET` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:08:38.493527 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7be28b96-ae95-437f-87a4-449e43428809 admin admin] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:08:38.494011 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7be28b96-ae95-437f-87a4-449e43428809 admin admin] PATH_INFO: `/v3/groups/d232567016e94eeca7524d2c09a20c3b` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:08:38.498026 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 129/263] 10.222.0.27 () {56 vars in 1122 bytes} [Thu Oct 13 04:08:38 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 59 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 13 04:08:38.503793 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e8590e12-de98-4c95-80d1-42896bd87726 admin admin] Authenticating user token {{(pid=81408) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:08:38.508210 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7be28b96-ae95-437f-87a4-449e43428809 admin admin] RBAC: Authorizing `identity:get_group(group_id=d232567016e94eeca7524d2c09a20c3b)` {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:08:38.509939 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: WARNING py.warnings [None req-7be28b96-ae95-437f-87a4-449e43428809 admin admin] /usr/local/lib/python3.8/dist-packages/oslo_policy/policy.py:1129: UserWarning: Policy "identity:get_group": "(role:reader and system_scope:all) or (role:reader and domain_id:%(target.group.domain_id)s)" failed scope check. The token used to make the request was project scoped but the policy requires ['system', 'domain'] scope. This behavior may change in the future where using the intended scope is required Oct 13 04:08:38.509939 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: warnings.warn(msg) Oct 13 04:08:38.509939 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: Oct 13 04:08:38.511541 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7be28b96-ae95-437f-87a4-449e43428809 admin admin] RBAC: Authorization granted {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:08:38.513767 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 135/264] 10.222.0.27 () {56 vars in 1201 bytes} [Thu Oct 13 04:08:38 2022] GET /identity/v3/groups/d232567016e94eeca7524d2c09a20c3b => generated 226 bytes in 50 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 13 04:08:38.520504 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7be28b96-ae95-437f-87a4-449e43428809 admin admin] Authenticating user token {{(pid=81409) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:08:38.522920 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e8590e12-de98-4c95-80d1-42896bd87726 admin admin] Validating token access rules against request {{(pid=81408) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:08:38.530668 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7429be50-91e9-4cde-86ec-734105802b51 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8ba823f313144da3b683bacef8fd62e0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81408) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:08:38.531378 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7429be50-91e9-4cde-86ec-734105802b51 admin admin] REQUEST_METHOD: `GET` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:08:38.531654 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7429be50-91e9-4cde-86ec-734105802b51 admin admin] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:08:38.531925 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7429be50-91e9-4cde-86ec-734105802b51 admin admin] PATH_INFO: `/v3/groups/d232567016e94eeca7524d2c09a20c3b` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:08:38.534336 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7429be50-91e9-4cde-86ec-734105802b51 admin admin] RBAC: Authorizing `identity:get_group(group_id=d232567016e94eeca7524d2c09a20c3b)` {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:08:38.535436 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: WARNING py.warnings [None req-7429be50-91e9-4cde-86ec-734105802b51 admin admin] /usr/local/lib/python3.8/dist-packages/oslo_policy/policy.py:1129: UserWarning: Policy "identity:get_group": "(role:reader and system_scope:all) or (role:reader and domain_id:%(target.group.domain_id)s)" failed scope check. The token used to make the request was project scoped but the policy requires ['system', 'domain'] scope. This behavior may change in the future where using the intended scope is required Oct 13 04:08:38.535436 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: warnings.warn(msg) Oct 13 04:08:38.535436 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: Oct 13 04:08:38.536710 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7429be50-91e9-4cde-86ec-734105802b51 admin admin] RBAC: Authorization granted {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:08:38.537256 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7be28b96-ae95-437f-87a4-449e43428809 admin admin] Validating token access rules against request {{(pid=81409) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:08:38.538727 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 130/265] 10.222.0.27 () {56 vars in 1201 bytes} [Thu Oct 13 04:08:38 2022] GET /identity/v3/groups/d232567016e94eeca7524d2c09a20c3b => generated 226 bytes in 35 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 13 04:08:38.544147 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7429be50-91e9-4cde-86ec-734105802b51 admin admin] Authenticating user token {{(pid=81408) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:08:38.549718 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fa301a28-d2ec-4d7d-a8d8-b10fb03bd1d7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8ba823f313144da3b683bacef8fd62e0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81409) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:08:38.550667 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa301a28-d2ec-4d7d-a8d8-b10fb03bd1d7 admin admin] REQUEST_METHOD: `GET` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:08:38.551142 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa301a28-d2ec-4d7d-a8d8-b10fb03bd1d7 admin admin] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:08:38.551609 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa301a28-d2ec-4d7d-a8d8-b10fb03bd1d7 admin admin] PATH_INFO: `/v3/projects/eeec419e34e44c1f881179933c828d97` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:08:38.558578 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fa301a28-d2ec-4d7d-a8d8-b10fb03bd1d7 admin admin] RBAC: Authorizing `identity:get_project(project_id=eeec419e34e44c1f881179933c828d97)` {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:08:38.560584 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7429be50-91e9-4cde-86ec-734105802b51 admin admin] Validating token access rules against request {{(pid=81408) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:08:38.561203 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fa301a28-d2ec-4d7d-a8d8-b10fb03bd1d7 admin admin] RBAC: Authorization granted {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:08:38.563153 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 136/266] 10.222.0.27 () {56 vars in 1207 bytes} [Thu Oct 13 04:08:38 2022] GET /identity/v3/projects/eeec419e34e44c1f881179933c828d97 => generated 298 bytes in 43 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 13 04:08:38.568563 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f1e6c1c0-dbfd-486b-a459-c7360ab53064 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8ba823f313144da3b683bacef8fd62e0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81408) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:08:38.569147 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1e6c1c0-dbfd-486b-a459-c7360ab53064 admin admin] REQUEST_METHOD: `GET` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:08:38.569476 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fa301a28-d2ec-4d7d-a8d8-b10fb03bd1d7 admin admin] Authenticating user token {{(pid=81409) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:08:38.569671 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1e6c1c0-dbfd-486b-a459-c7360ab53064 admin admin] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:08:38.569831 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1e6c1c0-dbfd-486b-a459-c7360ab53064 admin admin] PATH_INFO: `/v3/projects/eeec419e34e44c1f881179933c828d97` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:08:38.574943 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f1e6c1c0-dbfd-486b-a459-c7360ab53064 admin admin] RBAC: Authorizing `identity:get_project(project_id=eeec419e34e44c1f881179933c828d97)` {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:08:38.577208 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f1e6c1c0-dbfd-486b-a459-c7360ab53064 admin admin] RBAC: Authorization granted {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:08:38.579004 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 131/267] 10.222.0.27 () {56 vars in 1207 bytes} [Thu Oct 13 04:08:38 2022] GET /identity/v3/projects/eeec419e34e44c1f881179933c828d97 => generated 298 bytes in 35 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 13 04:08:38.584970 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fa301a28-d2ec-4d7d-a8d8-b10fb03bd1d7 admin admin] Validating token access rules against request {{(pid=81409) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:08:38.585106 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f1e6c1c0-dbfd-486b-a459-c7360ab53064 admin admin] Authenticating user token {{(pid=81408) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:08:38.593501 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-461a3b13-21af-482e-961e-12873268c3a6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8ba823f313144da3b683bacef8fd62e0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81409) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:08:38.594449 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-461a3b13-21af-482e-961e-12873268c3a6 admin admin] REQUEST_METHOD: `PUT` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:08:38.594926 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-461a3b13-21af-482e-961e-12873268c3a6 admin admin] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:08:38.595420 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-461a3b13-21af-482e-961e-12873268c3a6 admin admin] PATH_INFO: `/v3/projects/eeec419e34e44c1f881179933c828d97/groups/d232567016e94eeca7524d2c09a20c3b/roles/743995e93fef47fd91204c3d701054b0` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:08:38.600972 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f1e6c1c0-dbfd-486b-a459-c7360ab53064 admin admin] Validating token access rules against request {{(pid=81408) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:08:38.601869 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-461a3b13-21af-482e-961e-12873268c3a6 admin admin] RBAC: Authorizing `identity:create_grant(project_id=eeec419e34e44c1f881179933c828d97, group_id=d232567016e94eeca7524d2c09a20c3b, role_id=743995e93fef47fd91204c3d701054b0)` {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:08:38.603306 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: WARNING py.warnings [None req-461a3b13-21af-482e-961e-12873268c3a6 admin admin] /usr/local/lib/python3.8/dist-packages/oslo_policy/policy.py:1129: UserWarning: Policy "identity:create_grant": "(role:admin and system_scope:all) or ((role:admin and domain_id:%(target.user.domain_id)s and domain_id:%(target.project.domain_id)s) or (role:admin and domain_id:%(target.user.domain_id)s and domain_id:%(target.domain.id)s) or (role:admin and domain_id:%(target.group.domain_id)s and domain_id:%(target.project.domain_id)s) or (role:admin and domain_id:%(target.group.domain_id)s and domain_id:%(target.domain.id)s)) and (domain_id:%(target.role.domain_id)s or None:%(target.role.domain_id)s)" failed scope check. The token used to make the request was project scoped but the policy requires ['system', 'domain'] scope. This behavior may change in the future where using the intended scope is required Oct 13 04:08:38.603306 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: warnings.warn(msg) Oct 13 04:08:38.603306 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: Oct 13 04:08:38.605637 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-461a3b13-21af-482e-961e-12873268c3a6 admin admin] RBAC: Authorization granted {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:08:38.611181 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-754e9a04-5dd9-4e0e-8fea-b1788e3fb76d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8ba823f313144da3b683bacef8fd62e0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81408) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:08:38.611820 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-754e9a04-5dd9-4e0e-8fea-b1788e3fb76d admin admin] REQUEST_METHOD: `PUT` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:08:38.612151 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-754e9a04-5dd9-4e0e-8fea-b1788e3fb76d admin admin] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:08:38.612421 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-754e9a04-5dd9-4e0e-8fea-b1788e3fb76d admin admin] PATH_INFO: `/v3/projects/eeec419e34e44c1f881179933c828d97/groups/d232567016e94eeca7524d2c09a20c3b/roles/cc56c4ae13a7424ea771c15da4b153c1` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:08:38.615533 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-754e9a04-5dd9-4e0e-8fea-b1788e3fb76d admin admin] RBAC: Authorizing `identity:create_grant(project_id=eeec419e34e44c1f881179933c828d97, group_id=d232567016e94eeca7524d2c09a20c3b, role_id=cc56c4ae13a7424ea771c15da4b153c1)` {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:08:38.616636 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: WARNING py.warnings [None req-754e9a04-5dd9-4e0e-8fea-b1788e3fb76d admin admin] /usr/local/lib/python3.8/dist-packages/oslo_policy/policy.py:1129: UserWarning: Policy "identity:create_grant": "(role:admin and system_scope:all) or ((role:admin and domain_id:%(target.user.domain_id)s and domain_id:%(target.project.domain_id)s) or (role:admin and domain_id:%(target.user.domain_id)s and domain_id:%(target.domain.id)s) or (role:admin and domain_id:%(target.group.domain_id)s and domain_id:%(target.project.domain_id)s) or (role:admin and domain_id:%(target.group.domain_id)s and domain_id:%(target.domain.id)s)) and (domain_id:%(target.role.domain_id)s or None:%(target.role.domain_id)s)" failed scope check. The token used to make the request was project scoped but the policy requires ['system', 'domain'] scope. This behavior may change in the future where using the intended scope is required Oct 13 04:08:38.616636 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: warnings.warn(msg) Oct 13 04:08:38.616636 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: Oct 13 04:08:38.619495 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-754e9a04-5dd9-4e0e-8fea-b1788e3fb76d admin admin] RBAC: Authorization granted {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:08:38.631272 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 137/268] 10.222.0.27 () {58 vars in 1463 bytes} [Thu Oct 13 04:08:38 2022] PUT /identity/v3/projects/eeec419e34e44c1f881179933c828d97/groups/d232567016e94eeca7524d2c09a20c3b/roles/743995e93fef47fd91204c3d701054b0 => generated 0 bytes in 63 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 13 04:08:38.645954 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 132/269] 10.222.0.27 () {58 vars in 1463 bytes} [Thu Oct 13 04:08:38 2022] PUT /identity/v3/projects/eeec419e34e44c1f881179933c828d97/groups/d232567016e94eeca7524d2c09a20c3b/roles/cc56c4ae13a7424ea771c15da4b153c1 => generated 0 bytes in 61 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 13 04:08:38.980548 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-015ab760-fe25-43da-9c7d-c9c456356de6 None None] REQUEST_METHOD: `GET` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:08:38.981027 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-015ab760-fe25-43da-9c7d-c9c456356de6 None None] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:08:38.981027 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-015ab760-fe25-43da-9c7d-c9c456356de6 None None] PATH_INFO: `/` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:08:38.982699 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 138/270] 10.222.0.27 () {54 vars in 927 bytes} [Thu Oct 13 04:08:38 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 13 04:08:38.988118 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-876025d9-f98b-4f27-b3db-b2f076ea8a8f None None] REQUEST_METHOD: `POST` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:08:38.988576 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-876025d9-f98b-4f27-b3db-b2f076ea8a8f None None] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:08:38.988648 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-876025d9-f98b-4f27-b3db-b2f076ea8a8f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:08:39.007487 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-982502d4-ae3a-40ea-ba5e-76657439d5a5 None None] REQUEST_METHOD: `GET` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:08:39.007813 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-982502d4-ae3a-40ea-ba5e-76657439d5a5 None None] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:08:39.008094 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-982502d4-ae3a-40ea-ba5e-76657439d5a5 None None] PATH_INFO: `/` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:08:39.010625 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 139/271] 10.222.0.27 () {54 vars in 927 bytes} [Thu Oct 13 04:08:39 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 13 04:08:39.016169 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6a7e4d4-d28e-4e62-a7bc-73e249e28539 None None] REQUEST_METHOD: `POST` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:08:39.016443 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6a7e4d4-d28e-4e62-a7bc-73e249e28539 None None] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:08:39.016709 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6a7e4d4-d28e-4e62-a7bc-73e249e28539 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:08:39.023226 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.auth.core [None req-876025d9-f98b-4f27-b3db-b2f076ea8a8f None None] MFA Rules not processed for user `0e2d979563d6421995451a0e86bf3022`. Rule list: `[]` (Enabled: `True`). {{(pid=81408) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 13 04:08:39.024506 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.fernet_utils [None req-876025d9-f98b-4f27-b3db-b2f076ea8a8f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81408) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:08:39.046373 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.auth.core [None req-b6a7e4d4-d28e-4e62-a7bc-73e249e28539 None None] MFA Rules not processed for user `0e2d979563d6421995451a0e86bf3022`. Rule list: `[]` (Enabled: `True`). {{(pid=81409) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 13 04:08:39.047597 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.fernet_utils [None req-b6a7e4d4-d28e-4e62-a7bc-73e249e28539 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81409) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:08:39.062621 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG dbcounter [-] [81408] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81408) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:08:39.105179 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG dbcounter [-] [81409] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81409) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:08:39.111075 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 133/272] 10.222.0.27 () {58 vars in 1024 bytes} [Thu Oct 13 04:08:38 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 124 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 13 04:08:39.129709 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 140/273] 10.222.0.27 () {58 vars in 1024 bytes} [Thu Oct 13 04:08:39 2022] POST /identity/v3/auth/tokens => generated 914 bytes in 114 msecs (HTTP/1.1 201) 6 headers in 384 bytes (2 switches on core 0) Oct 13 04:08:39.421009 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-876025d9-f98b-4f27-b3db-b2f076ea8a8f None None] Authenticating user token {{(pid=81408) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:08:39.534002 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-876025d9-f98b-4f27-b3db-b2f076ea8a8f None None] Validating token access rules against request {{(pid=81408) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:08:39.573040 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d4f26b7e-6e81-4844-8a48-043c5e62a4e7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81408) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:08:39.573677 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4f26b7e-6e81-4844-8a48-043c5e62a4e7 None admin] REQUEST_METHOD: `GET` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:08:39.573949 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4f26b7e-6e81-4844-8a48-043c5e62a4e7 None admin] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:08:39.574230 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4f26b7e-6e81-4844-8a48-043c5e62a4e7 None admin] PATH_INFO: `/v3/roles/reader` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:08:39.586435 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d4f26b7e-6e81-4844-8a48-043c5e62a4e7 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 13 04:08:39.591998 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application [None req-d4f26b7e-6e81-4844-8a48-043c5e62a4e7 None admin] Could not find role: reader.: keystone.exception.RoleNotFound: Could not find role: reader. Oct 13 04:08:39.591998 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 13 04:08:39.591998 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 13 04:08:39.591998 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 13 04:08:39.591998 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 13 04:08:39.591998 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 13 04:08:39.591998 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 13 04:08:39.591998 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 13 04:08:39.591998 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 13 04:08:39.591998 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 13 04:08:39.591998 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 13 04:08:39.591998 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 13 04:08:39.591998 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 13 04:08:39.591998 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 13 04:08:39.591998 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 13 04:08:39.591998 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 13 04:08:39.592856 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 13 04:08:39.592856 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 13 04:08:39.592856 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 13 04:08:39.592856 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 13 04:08:39.592856 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 13 04:08:39.592856 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 13 04:08:39.592856 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 13 04:08:39.592856 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 13 04:08:39.592856 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 13 04:08:39.592856 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application return self.get_or_create( Oct 13 04:08:39.592856 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 13 04:08:39.592856 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application with Lock( Oct 13 04:08:39.592856 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 13 04:08:39.592856 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application return self._enter() Oct 13 04:08:39.592856 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 13 04:08:39.592856 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 13 04:08:39.593643 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 13 04:08:39.593643 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application return self.creator() Oct 13 04:08:39.593643 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 13 04:08:39.593643 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application created_value = creator( Oct 13 04:08:39.593643 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 13 04:08:39.593643 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 13 04:08:39.593643 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 13 04:08:39.593643 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 13 04:08:39.593643 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 13 04:08:39.593643 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 13 04:08:39.593643 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: reader. Oct 13 04:08:39.593643 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application Oct 13 04:08:39.593643 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 134/274] 10.222.0.27 () {56 vars in 1099 bytes} [Thu Oct 13 04:08:39 2022] GET /identity/v3/roles/reader => generated 84 bytes in 173 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 13 04:08:39.598916 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b6a7e4d4-d28e-4e62-a7bc-73e249e28539 None None] Authenticating user token {{(pid=81409) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:08:39.687490 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d4f26b7e-6e81-4844-8a48-043c5e62a4e7 None admin] Authenticating user token {{(pid=81408) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:08:39.711197 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d4f26b7e-6e81-4844-8a48-043c5e62a4e7 None admin] Validating token access rules against request {{(pid=81408) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:08:39.717816 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-899fe255-5777-4905-b01a-c9e9d4508bb8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8ba823f313144da3b683bacef8fd62e0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81408) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:08:39.718744 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-899fe255-5777-4905-b01a-c9e9d4508bb8 admin admin] REQUEST_METHOD: `GET` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:08:39.719203 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-899fe255-5777-4905-b01a-c9e9d4508bb8 admin admin] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:08:39.719693 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-899fe255-5777-4905-b01a-c9e9d4508bb8 admin admin] PATH_INFO: `/v3/roles/member` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:08:39.719796 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b6a7e4d4-d28e-4e62-a7bc-73e249e28539 None None] Validating token access rules against request {{(pid=81409) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:08:39.753216 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-899fe255-5777-4905-b01a-c9e9d4508bb8 admin admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 13 04:08:39.758476 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application [None req-899fe255-5777-4905-b01a-c9e9d4508bb8 admin admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Oct 13 04:08:39.758476 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 13 04:08:39.758476 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 13 04:08:39.758476 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 13 04:08:39.758476 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 13 04:08:39.758476 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 13 04:08:39.758476 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 13 04:08:39.758476 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 13 04:08:39.758476 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 13 04:08:39.758476 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 13 04:08:39.758476 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 13 04:08:39.758476 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 13 04:08:39.758476 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 13 04:08:39.758476 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 13 04:08:39.758476 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 13 04:08:39.758476 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 13 04:08:39.759855 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 13 04:08:39.759855 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 13 04:08:39.759855 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 13 04:08:39.759855 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 13 04:08:39.759855 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 13 04:08:39.759855 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 13 04:08:39.759855 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 13 04:08:39.759855 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 13 04:08:39.759855 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 13 04:08:39.759855 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application return self.get_or_create( Oct 13 04:08:39.759855 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 13 04:08:39.759855 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application with Lock( Oct 13 04:08:39.759855 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 13 04:08:39.759855 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application return self._enter() Oct 13 04:08:39.759855 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 13 04:08:39.759855 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 13 04:08:39.760646 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 13 04:08:39.760646 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application return self.creator() Oct 13 04:08:39.760646 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 13 04:08:39.760646 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application created_value = creator( Oct 13 04:08:39.760646 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 13 04:08:39.760646 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 13 04:08:39.760646 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 13 04:08:39.760646 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 13 04:08:39.760646 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 13 04:08:39.760646 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 13 04:08:39.760646 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Oct 13 04:08:39.760646 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application Oct 13 04:08:39.766228 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 135/275] 10.222.0.27 () {56 vars in 1120 bytes} [Thu Oct 13 04:08:39 2022] GET /identity/v3/roles/member => generated 84 bytes in 76 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 13 04:08:39.769290 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ab70b854-d719-45c3-8d68-d20291145056 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81409) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:08:39.769921 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab70b854-d719-45c3-8d68-d20291145056 None admin] REQUEST_METHOD: `GET` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:08:39.770108 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-899fe255-5777-4905-b01a-c9e9d4508bb8 admin admin] Authenticating user token {{(pid=81408) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:08:39.770225 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab70b854-d719-45c3-8d68-d20291145056 None admin] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:08:39.770470 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab70b854-d719-45c3-8d68-d20291145056 None admin] PATH_INFO: `/v3/roles` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:08:39.774553 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ab70b854-d719-45c3-8d68-d20291145056 None admin] RBAC: Adding query filter params (name=reader) {{(pid=81409) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 13 04:08:39.774553 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ab70b854-d719-45c3-8d68-d20291145056 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:08:39.774553 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ab70b854-d719-45c3-8d68-d20291145056 None admin] RBAC: Authorization granted {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:08:39.787435 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-899fe255-5777-4905-b01a-c9e9d4508bb8 admin admin] Validating token access rules against request {{(pid=81408) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:08:39.791714 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 141/276] 10.222.0.27 () {56 vars in 1101 bytes} [Thu Oct 13 04:08:39 2022] GET /identity/v3/roles?name=reader => generated 341 bytes in 193 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 13 04:08:39.797669 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ab70b854-d719-45c3-8d68-d20291145056 None admin] Authenticating user token {{(pid=81409) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:08:39.799497 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c9fb37e9-61c0-4f16-89b3-7073f9c0ed2f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8ba823f313144da3b683bacef8fd62e0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81408) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:08:39.800344 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9fb37e9-61c0-4f16-89b3-7073f9c0ed2f admin admin] REQUEST_METHOD: `GET` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:08:39.800840 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9fb37e9-61c0-4f16-89b3-7073f9c0ed2f admin admin] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:08:39.801317 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9fb37e9-61c0-4f16-89b3-7073f9c0ed2f admin admin] PATH_INFO: `/v3/roles` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:08:39.802195 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c9fb37e9-61c0-4f16-89b3-7073f9c0ed2f admin admin] RBAC: Adding query filter params (name=member) {{(pid=81408) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 13 04:08:39.802719 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c9fb37e9-61c0-4f16-89b3-7073f9c0ed2f admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:08:39.804535 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c9fb37e9-61c0-4f16-89b3-7073f9c0ed2f admin admin] RBAC: Authorization granted {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:08:39.820969 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 136/277] 10.222.0.27 () {56 vars in 1122 bytes} [Thu Oct 13 04:08:39 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 52 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 13 04:08:39.827121 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c9fb37e9-61c0-4f16-89b3-7073f9c0ed2f admin admin] Authenticating user token {{(pid=81408) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:08:39.839672 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG dbcounter [-] [81408] Writing DB stats keystone:SELECT=100 {{(pid=81408) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:08:39.844854 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c9fb37e9-61c0-4f16-89b3-7073f9c0ed2f admin admin] Validating token access rules against request {{(pid=81408) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:08:39.853647 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d9b6bf4c-5efa-43f2-8ac9-9b2ce12e7a90 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8ba823f313144da3b683bacef8fd62e0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81408) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:08:39.854633 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9b6bf4c-5efa-43f2-8ac9-9b2ce12e7a90 admin admin] REQUEST_METHOD: `GET` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:08:39.855153 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9b6bf4c-5efa-43f2-8ac9-9b2ce12e7a90 admin admin] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:08:39.855622 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9b6bf4c-5efa-43f2-8ac9-9b2ce12e7a90 admin admin] PATH_INFO: `/v3/groups/d232567016e94eeca7524d2c09a20c3b` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:08:39.858713 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d9b6bf4c-5efa-43f2-8ac9-9b2ce12e7a90 admin admin] RBAC: Authorizing `identity:get_group(group_id=d232567016e94eeca7524d2c09a20c3b)` {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:08:39.860879 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d9b6bf4c-5efa-43f2-8ac9-9b2ce12e7a90 admin admin] RBAC: Authorization granted {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:08:39.862735 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 137/278] 10.222.0.27 () {56 vars in 1201 bytes} [Thu Oct 13 04:08:39 2022] GET /identity/v3/groups/d232567016e94eeca7524d2c09a20c3b => generated 226 bytes in 36 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 13 04:08:39.868325 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d9b6bf4c-5efa-43f2-8ac9-9b2ce12e7a90 admin admin] Authenticating user token {{(pid=81408) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:08:39.882707 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG dbcounter [-] [81409] Writing DB stats keystone:SELECT=100 {{(pid=81409) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:08:39.884056 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d9b6bf4c-5efa-43f2-8ac9-9b2ce12e7a90 admin admin] Validating token access rules against request {{(pid=81408) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:08:39.894086 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8871a14c-6850-40e1-a2c4-231411bbda01 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8ba823f313144da3b683bacef8fd62e0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81408) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:08:39.895086 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8871a14c-6850-40e1-a2c4-231411bbda01 admin admin] REQUEST_METHOD: `GET` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:08:39.895565 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8871a14c-6850-40e1-a2c4-231411bbda01 admin admin] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:08:39.896054 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8871a14c-6850-40e1-a2c4-231411bbda01 admin admin] PATH_INFO: `/v3/projects/0114058aecc645ea8b47ae6d1301c17d` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:08:39.901302 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8871a14c-6850-40e1-a2c4-231411bbda01 admin admin] RBAC: Authorizing `identity:get_project(project_id=0114058aecc645ea8b47ae6d1301c17d)` {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:08:39.903664 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8871a14c-6850-40e1-a2c4-231411bbda01 admin admin] RBAC: Authorization granted {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:08:39.905900 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 138/279] 10.222.0.27 () {56 vars in 1207 bytes} [Thu Oct 13 04:08:39 2022] GET /identity/v3/projects/0114058aecc645ea8b47ae6d1301c17d => generated 302 bytes in 38 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 13 04:08:39.914269 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8871a14c-6850-40e1-a2c4-231411bbda01 admin admin] Authenticating user token {{(pid=81408) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:08:39.928821 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8871a14c-6850-40e1-a2c4-231411bbda01 admin admin] Validating token access rules against request {{(pid=81408) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:08:39.940382 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-48d02b30-5c46-4f5f-a8c5-1450c5a67085 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8ba823f313144da3b683bacef8fd62e0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81408) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:08:39.941360 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48d02b30-5c46-4f5f-a8c5-1450c5a67085 admin admin] REQUEST_METHOD: `PUT` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:08:39.941834 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48d02b30-5c46-4f5f-a8c5-1450c5a67085 admin admin] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:08:39.942345 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48d02b30-5c46-4f5f-a8c5-1450c5a67085 admin admin] PATH_INFO: `/v3/projects/0114058aecc645ea8b47ae6d1301c17d/groups/d232567016e94eeca7524d2c09a20c3b/roles/cc56c4ae13a7424ea771c15da4b153c1` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:08:39.951042 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-48d02b30-5c46-4f5f-a8c5-1450c5a67085 admin admin] RBAC: Authorizing `identity:create_grant(project_id=0114058aecc645ea8b47ae6d1301c17d, group_id=d232567016e94eeca7524d2c09a20c3b, role_id=cc56c4ae13a7424ea771c15da4b153c1)` {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:08:39.954361 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-48d02b30-5c46-4f5f-a8c5-1450c5a67085 admin admin] RBAC: Authorization granted {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:08:39.977661 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 139/280] 10.222.0.27 () {58 vars in 1463 bytes} [Thu Oct 13 04:08:39 2022] PUT /identity/v3/projects/0114058aecc645ea8b47ae6d1301c17d/groups/d232567016e94eeca7524d2c09a20c3b/roles/cc56c4ae13a7424ea771c15da4b153c1 => generated 0 bytes in 64 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 13 04:08:39.983117 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ab70b854-d719-45c3-8d68-d20291145056 None admin] Validating token access rules against request {{(pid=81409) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:08:40.033266 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f8d46e3e-cddf-445b-a489-cf617592358e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81409) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:08:40.033914 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8d46e3e-cddf-445b-a489-cf617592358e None admin] REQUEST_METHOD: `GET` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:08:40.034188 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8d46e3e-cddf-445b-a489-cf617592358e None admin] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:08:40.034477 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8d46e3e-cddf-445b-a489-cf617592358e None admin] PATH_INFO: `/v3/users/0bcd00d1131e4d8caebc3d3f8c6f8777` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:08:40.035647 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f8d46e3e-cddf-445b-a489-cf617592358e None admin] RBAC: Authorizing `identity:get_user(user_id=0bcd00d1131e4d8caebc3d3f8c6f8777)` {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:08:40.037102 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f8d46e3e-cddf-445b-a489-cf617592358e None admin] RBAC: Authorization granted {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:08:40.038732 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 142/281] 10.222.0.27 () {56 vars in 1177 bytes} [Thu Oct 13 04:08:39 2022] GET /identity/v3/users/0bcd00d1131e4d8caebc3d3f8c6f8777 => generated 293 bytes in 242 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 13 04:08:40.045766 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-48d02b30-5c46-4f5f-a8c5-1450c5a67085 admin admin] Authenticating user token {{(pid=81408) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:08:40.187099 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-48d02b30-5c46-4f5f-a8c5-1450c5a67085 admin admin] Validating token access rules against request {{(pid=81408) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:08:40.233288 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-66d4d56d-13cc-44db-b69e-a40fba9f4b62 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81408) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:08:40.234313 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66d4d56d-13cc-44db-b69e-a40fba9f4b62 None admin] REQUEST_METHOD: `GET` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:08:40.234792 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66d4d56d-13cc-44db-b69e-a40fba9f4b62 None admin] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:08:40.235256 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66d4d56d-13cc-44db-b69e-a40fba9f4b62 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:08:40.236193 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-66d4d56d-13cc-44db-b69e-a40fba9f4b62 None admin] RBAC: Adding query filter params (role.id=2306a441792b49f19af2d869acf5b3b6, user.id=0bcd00d1131e4d8caebc3d3f8c6f8777, scope.system=all) {{(pid=81408) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 13 04:08:40.236711 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-66d4d56d-13cc-44db-b69e-a40fba9f4b62 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:08:40.238444 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-66d4d56d-13cc-44db-b69e-a40fba9f4b62 None admin] RBAC: Authorization granted {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:08:40.253173 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 140/282] 10.222.0.27 () {56 vars in 1308 bytes} [Thu Oct 13 04:08:40 2022] GET /identity/v3/role_assignments?user.id=0bcd00d1131e4d8caebc3d3f8c6f8777&scope.system=all&role.id=2306a441792b49f19af2d869acf5b3b6 => generated 510 bytes in 208 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 13 04:08:40.782802 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a415d47f-af2d-4636-9439-9d1433dd5e59 None None] REQUEST_METHOD: `GET` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:08:40.783173 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a415d47f-af2d-4636-9439-9d1433dd5e59 None None] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:08:40.783370 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a415d47f-af2d-4636-9439-9d1433dd5e59 None None] PATH_INFO: `/` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:08:40.784964 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 143/283] 10.222.0.27 () {54 vars in 927 bytes} [Thu Oct 13 04:08:40 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 13 04:08:40.790569 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92bf42e3-20c1-4bdc-8f22-ec4c0b4fd51d None None] REQUEST_METHOD: `POST` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:08:40.790856 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92bf42e3-20c1-4bdc-8f22-ec4c0b4fd51d None None] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:08:40.791129 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92bf42e3-20c1-4bdc-8f22-ec4c0b4fd51d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:08:40.814809 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.auth.core [None req-92bf42e3-20c1-4bdc-8f22-ec4c0b4fd51d None None] MFA Rules not processed for user `0e2d979563d6421995451a0e86bf3022`. Rule list: `[]` (Enabled: `True`). {{(pid=81408) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 13 04:08:40.816027 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.fernet_utils [None req-92bf42e3-20c1-4bdc-8f22-ec4c0b4fd51d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81408) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:08:40.865990 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG dbcounter [-] [81408] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81408) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:08:40.887047 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 141/284] 10.222.0.27 () {58 vars in 1024 bytes} [Thu Oct 13 04:08:40 2022] POST /identity/v3/auth/tokens => generated 914 bytes in 97 msecs (HTTP/1.1 201) 6 headers in 384 bytes (2 switches on core 0) Oct 13 04:08:41.178198 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a415d47f-af2d-4636-9439-9d1433dd5e59 None None] Authenticating user token {{(pid=81409) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:08:41.191109 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a415d47f-af2d-4636-9439-9d1433dd5e59 None None] Validating token access rules against request {{(pid=81409) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:08:41.195465 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3c63b646-4be9-4f48-993f-761e39366a55 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8ba823f313144da3b683bacef8fd62e0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81409) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:08:41.196612 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c63b646-4be9-4f48-993f-761e39366a55 admin admin] REQUEST_METHOD: `GET` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:08:41.196895 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c63b646-4be9-4f48-993f-761e39366a55 admin admin] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:08:41.197163 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c63b646-4be9-4f48-993f-761e39366a55 admin admin] PATH_INFO: `/v3/roles/anotherrole` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:08:41.208900 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3c63b646-4be9-4f48-993f-761e39366a55 admin admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 13 04:08:41.213913 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application [None req-3c63b646-4be9-4f48-993f-761e39366a55 admin admin] Could not find role: anotherrole.: keystone.exception.RoleNotFound: Could not find role: anotherrole. Oct 13 04:08:41.213913 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 13 04:08:41.213913 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 13 04:08:41.213913 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 13 04:08:41.213913 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 13 04:08:41.213913 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 13 04:08:41.213913 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 13 04:08:41.213913 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 13 04:08:41.213913 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 13 04:08:41.213913 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 13 04:08:41.213913 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 13 04:08:41.213913 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 13 04:08:41.213913 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 13 04:08:41.213913 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 13 04:08:41.213913 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 13 04:08:41.213913 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 13 04:08:41.214828 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 13 04:08:41.214828 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 13 04:08:41.214828 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 13 04:08:41.214828 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 13 04:08:41.214828 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 13 04:08:41.214828 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 13 04:08:41.214828 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 13 04:08:41.214828 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 13 04:08:41.214828 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 13 04:08:41.214828 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application return self.get_or_create( Oct 13 04:08:41.214828 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 13 04:08:41.214828 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application with Lock( Oct 13 04:08:41.214828 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 13 04:08:41.214828 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application return self._enter() Oct 13 04:08:41.214828 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 13 04:08:41.214828 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 13 04:08:41.215619 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 13 04:08:41.215619 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application return self.creator() Oct 13 04:08:41.215619 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 13 04:08:41.215619 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application created_value = creator( Oct 13 04:08:41.215619 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 13 04:08:41.215619 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 13 04:08:41.215619 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 13 04:08:41.215619 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 13 04:08:41.215619 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 13 04:08:41.215619 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 13 04:08:41.215619 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: anotherrole. Oct 13 04:08:41.215619 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application Oct 13 04:08:41.215619 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 144/285] 10.222.0.27 () {56 vars in 1135 bytes} [Thu Oct 13 04:08:41 2022] GET /identity/v3/roles/anotherrole => generated 89 bytes in 38 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 13 04:08:41.221155 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-92bf42e3-20c1-4bdc-8f22-ec4c0b4fd51d None None] Authenticating user token {{(pid=81408) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:08:41.231216 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-92bf42e3-20c1-4bdc-8f22-ec4c0b4fd51d None None] Validating token access rules against request {{(pid=81408) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:08:41.235728 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-02c1e270-2b53-4370-8c50-3e5c6f4332f1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8ba823f313144da3b683bacef8fd62e0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81408) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:08:41.236338 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-02c1e270-2b53-4370-8c50-3e5c6f4332f1 admin admin] REQUEST_METHOD: `GET` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:08:41.236616 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-02c1e270-2b53-4370-8c50-3e5c6f4332f1 admin admin] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:08:41.236888 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-02c1e270-2b53-4370-8c50-3e5c6f4332f1 admin admin] PATH_INFO: `/v3/roles` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:08:41.237566 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-02c1e270-2b53-4370-8c50-3e5c6f4332f1 admin admin] RBAC: Adding query filter params (name=anotherrole) {{(pid=81408) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 13 04:08:41.237868 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-02c1e270-2b53-4370-8c50-3e5c6f4332f1 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:08:41.239533 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-02c1e270-2b53-4370-8c50-3e5c6f4332f1 admin admin] RBAC: Authorization granted {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:08:41.250436 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 142/286] 10.222.0.27 () {56 vars in 1132 bytes} [Thu Oct 13 04:08:41 2022] GET /identity/v3/roles?name=anotherrole => generated 334 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 13 04:08:41.255998 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3c63b646-4be9-4f48-993f-761e39366a55 admin admin] Authenticating user token {{(pid=81409) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:08:41.266555 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3c63b646-4be9-4f48-993f-761e39366a55 admin admin] Validating token access rules against request {{(pid=81409) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:08:41.272421 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7ba3598-5fb5-484b-b3fc-807702423e95 None None] REQUEST_METHOD: `GET` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:08:41.272700 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7ba3598-5fb5-484b-b3fc-807702423e95 None None] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:08:41.272973 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7ba3598-5fb5-484b-b3fc-807702423e95 None None] PATH_INFO: `/` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:08:41.274433 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 143/287] 10.222.0.27 () {54 vars in 927 bytes} [Thu Oct 13 04:08:41 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 13 04:08:41.274636 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-26e760de-3d43-4ad6-be89-bf10117e6c9c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8ba823f313144da3b683bacef8fd62e0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81409) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:08:41.275470 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26e760de-3d43-4ad6-be89-bf10117e6c9c admin admin] REQUEST_METHOD: `GET` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:08:41.275742 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26e760de-3d43-4ad6-be89-bf10117e6c9c admin admin] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:08:41.276009 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26e760de-3d43-4ad6-be89-bf10117e6c9c admin admin] PATH_INFO: `/v3/groups/d232567016e94eeca7524d2c09a20c3b` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:08:41.277265 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-26e760de-3d43-4ad6-be89-bf10117e6c9c admin admin] RBAC: Authorizing `identity:get_group(group_id=d232567016e94eeca7524d2c09a20c3b)` {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:08:41.279253 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-26e760de-3d43-4ad6-be89-bf10117e6c9c admin admin] RBAC: Authorization granted {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:08:41.281284 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 145/288] 10.222.0.27 () {56 vars in 1201 bytes} [Thu Oct 13 04:08:41 2022] GET /identity/v3/groups/d232567016e94eeca7524d2c09a20c3b => generated 226 bytes in 25 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 13 04:08:41.283696 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f9b424f-3120-4897-98f8-bcfa0b40847e None None] REQUEST_METHOD: `POST` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:08:41.284204 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f9b424f-3120-4897-98f8-bcfa0b40847e None None] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:08:41.284680 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f9b424f-3120-4897-98f8-bcfa0b40847e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:08:41.286284 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-26e760de-3d43-4ad6-be89-bf10117e6c9c admin admin] Authenticating user token {{(pid=81409) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:08:41.298678 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-26e760de-3d43-4ad6-be89-bf10117e6c9c admin admin] Validating token access rules against request {{(pid=81409) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:08:41.310743 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-089934e9-e777-42c3-abe2-99bd636f3c37 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8ba823f313144da3b683bacef8fd62e0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81409) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:08:41.310743 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-089934e9-e777-42c3-abe2-99bd636f3c37 admin admin] REQUEST_METHOD: `GET` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:08:41.310743 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-089934e9-e777-42c3-abe2-99bd636f3c37 admin admin] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:08:41.310743 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-089934e9-e777-42c3-abe2-99bd636f3c37 admin admin] PATH_INFO: `/v3/projects/0114058aecc645ea8b47ae6d1301c17d` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:08:41.311105 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-089934e9-e777-42c3-abe2-99bd636f3c37 admin admin] RBAC: Authorizing `identity:get_project(project_id=0114058aecc645ea8b47ae6d1301c17d)` {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:08:41.311105 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-089934e9-e777-42c3-abe2-99bd636f3c37 admin admin] RBAC: Authorization granted {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:08:41.311105 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 146/289] 10.222.0.27 () {56 vars in 1207 bytes} [Thu Oct 13 04:08:41 2022] GET /identity/v3/projects/0114058aecc645ea8b47ae6d1301c17d => generated 302 bytes in 23 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 13 04:08:41.316186 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-089934e9-e777-42c3-abe2-99bd636f3c37 admin admin] Authenticating user token {{(pid=81409) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:08:41.321951 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.auth.core [None req-0f9b424f-3120-4897-98f8-bcfa0b40847e None None] MFA Rules not processed for user `0e2d979563d6421995451a0e86bf3022`. Rule list: `[]` (Enabled: `True`). {{(pid=81408) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 13 04:08:41.323535 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.fernet_utils [None req-0f9b424f-3120-4897-98f8-bcfa0b40847e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81408) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:08:41.328001 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-089934e9-e777-42c3-abe2-99bd636f3c37 admin admin] Validating token access rules against request {{(pid=81409) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:08:41.336727 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 144/290] 10.222.0.27 () {58 vars in 1024 bytes} [Thu Oct 13 04:08:41 2022] POST /identity/v3/auth/tokens => generated 914 bytes in 54 msecs (HTTP/1.1 201) 6 headers in 384 bytes (1 switches on core 0) Oct 13 04:08:41.337728 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-342b4d9f-90b1-4e48-82f4-fc338dec14ca admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8ba823f313144da3b683bacef8fd62e0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81409) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:08:41.338443 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-342b4d9f-90b1-4e48-82f4-fc338dec14ca admin admin] REQUEST_METHOD: `PUT` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:08:41.338722 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-342b4d9f-90b1-4e48-82f4-fc338dec14ca admin admin] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:08:41.338993 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-342b4d9f-90b1-4e48-82f4-fc338dec14ca admin admin] PATH_INFO: `/v3/projects/0114058aecc645ea8b47ae6d1301c17d/groups/d232567016e94eeca7524d2c09a20c3b/roles/743995e93fef47fd91204c3d701054b0` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:08:41.341190 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-342b4d9f-90b1-4e48-82f4-fc338dec14ca admin admin] RBAC: Authorizing `identity:create_grant(project_id=0114058aecc645ea8b47ae6d1301c17d, group_id=d232567016e94eeca7524d2c09a20c3b, role_id=743995e93fef47fd91204c3d701054b0)` {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:08:41.343904 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-342b4d9f-90b1-4e48-82f4-fc338dec14ca admin admin] RBAC: Authorization granted {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:08:41.362956 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 147/291] 10.222.0.27 () {58 vars in 1463 bytes} [Thu Oct 13 04:08:41 2022] PUT /identity/v3/projects/0114058aecc645ea8b47ae6d1301c17d/groups/d232567016e94eeca7524d2c09a20c3b/roles/743995e93fef47fd91204c3d701054b0 => generated 0 bytes in 47 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 13 04:08:41.730249 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0f9b424f-3120-4897-98f8-bcfa0b40847e None None] Authenticating user token {{(pid=81408) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:08:41.739632 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0f9b424f-3120-4897-98f8-bcfa0b40847e None None] Validating token access rules against request {{(pid=81408) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:08:41.748731 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-207af9b9-94f4-4457-966a-405f9cf1d0cc None None] REQUEST_METHOD: `GET` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:08:41.749006 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-207af9b9-94f4-4457-966a-405f9cf1d0cc None None] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:08:41.749277 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-207af9b9-94f4-4457-966a-405f9cf1d0cc None None] PATH_INFO: `/` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:08:41.751013 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 148/292] 10.222.0.27 () {54 vars in 927 bytes} [Thu Oct 13 04:08:41 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Oct 13 04:08:41.768933 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d230e1a-7695-490e-b0a6-c7852bf03ef6 None None] REQUEST_METHOD: `POST` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:08:41.769214 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d230e1a-7695-490e-b0a6-c7852bf03ef6 None None] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:08:41.769486 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d230e1a-7695-490e-b0a6-c7852bf03ef6 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:08:41.804284 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.auth.core [None req-0d230e1a-7695-490e-b0a6-c7852bf03ef6 None None] MFA Rules not processed for user `0e2d979563d6421995451a0e86bf3022`. Rule list: `[]` (Enabled: `True`). {{(pid=81409) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 13 04:08:41.806796 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.fernet_utils [None req-0d230e1a-7695-490e-b0a6-c7852bf03ef6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81409) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:08:41.816967 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-777e5b77-12e3-4b80-904e-23aba3a98eae admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8ba823f313144da3b683bacef8fd62e0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81408) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:08:41.817863 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-777e5b77-12e3-4b80-904e-23aba3a98eae admin admin] REQUEST_METHOD: `GET` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:08:41.818346 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-777e5b77-12e3-4b80-904e-23aba3a98eae admin admin] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:08:41.818753 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-777e5b77-12e3-4b80-904e-23aba3a98eae admin admin] PATH_INFO: `/v3/roles/admin` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:08:41.840938 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-777e5b77-12e3-4b80-904e-23aba3a98eae admin admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 13 04:08:41.847716 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application [None req-777e5b77-12e3-4b80-904e-23aba3a98eae admin admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Oct 13 04:08:41.847716 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 13 04:08:41.847716 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 13 04:08:41.847716 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 13 04:08:41.847716 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 13 04:08:41.847716 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 13 04:08:41.847716 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 13 04:08:41.847716 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 13 04:08:41.847716 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 13 04:08:41.847716 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 13 04:08:41.847716 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 13 04:08:41.847716 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 13 04:08:41.847716 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 13 04:08:41.847716 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 13 04:08:41.847716 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 13 04:08:41.847716 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 13 04:08:41.848767 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 13 04:08:41.848767 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 13 04:08:41.848767 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 13 04:08:41.848767 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 13 04:08:41.848767 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 13 04:08:41.848767 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 13 04:08:41.848767 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 13 04:08:41.848767 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 13 04:08:41.848767 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 13 04:08:41.848767 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application return self.get_or_create( Oct 13 04:08:41.848767 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 13 04:08:41.848767 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application with Lock( Oct 13 04:08:41.848767 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 13 04:08:41.848767 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application return self._enter() Oct 13 04:08:41.848767 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 13 04:08:41.848767 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 13 04:08:41.849894 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 13 04:08:41.849894 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application return self.creator() Oct 13 04:08:41.849894 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 13 04:08:41.849894 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application created_value = creator( Oct 13 04:08:41.849894 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 13 04:08:41.849894 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 13 04:08:41.849894 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 13 04:08:41.849894 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 13 04:08:41.849894 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 13 04:08:41.849894 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 13 04:08:41.849894 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Oct 13 04:08:41.849894 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application Oct 13 04:08:41.856367 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 145/293] 10.222.0.27 () {56 vars in 1117 bytes} [Thu Oct 13 04:08:41 2022] GET /identity/v3/roles/admin => generated 83 bytes in 127 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 13 04:08:41.863025 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-777e5b77-12e3-4b80-904e-23aba3a98eae admin admin] Authenticating user token {{(pid=81408) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:08:41.877955 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-777e5b77-12e3-4b80-904e-23aba3a98eae admin admin] Validating token access rules against request {{(pid=81408) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:08:41.893131 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3614d513-72a8-4167-b521-8a1653eca9a1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8ba823f313144da3b683bacef8fd62e0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81408) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:08:41.894042 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3614d513-72a8-4167-b521-8a1653eca9a1 admin admin] REQUEST_METHOD: `GET` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:08:41.894496 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3614d513-72a8-4167-b521-8a1653eca9a1 admin admin] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:08:41.894903 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3614d513-72a8-4167-b521-8a1653eca9a1 admin admin] PATH_INFO: `/v3/roles` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:08:41.896210 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3614d513-72a8-4167-b521-8a1653eca9a1 admin admin] RBAC: Adding query filter params (name=admin) {{(pid=81408) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 13 04:08:41.897213 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3614d513-72a8-4167-b521-8a1653eca9a1 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:08:41.901689 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3614d513-72a8-4167-b521-8a1653eca9a1 admin admin] RBAC: Authorization granted {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:08:41.918052 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 146/294] 10.222.0.27 () {56 vars in 1120 bytes} [Thu Oct 13 04:08:41 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 56 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 13 04:08:41.923956 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3614d513-72a8-4167-b521-8a1653eca9a1 admin admin] Authenticating user token {{(pid=81408) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:08:41.941252 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3614d513-72a8-4167-b521-8a1653eca9a1 admin admin] Validating token access rules against request {{(pid=81408) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:08:41.951959 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 149/295] 10.222.0.27 () {58 vars in 1024 bytes} [Thu Oct 13 04:08:41 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 184 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Oct 13 04:08:41.957745 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0d339569-efda-4d47-b485-4c479dce9a2f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8ba823f313144da3b683bacef8fd62e0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81408) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:08:41.958725 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d339569-efda-4d47-b485-4c479dce9a2f admin admin] REQUEST_METHOD: `GET` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:08:41.959139 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d339569-efda-4d47-b485-4c479dce9a2f admin admin] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:08:41.959545 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d339569-efda-4d47-b485-4c479dce9a2f admin admin] PATH_INFO: `/v3/groups/63a8f28fb01c46fab610e79058cf47ae` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:08:41.967024 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0d339569-efda-4d47-b485-4c479dce9a2f admin admin] RBAC: Authorizing `identity:get_group(group_id=63a8f28fb01c46fab610e79058cf47ae)` {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:08:41.969163 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0d339569-efda-4d47-b485-4c479dce9a2f admin admin] RBAC: Authorization granted {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:08:41.972043 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 147/296] 10.222.0.27 () {56 vars in 1201 bytes} [Thu Oct 13 04:08:41 2022] GET /identity/v3/groups/63a8f28fb01c46fab610e79058cf47ae => generated 229 bytes in 48 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 13 04:08:41.976985 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0d230e1a-7695-490e-b0a6-c7852bf03ef6 None None] Authenticating user token {{(pid=81409) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:08:41.990558 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0d230e1a-7695-490e-b0a6-c7852bf03ef6 None None] Validating token access rules against request {{(pid=81409) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:08:41.995555 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fb22856f-e356-436e-b636-565d8a41a4c1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8ba823f313144da3b683bacef8fd62e0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81409) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:08:41.996431 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb22856f-e356-436e-b636-565d8a41a4c1 admin admin] REQUEST_METHOD: `GET` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:08:41.996862 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb22856f-e356-436e-b636-565d8a41a4c1 admin admin] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:08:41.997294 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb22856f-e356-436e-b636-565d8a41a4c1 admin admin] PATH_INFO: `/v3/projects/8ba823f313144da3b683bacef8fd62e0` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:08:42.003039 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fb22856f-e356-436e-b636-565d8a41a4c1 admin admin] RBAC: Authorizing `identity:get_project(project_id=8ba823f313144da3b683bacef8fd62e0)` {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:08:42.005280 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fb22856f-e356-436e-b636-565d8a41a4c1 admin admin] RBAC: Authorization granted {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:08:42.007740 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 150/297] 10.222.0.27 () {56 vars in 1207 bytes} [Thu Oct 13 04:08:41 2022] GET /identity/v3/projects/8ba823f313144da3b683bacef8fd62e0 => generated 344 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 13 04:08:42.013999 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0d339569-efda-4d47-b485-4c479dce9a2f admin admin] Authenticating user token {{(pid=81408) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:08:42.024701 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0d339569-efda-4d47-b485-4c479dce9a2f admin admin] Validating token access rules against request {{(pid=81408) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:08:42.029652 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-16bd7cbc-63ed-4ea4-bd36-f3148ad4fa41 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8ba823f313144da3b683bacef8fd62e0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81408) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:08:42.030571 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16bd7cbc-63ed-4ea4-bd36-f3148ad4fa41 admin admin] REQUEST_METHOD: `PUT` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:08:42.031003 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16bd7cbc-63ed-4ea4-bd36-f3148ad4fa41 admin admin] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:08:42.031467 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16bd7cbc-63ed-4ea4-bd36-f3148ad4fa41 admin admin] PATH_INFO: `/v3/projects/8ba823f313144da3b683bacef8fd62e0/groups/63a8f28fb01c46fab610e79058cf47ae/roles/ef55fbd94e1040be83695754075a7ed5` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:08:42.033588 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-16bd7cbc-63ed-4ea4-bd36-f3148ad4fa41 admin admin] RBAC: Authorizing `identity:create_grant(project_id=8ba823f313144da3b683bacef8fd62e0, group_id=63a8f28fb01c46fab610e79058cf47ae, role_id=ef55fbd94e1040be83695754075a7ed5)` {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:08:42.036566 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-16bd7cbc-63ed-4ea4-bd36-f3148ad4fa41 admin admin] RBAC: Authorization granted {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:08:42.059868 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 148/298] 10.222.0.27 () {58 vars in 1463 bytes} [Thu Oct 13 04:08:42 2022] PUT /identity/v3/projects/8ba823f313144da3b683bacef8fd62e0/groups/63a8f28fb01c46fab610e79058cf47ae/roles/ef55fbd94e1040be83695754075a7ed5 => generated 0 bytes in 46 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 13 04:08:42.435114 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d69d9467-7a59-4a41-a401-acdd716a1600 None None] REQUEST_METHOD: `GET` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:08:42.436071 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d69d9467-7a59-4a41-a401-acdd716a1600 None None] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:08:42.436531 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d69d9467-7a59-4a41-a401-acdd716a1600 None None] PATH_INFO: `/` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:08:42.438063 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 151/299] 10.222.0.27 () {54 vars in 927 bytes} [Thu Oct 13 04:08:42 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 13 04:08:42.443829 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14d65e3d-0334-407f-9cd9-39d85bb5ab7c None None] REQUEST_METHOD: `POST` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:08:42.444655 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14d65e3d-0334-407f-9cd9-39d85bb5ab7c None None] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:08:42.445471 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14d65e3d-0334-407f-9cd9-39d85bb5ab7c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:08:42.467355 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.auth.core [None req-14d65e3d-0334-407f-9cd9-39d85bb5ab7c None None] MFA Rules not processed for user `0e2d979563d6421995451a0e86bf3022`. Rule list: `[]` (Enabled: `True`). {{(pid=81408) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 13 04:08:42.468863 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.fernet_utils [None req-14d65e3d-0334-407f-9cd9-39d85bb5ab7c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81408) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:08:42.496453 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d69d9467-7a59-4a41-a401-acdd716a1600 None None] Authenticating user token {{(pid=81409) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:08:42.512183 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG dbcounter [-] [81409] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81409) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:08:42.590760 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 149/300] 10.222.0.27 () {58 vars in 1024 bytes} [Thu Oct 13 04:08:42 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 148 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Oct 13 04:08:42.623881 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d69d9467-7a59-4a41-a401-acdd716a1600 None None] Validating token access rules against request {{(pid=81409) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:08:42.666850 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d4aa0663-0a28-4a02-b050-d85d0b64a4a9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81409) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:08:42.667413 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4aa0663-0a28-4a02-b050-d85d0b64a4a9 None admin] REQUEST_METHOD: `GET` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:08:42.667685 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4aa0663-0a28-4a02-b050-d85d0b64a4a9 None admin] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:08:42.667948 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4aa0663-0a28-4a02-b050-d85d0b64a4a9 None admin] PATH_INFO: `/v3/roles/anotherrole` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:08:42.678957 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d4aa0663-0a28-4a02-b050-d85d0b64a4a9 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 13 04:08:42.684931 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application [None req-d4aa0663-0a28-4a02-b050-d85d0b64a4a9 None admin] Could not find role: anotherrole.: keystone.exception.RoleNotFound: Could not find role: anotherrole. Oct 13 04:08:42.684931 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 13 04:08:42.684931 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 13 04:08:42.684931 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 13 04:08:42.684931 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 13 04:08:42.684931 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 13 04:08:42.684931 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 13 04:08:42.684931 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 13 04:08:42.684931 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 13 04:08:42.684931 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 13 04:08:42.684931 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 13 04:08:42.684931 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 13 04:08:42.684931 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 13 04:08:42.684931 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 13 04:08:42.684931 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 13 04:08:42.684931 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 13 04:08:42.685795 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 13 04:08:42.685795 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 13 04:08:42.685795 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 13 04:08:42.685795 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 13 04:08:42.685795 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 13 04:08:42.685795 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 13 04:08:42.685795 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 13 04:08:42.685795 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 13 04:08:42.685795 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 13 04:08:42.685795 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application return self.get_or_create( Oct 13 04:08:42.685795 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 13 04:08:42.685795 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application with Lock( Oct 13 04:08:42.685795 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 13 04:08:42.685795 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application return self._enter() Oct 13 04:08:42.685795 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 13 04:08:42.685795 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 13 04:08:42.687046 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 13 04:08:42.687046 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application return self.creator() Oct 13 04:08:42.687046 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 13 04:08:42.687046 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application created_value = creator( Oct 13 04:08:42.687046 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 13 04:08:42.687046 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 13 04:08:42.687046 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 13 04:08:42.687046 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 13 04:08:42.687046 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 13 04:08:42.687046 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 13 04:08:42.687046 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: anotherrole. Oct 13 04:08:42.687046 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application Oct 13 04:08:42.687046 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 152/301] 10.222.0.27 () {56 vars in 1114 bytes} [Thu Oct 13 04:08:42 2022] GET /identity/v3/roles/anotherrole => generated 89 bytes in 190 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 13 04:08:42.692170 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-14d65e3d-0334-407f-9cd9-39d85bb5ab7c None None] Authenticating user token {{(pid=81408) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:08:42.714706 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG dbcounter [-] [81408] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81408) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:08:42.825755 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-14d65e3d-0334-407f-9cd9-39d85bb5ab7c None None] Validating token access rules against request {{(pid=81408) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:08:42.844172 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2926e1f0-e1be-4836-98c6-8edb5edfa2b9 None None] REQUEST_METHOD: `GET` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:08:42.844488 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2926e1f0-e1be-4836-98c6-8edb5edfa2b9 None None] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:08:42.844725 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2926e1f0-e1be-4836-98c6-8edb5edfa2b9 None None] PATH_INFO: `/` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:08:42.846081 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 153/302] 10.222.0.27 () {54 vars in 927 bytes} [Thu Oct 13 04:08:42 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 13 04:08:42.855792 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05136ecc-8080-43dc-8b97-3577b931c00a None None] REQUEST_METHOD: `POST` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:08:42.856080 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05136ecc-8080-43dc-8b97-3577b931c00a None None] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:08:42.856355 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05136ecc-8080-43dc-8b97-3577b931c00a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:08:42.873196 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c94477f8-2c72-4639-9a68-b5ca7e6a1274 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81408) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:08:42.874159 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c94477f8-2c72-4639-9a68-b5ca7e6a1274 None admin] REQUEST_METHOD: `GET` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:08:42.874656 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c94477f8-2c72-4639-9a68-b5ca7e6a1274 None admin] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:08:42.875123 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c94477f8-2c72-4639-9a68-b5ca7e6a1274 None admin] PATH_INFO: `/v3/roles` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:08:42.876056 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c94477f8-2c72-4639-9a68-b5ca7e6a1274 None admin] RBAC: Adding query filter params (name=anotherrole) {{(pid=81408) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 13 04:08:42.876575 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c94477f8-2c72-4639-9a68-b5ca7e6a1274 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:08:42.878156 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c94477f8-2c72-4639-9a68-b5ca7e6a1274 None admin] RBAC: Authorization granted {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:08:42.885449 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.auth.core [None req-05136ecc-8080-43dc-8b97-3577b931c00a None None] MFA Rules not processed for user `0e2d979563d6421995451a0e86bf3022`. Rule list: `[]` (Enabled: `True`). {{(pid=81409) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 13 04:08:42.885449 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.fernet_utils [None req-05136ecc-8080-43dc-8b97-3577b931c00a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81409) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:08:42.901076 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 150/303] 10.222.0.27 () {56 vars in 1111 bytes} [Thu Oct 13 04:08:42 2022] GET /identity/v3/roles?name=anotherrole => generated 334 bytes in 209 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 13 04:08:42.907556 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c94477f8-2c72-4639-9a68-b5ca7e6a1274 None admin] Authenticating user token {{(pid=81408) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:08:42.999269 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG dbcounter [-] [81409] Writing DB stats keystone:SELECT=100 {{(pid=81409) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:08:43.009486 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 154/304] 10.222.0.27 () {58 vars in 1024 bytes} [Thu Oct 13 04:08:42 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 154 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Oct 13 04:08:43.031931 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG dbcounter [-] [81408] Writing DB stats keystone:SELECT=100 {{(pid=81408) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:08:43.060229 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c94477f8-2c72-4639-9a68-b5ca7e6a1274 None admin] Validating token access rules against request {{(pid=81408) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:08:43.096467 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7de69a78-16d4-48da-8ae4-169556ef2634 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81408) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:08:43.097134 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7de69a78-16d4-48da-8ae4-169556ef2634 None admin] REQUEST_METHOD: `GET` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:08:43.097415 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7de69a78-16d4-48da-8ae4-169556ef2634 None admin] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:08:43.097684 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7de69a78-16d4-48da-8ae4-169556ef2634 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:08:43.114664 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-05136ecc-8080-43dc-8b97-3577b931c00a None None] Authenticating user token {{(pid=81409) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:08:43.180149 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7de69a78-16d4-48da-8ae4-169556ef2634 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:08:43.182586 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7de69a78-16d4-48da-8ae4-169556ef2634 None admin] RBAC: Authorization granted {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:08:43.320078 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-05136ecc-8080-43dc-8b97-3577b931c00a None None] Validating token access rules against request {{(pid=81409) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:08:43.322365 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG dbcounter [-] [81408] Writing DB stats keystone:SELECT=100 {{(pid=81408) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:08:43.375969 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 151/305] 10.222.0.27 () {58 vars in 1282 bytes} [Thu Oct 13 04:08:42 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 469 msecs (HTTP/1.1 200) 6 headers in 358 bytes (2 switches on core 0) Oct 13 04:08:43.376758 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7de69a78-16d4-48da-8ae4-169556ef2634 None admin] Authenticating user token {{(pid=81408) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:08:43.394696 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-04290679-2dc4-4518-8b61-202ac3c6238e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81409) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:08:43.395339 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04290679-2dc4-4518-8b61-202ac3c6238e None admin] REQUEST_METHOD: `GET` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:08:43.395612 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04290679-2dc4-4518-8b61-202ac3c6238e None admin] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:08:43.395882 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04290679-2dc4-4518-8b61-202ac3c6238e None admin] PATH_INFO: `/v3/roles/member` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:08:43.420958 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-04290679-2dc4-4518-8b61-202ac3c6238e None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 13 04:08:43.426601 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application [None req-04290679-2dc4-4518-8b61-202ac3c6238e None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Oct 13 04:08:43.426601 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 13 04:08:43.426601 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 13 04:08:43.426601 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 13 04:08:43.426601 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 13 04:08:43.426601 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 13 04:08:43.426601 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 13 04:08:43.426601 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 13 04:08:43.426601 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 13 04:08:43.426601 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 13 04:08:43.426601 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 13 04:08:43.426601 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 13 04:08:43.426601 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 13 04:08:43.426601 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 13 04:08:43.426601 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 13 04:08:43.426601 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 13 04:08:43.427476 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 13 04:08:43.427476 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 13 04:08:43.427476 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 13 04:08:43.427476 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 13 04:08:43.427476 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 13 04:08:43.427476 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 13 04:08:43.427476 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 13 04:08:43.427476 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 13 04:08:43.427476 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 13 04:08:43.427476 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application return self.get_or_create( Oct 13 04:08:43.427476 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 13 04:08:43.427476 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application with Lock( Oct 13 04:08:43.427476 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 13 04:08:43.427476 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application return self._enter() Oct 13 04:08:43.427476 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 13 04:08:43.427476 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 13 04:08:43.428253 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 13 04:08:43.428253 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application return self.creator() Oct 13 04:08:43.428253 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 13 04:08:43.428253 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application created_value = creator( Oct 13 04:08:43.428253 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 13 04:08:43.428253 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 13 04:08:43.428253 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 13 04:08:43.428253 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 13 04:08:43.428253 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 13 04:08:43.428253 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 13 04:08:43.428253 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Oct 13 04:08:43.428253 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application Oct 13 04:08:43.428253 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 155/306] 10.222.0.27 () {56 vars in 1099 bytes} [Thu Oct 13 04:08:43 2022] GET /identity/v3/roles/member => generated 84 bytes in 314 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Oct 13 04:08:43.429423 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-04290679-2dc4-4518-8b61-202ac3c6238e None admin] Authenticating user token {{(pid=81409) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:08:43.503271 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG dbcounter [-] [81409] Writing DB stats keystone:SELECT=100 {{(pid=81409) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:08:43.527339 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7de69a78-16d4-48da-8ae4-169556ef2634 None admin] Validating token access rules against request {{(pid=81408) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:08:43.553950 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-04290679-2dc4-4518-8b61-202ac3c6238e None admin] Validating token access rules against request {{(pid=81409) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:08:43.576917 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bd9f58e5-62d6-4174-ae59-5fc3bb9cbae2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81408) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:08:43.577743 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd9f58e5-62d6-4174-ae59-5fc3bb9cbae2 None admin] REQUEST_METHOD: `GET` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:08:43.577913 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd9f58e5-62d6-4174-ae59-5fc3bb9cbae2 None admin] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:08:43.578281 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd9f58e5-62d6-4174-ae59-5fc3bb9cbae2 None admin] PATH_INFO: `/v3/roles/member` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:08:43.589047 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bd9f58e5-62d6-4174-ae59-5fc3bb9cbae2 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 13 04:08:43.593666 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4c60101a-19bd-4733-aed6-be4c63bd0ef1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81409) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:08:43.594506 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application [None req-bd9f58e5-62d6-4174-ae59-5fc3bb9cbae2 None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Oct 13 04:08:43.594506 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 13 04:08:43.594506 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 13 04:08:43.594506 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 13 04:08:43.594506 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 13 04:08:43.594506 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 13 04:08:43.594506 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 13 04:08:43.594506 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 13 04:08:43.594506 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 13 04:08:43.594506 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 13 04:08:43.594506 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 13 04:08:43.594506 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 13 04:08:43.594506 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 13 04:08:43.594506 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 13 04:08:43.594506 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 13 04:08:43.594506 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 13 04:08:43.595565 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 13 04:08:43.595565 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 13 04:08:43.595565 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 13 04:08:43.595565 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 13 04:08:43.595565 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 13 04:08:43.595565 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 13 04:08:43.595565 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 13 04:08:43.595565 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 13 04:08:43.595565 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 13 04:08:43.595565 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application return self.get_or_create( Oct 13 04:08:43.595565 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 13 04:08:43.595565 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application with Lock( Oct 13 04:08:43.595565 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 13 04:08:43.595565 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application return self._enter() Oct 13 04:08:43.595565 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 13 04:08:43.595565 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 13 04:08:43.596541 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 13 04:08:43.596541 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application return self.creator() Oct 13 04:08:43.596541 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 13 04:08:43.596541 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application created_value = creator( Oct 13 04:08:43.596541 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 13 04:08:43.596541 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 13 04:08:43.596541 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 13 04:08:43.596541 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 13 04:08:43.596541 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 13 04:08:43.596541 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 13 04:08:43.596541 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Oct 13 04:08:43.596541 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application Oct 13 04:08:43.597370 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c60101a-19bd-4733-aed6-be4c63bd0ef1 None admin] REQUEST_METHOD: `GET` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:08:43.597769 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c60101a-19bd-4733-aed6-be4c63bd0ef1 None admin] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:08:43.597872 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 152/307] 10.222.0.27 () {56 vars in 1099 bytes} [Thu Oct 13 04:08:43 2022] GET /identity/v3/roles/member => generated 84 bytes in 221 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 13 04:08:43.598278 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c60101a-19bd-4733-aed6-be4c63bd0ef1 None admin] PATH_INFO: `/v3/projects/eeec419e34e44c1f881179933c828d97` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:08:43.599950 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4c60101a-19bd-4733-aed6-be4c63bd0ef1 None admin] RBAC: Authorizing `identity:get_project(project_id=eeec419e34e44c1f881179933c828d97)` {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:08:43.600341 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bd9f58e5-62d6-4174-ae59-5fc3bb9cbae2 None admin] Authenticating user token {{(pid=81408) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:08:43.601898 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4c60101a-19bd-4733-aed6-be4c63bd0ef1 None admin] RBAC: Authorization granted {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:08:43.604514 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 156/308] 10.222.0.27 () {56 vars in 1186 bytes} [Thu Oct 13 04:08:43 2022] GET /identity/v3/projects/eeec419e34e44c1f881179933c828d97 => generated 298 bytes in 176 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 13 04:08:43.610448 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4c60101a-19bd-4733-aed6-be4c63bd0ef1 None admin] Authenticating user token {{(pid=81409) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:08:43.647872 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG dbcounter [-] [81408] Writing DB stats keystone:SELECT=100 {{(pid=81408) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:08:43.741051 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bd9f58e5-62d6-4174-ae59-5fc3bb9cbae2 None admin] Validating token access rules against request {{(pid=81408) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:08:43.761009 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4c60101a-19bd-4733-aed6-be4c63bd0ef1 None admin] Validating token access rules against request {{(pid=81409) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:08:43.787273 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c0c666ab-984d-49c3-b502-678c3c8aa294 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81408) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:08:43.787886 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0c666ab-984d-49c3-b502-678c3c8aa294 None admin] REQUEST_METHOD: `GET` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:08:43.788164 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0c666ab-984d-49c3-b502-678c3c8aa294 None admin] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:08:43.788433 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0c666ab-984d-49c3-b502-678c3c8aa294 None admin] PATH_INFO: `/v3/roles` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:08:43.789106 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c0c666ab-984d-49c3-b502-678c3c8aa294 None admin] RBAC: Adding query filter params (name=member) {{(pid=81408) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 13 04:08:43.789405 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c0c666ab-984d-49c3-b502-678c3c8aa294 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:08:43.792099 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c0c666ab-984d-49c3-b502-678c3c8aa294 None admin] RBAC: Authorization granted {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:08:43.799419 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG dbcounter [-] [81409] Writing DB stats keystone:SELECT=100 {{(pid=81409) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:08:43.806792 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 153/309] 10.222.0.27 () {56 vars in 1101 bytes} [Thu Oct 13 04:08:43 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 208 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 13 04:08:43.807670 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c0c666ab-984d-49c3-b502-678c3c8aa294 None admin] Authenticating user token {{(pid=81408) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:08:43.811338 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-208a2ba6-8495-459e-94a4-49a4fb216820 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81409) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:08:43.813165 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-208a2ba6-8495-459e-94a4-49a4fb216820 None admin] REQUEST_METHOD: `GET` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:08:43.813723 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-208a2ba6-8495-459e-94a4-49a4fb216820 None admin] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:08:43.814291 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-208a2ba6-8495-459e-94a4-49a4fb216820 None admin] PATH_INFO: `/v3/roles` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:08:43.815166 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-208a2ba6-8495-459e-94a4-49a4fb216820 None admin] RBAC: Adding query filter params (name=member) {{(pid=81409) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 13 04:08:43.815668 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-208a2ba6-8495-459e-94a4-49a4fb216820 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:08:43.817788 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-208a2ba6-8495-459e-94a4-49a4fb216820 None admin] RBAC: Authorization granted {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:08:43.833658 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 157/310] 10.222.0.27 () {56 vars in 1101 bytes} [Thu Oct 13 04:08:43 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 224 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 13 04:08:43.835500 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-208a2ba6-8495-459e-94a4-49a4fb216820 None admin] Authenticating user token {{(pid=81409) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:08:43.941071 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG dbcounter [-] [81408] Writing DB stats keystone:SELECT=100 {{(pid=81408) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:08:43.950477 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c0c666ab-984d-49c3-b502-678c3c8aa294 None admin] Validating token access rules against request {{(pid=81408) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:08:43.986347 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-208a2ba6-8495-459e-94a4-49a4fb216820 None admin] Validating token access rules against request {{(pid=81409) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:08:44.006160 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7311b438-e304-4c8f-a82c-d9375e827f16 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81408) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:08:44.007164 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7311b438-e304-4c8f-a82c-d9375e827f16 None admin] REQUEST_METHOD: `GET` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:08:44.007644 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7311b438-e304-4c8f-a82c-d9375e827f16 None admin] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:08:44.008114 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7311b438-e304-4c8f-a82c-d9375e827f16 None admin] PATH_INFO: `/v3/groups/d232567016e94eeca7524d2c09a20c3b` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:08:44.009579 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7311b438-e304-4c8f-a82c-d9375e827f16 None admin] RBAC: Authorizing `identity:get_group(group_id=d232567016e94eeca7524d2c09a20c3b)` {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:08:44.011322 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7311b438-e304-4c8f-a82c-d9375e827f16 None admin] RBAC: Authorization granted {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:08:44.013144 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 154/311] 10.222.0.27 () {56 vars in 1180 bytes} [Thu Oct 13 04:08:43 2022] GET /identity/v3/groups/d232567016e94eeca7524d2c09a20c3b => generated 226 bytes in 206 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 13 04:08:44.014110 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7311b438-e304-4c8f-a82c-d9375e827f16 None admin] Authenticating user token {{(pid=81408) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:08:44.029391 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-edefb541-6a28-4318-8b31-9b0ec53ac60b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81409) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:08:44.030390 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-edefb541-6a28-4318-8b31-9b0ec53ac60b None admin] REQUEST_METHOD: `GET` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:08:44.030808 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-edefb541-6a28-4318-8b31-9b0ec53ac60b None admin] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:08:44.031238 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-edefb541-6a28-4318-8b31-9b0ec53ac60b None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:08:44.105890 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG dbcounter [-] [81409] Writing DB stats keystone:SELECT=100 {{(pid=81409) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:08:44.119746 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-edefb541-6a28-4318-8b31-9b0ec53ac60b None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:08:44.122093 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-edefb541-6a28-4318-8b31-9b0ec53ac60b None admin] RBAC: Authorization granted {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:08:44.161280 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7311b438-e304-4c8f-a82c-d9375e827f16 None admin] Validating token access rules against request {{(pid=81408) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:08:44.208122 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e38563ea-e91a-4f34-bf81-774387c29ae6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81408) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:08:44.209017 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e38563ea-e91a-4f34-bf81-774387c29ae6 None admin] REQUEST_METHOD: `GET` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:08:44.209443 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e38563ea-e91a-4f34-bf81-774387c29ae6 None admin] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:08:44.209861 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e38563ea-e91a-4f34-bf81-774387c29ae6 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:08:44.251510 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG dbcounter [-] [81408] Writing DB stats keystone:SELECT=100 {{(pid=81408) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:08:44.261086 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 158/312] 10.222.0.27 () {58 vars in 1282 bytes} [Thu Oct 13 04:08:43 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 427 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Oct 13 04:08:44.262644 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-edefb541-6a28-4318-8b31-9b0ec53ac60b None admin] Authenticating user token {{(pid=81409) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:08:44.297347 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e38563ea-e91a-4f34-bf81-774387c29ae6 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:08:44.299136 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e38563ea-e91a-4f34-bf81-774387c29ae6 None admin] RBAC: Authorization granted {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:08:44.380487 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG dbcounter [-] [81409] Writing DB stats keystone:SELECT=100 {{(pid=81409) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:08:44.390075 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-edefb541-6a28-4318-8b31-9b0ec53ac60b None admin] Validating token access rules against request {{(pid=81409) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:08:44.432699 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 155/313] 10.222.0.27 () {58 vars in 1282 bytes} [Thu Oct 13 04:08:44 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 419 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Oct 13 04:08:44.433584 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e38563ea-e91a-4f34-bf81-774387c29ae6 None admin] Authenticating user token {{(pid=81408) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:08:44.452726 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3922aca0-0487-4169-8400-a681dc023d1b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81409) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:08:44.453613 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3922aca0-0487-4169-8400-a681dc023d1b None admin] REQUEST_METHOD: `GET` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:08:44.454028 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3922aca0-0487-4169-8400-a681dc023d1b None admin] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:08:44.454526 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3922aca0-0487-4169-8400-a681dc023d1b None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:08:44.455397 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3922aca0-0487-4169-8400-a681dc023d1b None admin] RBAC: Adding query filter params (group.id=d232567016e94eeca7524d2c09a20c3b, role.id=743995e93fef47fd91204c3d701054b0, scope.project.id=eeec419e34e44c1f881179933c828d97) {{(pid=81409) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 13 04:08:44.455860 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3922aca0-0487-4169-8400-a681dc023d1b None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:08:44.457311 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3922aca0-0487-4169-8400-a681dc023d1b None admin] RBAC: Authorization granted {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:08:44.464581 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 159/314] 10.222.0.27 () {56 vars in 1376 bytes} [Thu Oct 13 04:08:44 2022] GET /identity/v3/role_assignments?group.id=d232567016e94eeca7524d2c09a20c3b&scope.project.id=eeec419e34e44c1f881179933c828d97&role.id=743995e93fef47fd91204c3d701054b0 => generated 611 bytes in 203 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 13 04:08:44.465650 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a7d7833-df56-45d0-a441-715a75bda0f6 None None] REQUEST_METHOD: `GET` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:08:44.465930 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a7d7833-df56-45d0-a441-715a75bda0f6 None None] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:08:44.466220 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a7d7833-df56-45d0-a441-715a75bda0f6 None None] PATH_INFO: `/` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:08:44.469021 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 160/315] 10.222.0.27 () {54 vars in 927 bytes} [Thu Oct 13 04:08:44 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 13 04:08:44.469673 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6a7d7833-df56-45d0-a441-715a75bda0f6 None None] Authenticating user token {{(pid=81409) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:08:44.519454 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG dbcounter [-] [81408] Writing DB stats keystone:SELECT=100 {{(pid=81408) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:08:44.583463 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e38563ea-e91a-4f34-bf81-774387c29ae6 None admin] Validating token access rules against request {{(pid=81408) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:08:44.606217 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6a7d7833-df56-45d0-a441-715a75bda0f6 None None] Validating token access rules against request {{(pid=81409) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:08:44.635366 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ee51db4d-97b8-46ea-b79c-ce940f230d6d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81408) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:08:44.636094 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee51db4d-97b8-46ea-b79c-ce940f230d6d None admin] REQUEST_METHOD: `GET` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:08:44.636467 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee51db4d-97b8-46ea-b79c-ce940f230d6d None admin] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:08:44.636835 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee51db4d-97b8-46ea-b79c-ce940f230d6d None admin] PATH_INFO: `/v3/projects/eeec419e34e44c1f881179933c828d97` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:08:44.638725 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ee51db4d-97b8-46ea-b79c-ce940f230d6d None admin] RBAC: Authorizing `identity:get_project(project_id=eeec419e34e44c1f881179933c828d97)` {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:08:44.640497 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ee51db4d-97b8-46ea-b79c-ce940f230d6d None admin] RBAC: Authorization granted {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:08:44.642936 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 156/316] 10.222.0.27 () {56 vars in 1186 bytes} [Thu Oct 13 04:08:44 2022] GET /identity/v3/projects/eeec419e34e44c1f881179933c828d97 => generated 298 bytes in 210 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 13 04:08:44.644264 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf3868a4-ad38-4959-a4b9-87fbd2f7d872 None None] REQUEST_METHOD: `POST` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:08:44.644650 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf3868a4-ad38-4959-a4b9-87fbd2f7d872 None None] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:08:44.645063 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf3868a4-ad38-4959-a4b9-87fbd2f7d872 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:08:44.657273 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1f1bb05b-451e-4fd1-83a4-0eb32d46efb2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81409) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:08:44.658173 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f1bb05b-451e-4fd1-83a4-0eb32d46efb2 None admin] REQUEST_METHOD: `GET` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:08:44.658643 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f1bb05b-451e-4fd1-83a4-0eb32d46efb2 None admin] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:08:44.659056 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f1bb05b-451e-4fd1-83a4-0eb32d46efb2 None admin] PATH_INFO: `/v3/projects/0114058aecc645ea8b47ae6d1301c17d` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:08:44.660444 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1f1bb05b-451e-4fd1-83a4-0eb32d46efb2 None admin] RBAC: Authorizing `identity:get_project(project_id=0114058aecc645ea8b47ae6d1301c17d)` {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:08:44.662863 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1f1bb05b-451e-4fd1-83a4-0eb32d46efb2 None admin] RBAC: Authorization granted {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:08:44.665679 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 161/317] 10.222.0.27 () {56 vars in 1186 bytes} [Thu Oct 13 04:08:44 2022] GET /identity/v3/projects/0114058aecc645ea8b47ae6d1301c17d => generated 302 bytes in 196 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 13 04:08:44.666886 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1f1bb05b-451e-4fd1-83a4-0eb32d46efb2 None admin] Authenticating user token {{(pid=81409) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:08:44.669204 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.auth.core [None req-cf3868a4-ad38-4959-a4b9-87fbd2f7d872 None None] MFA Rules not processed for user `0e2d979563d6421995451a0e86bf3022`. Rule list: `[]` (Enabled: `True`). {{(pid=81408) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 13 04:08:44.671719 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.fernet_utils [None req-cf3868a4-ad38-4959-a4b9-87fbd2f7d872 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81408) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:08:44.701869 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG dbcounter [-] [81409] Writing DB stats keystone:SELECT=100 {{(pid=81409) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:08:44.779888 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 157/318] 10.222.0.27 () {58 vars in 1024 bytes} [Thu Oct 13 04:08:44 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 136 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 13 04:08:44.781818 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cf3868a4-ad38-4959-a4b9-87fbd2f7d872 None None] Authenticating user token {{(pid=81408) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:08:44.820694 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1f1bb05b-451e-4fd1-83a4-0eb32d46efb2 None admin] Validating token access rules against request {{(pid=81409) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:08:44.878155 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-747d09fb-a64f-418b-b202-4e3d64026e66 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81409) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:08:44.879156 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-747d09fb-a64f-418b-b202-4e3d64026e66 None admin] REQUEST_METHOD: `GET` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:08:44.880363 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-747d09fb-a64f-418b-b202-4e3d64026e66 None admin] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:08:44.880941 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-747d09fb-a64f-418b-b202-4e3d64026e66 None admin] PATH_INFO: `/v3/groups/d232567016e94eeca7524d2c09a20c3b` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:08:44.883221 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-747d09fb-a64f-418b-b202-4e3d64026e66 None admin] RBAC: Authorizing `identity:get_group(group_id=d232567016e94eeca7524d2c09a20c3b)` {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:08:44.884771 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG dbcounter [-] [81408] Writing DB stats keystone:SELECT=100 {{(pid=81408) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:08:44.885894 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-747d09fb-a64f-418b-b202-4e3d64026e66 None admin] RBAC: Authorization granted {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:08:44.888926 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 162/319] 10.222.0.27 () {56 vars in 1180 bytes} [Thu Oct 13 04:08:44 2022] GET /identity/v3/groups/d232567016e94eeca7524d2c09a20c3b => generated 226 bytes in 222 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 13 04:08:44.891742 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9dde9251-52b9-473d-a3a9-cdba43c2c5e6 None None] REQUEST_METHOD: `GET` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:08:44.892112 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9dde9251-52b9-473d-a3a9-cdba43c2c5e6 None None] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:08:44.892383 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9dde9251-52b9-473d-a3a9-cdba43c2c5e6 None None] PATH_INFO: `/` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:08:44.894321 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 163/320] 10.222.0.27 () {54 vars in 927 bytes} [Thu Oct 13 04:08:44 2022] GET /identity => generated 270 bytes in 5 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 13 04:08:44.895165 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9dde9251-52b9-473d-a3a9-cdba43c2c5e6 None None] Authenticating user token {{(pid=81409) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:08:44.939596 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cf3868a4-ad38-4959-a4b9-87fbd2f7d872 None None] Validating token access rules against request {{(pid=81408) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:08:44.980650 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-55b25b7d-4933-4f48-96ed-4be543c72ae5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81408) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:08:44.981296 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55b25b7d-4933-4f48-96ed-4be543c72ae5 None admin] REQUEST_METHOD: `GET` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:08:44.981566 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55b25b7d-4933-4f48-96ed-4be543c72ae5 None admin] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:08:44.982553 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55b25b7d-4933-4f48-96ed-4be543c72ae5 None admin] PATH_INFO: `/v3/groups/d232567016e94eeca7524d2c09a20c3b` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:08:44.983752 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-55b25b7d-4933-4f48-96ed-4be543c72ae5 None admin] RBAC: Authorizing `identity:get_group(group_id=d232567016e94eeca7524d2c09a20c3b)` {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:08:44.985179 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-55b25b7d-4933-4f48-96ed-4be543c72ae5 None admin] RBAC: Authorization granted {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:08:44.987436 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 158/321] 10.222.0.27 () {56 vars in 1180 bytes} [Thu Oct 13 04:08:44 2022] GET /identity/v3/groups/d232567016e94eeca7524d2c09a20c3b => generated 226 bytes in 206 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 13 04:08:44.992838 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b6985cd-1121-410a-abfb-af2798aecbfc None None] REQUEST_METHOD: `POST` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:08:44.993737 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b6985cd-1121-410a-abfb-af2798aecbfc None None] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:08:44.993997 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b6985cd-1121-410a-abfb-af2798aecbfc None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:08:45.018197 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.auth.core [None req-0b6985cd-1121-410a-abfb-af2798aecbfc None None] MFA Rules not processed for user `0e2d979563d6421995451a0e86bf3022`. Rule list: `[]` (Enabled: `True`). {{(pid=81408) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 13 04:08:45.019447 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.fernet_utils [None req-0b6985cd-1121-410a-abfb-af2798aecbfc None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81408) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:08:45.028580 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG dbcounter [-] [81409] Writing DB stats keystone:SELECT=100 {{(pid=81409) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:08:45.046735 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9dde9251-52b9-473d-a3a9-cdba43c2c5e6 None None] Validating token access rules against request {{(pid=81409) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:08:45.098016 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e561d7c8-6226-40db-a7eb-465ab56a704a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81409) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:08:45.098771 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e561d7c8-6226-40db-a7eb-465ab56a704a None admin] REQUEST_METHOD: `GET` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:08:45.099055 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e561d7c8-6226-40db-a7eb-465ab56a704a None admin] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:08:45.099580 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e561d7c8-6226-40db-a7eb-465ab56a704a None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:08:45.100437 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e561d7c8-6226-40db-a7eb-465ab56a704a None admin] RBAC: Adding query filter params (group.id=d232567016e94eeca7524d2c09a20c3b, role.id=cc56c4ae13a7424ea771c15da4b153c1, scope.project.id=eeec419e34e44c1f881179933c828d97) {{(pid=81409) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 13 04:08:45.100806 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e561d7c8-6226-40db-a7eb-465ab56a704a None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:08:45.102284 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e561d7c8-6226-40db-a7eb-465ab56a704a None admin] RBAC: Authorization granted {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:08:45.110037 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 164/322] 10.222.0.27 () {56 vars in 1376 bytes} [Thu Oct 13 04:08:44 2022] GET /identity/v3/role_assignments?group.id=d232567016e94eeca7524d2c09a20c3b&scope.project.id=eeec419e34e44c1f881179933c828d97&role.id=cc56c4ae13a7424ea771c15da4b153c1 => generated 611 bytes in 215 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 13 04:08:45.110748 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e561d7c8-6226-40db-a7eb-465ab56a704a None admin] Authenticating user token {{(pid=81409) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:08:45.114408 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 159/323] 10.222.0.27 () {58 vars in 1024 bytes} [Thu Oct 13 04:08:44 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 127 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 13 04:08:45.118690 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0b6985cd-1121-410a-abfb-af2798aecbfc None None] Authenticating user token {{(pid=81408) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:08:45.208507 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG dbcounter [-] [81408] Writing DB stats keystone:SELECT=100 {{(pid=81408) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:08:45.268797 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e561d7c8-6226-40db-a7eb-465ab56a704a None admin] Validating token access rules against request {{(pid=81409) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:08:45.270077 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0b6985cd-1121-410a-abfb-af2798aecbfc None None] Validating token access rules against request {{(pid=81408) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:08:45.320867 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a7f63b96-6097-4b74-97a8-c65fe7fba23a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81409) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:08:45.321731 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7f63b96-6097-4b74-97a8-c65fe7fba23a None admin] REQUEST_METHOD: `GET` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:08:45.322008 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7f63b96-6097-4b74-97a8-c65fe7fba23a None admin] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:08:45.322403 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7f63b96-6097-4b74-97a8-c65fe7fba23a None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:08:45.323139 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a7f63b96-6097-4b74-97a8-c65fe7fba23a None admin] RBAC: Adding query filter params (group.id=d232567016e94eeca7524d2c09a20c3b, role.id=cc56c4ae13a7424ea771c15da4b153c1, scope.project.id=0114058aecc645ea8b47ae6d1301c17d) {{(pid=81409) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 13 04:08:45.323624 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a7f63b96-6097-4b74-97a8-c65fe7fba23a None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:08:45.324888 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a7f63b96-6097-4b74-97a8-c65fe7fba23a None admin] RBAC: Authorization granted {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:08:45.327705 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-97c9770a-aa7a-425b-a1c1-e82c2e8eb4ac None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81408) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:08:45.328371 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97c9770a-aa7a-425b-a1c1-e82c2e8eb4ac None admin] REQUEST_METHOD: `GET` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:08:45.328642 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97c9770a-aa7a-425b-a1c1-e82c2e8eb4ac None admin] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:08:45.329901 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97c9770a-aa7a-425b-a1c1-e82c2e8eb4ac None admin] PATH_INFO: `/v3/roles/anotherrole` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:08:45.340554 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 165/324] 10.222.0.27 () {56 vars in 1376 bytes} [Thu Oct 13 04:08:45 2022] GET /identity/v3/role_assignments?group.id=d232567016e94eeca7524d2c09a20c3b&scope.project.id=0114058aecc645ea8b47ae6d1301c17d&role.id=cc56c4ae13a7424ea771c15da4b153c1 => generated 611 bytes in 230 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 13 04:08:45.345358 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-97c9770a-aa7a-425b-a1c1-e82c2e8eb4ac None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 13 04:08:45.351933 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application [None req-97c9770a-aa7a-425b-a1c1-e82c2e8eb4ac None admin] Could not find role: anotherrole.: keystone.exception.RoleNotFound: Could not find role: anotherrole. Oct 13 04:08:45.351933 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 13 04:08:45.351933 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 13 04:08:45.351933 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 13 04:08:45.351933 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 13 04:08:45.351933 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 13 04:08:45.351933 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 13 04:08:45.351933 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 13 04:08:45.351933 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 13 04:08:45.351933 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 13 04:08:45.351933 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 13 04:08:45.351933 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 13 04:08:45.351933 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 13 04:08:45.351933 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 13 04:08:45.351933 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 13 04:08:45.351933 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 13 04:08:45.353935 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 13 04:08:45.353935 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 13 04:08:45.353935 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 13 04:08:45.353935 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 13 04:08:45.353935 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 13 04:08:45.353935 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 13 04:08:45.353935 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 13 04:08:45.353935 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 13 04:08:45.353935 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 13 04:08:45.353935 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application return self.get_or_create( Oct 13 04:08:45.353935 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 13 04:08:45.353935 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application with Lock( Oct 13 04:08:45.353935 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 13 04:08:45.353935 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application return self._enter() Oct 13 04:08:45.353935 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 13 04:08:45.353935 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 13 04:08:45.354856 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 13 04:08:45.354856 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application return self.creator() Oct 13 04:08:45.354856 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 13 04:08:45.354856 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application created_value = creator( Oct 13 04:08:45.354856 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 13 04:08:45.354856 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 13 04:08:45.354856 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 13 04:08:45.354856 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 13 04:08:45.354856 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 13 04:08:45.354856 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 13 04:08:45.354856 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: anotherrole. Oct 13 04:08:45.354856 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application Oct 13 04:08:45.357599 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 160/325] 10.222.0.27 () {56 vars in 1114 bytes} [Thu Oct 13 04:08:45 2022] GET /identity/v3/roles/anotherrole => generated 89 bytes in 243 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 13 04:08:45.364348 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a7f63b96-6097-4b74-97a8-c65fe7fba23a None admin] Authenticating user token {{(pid=81409) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:08:45.390073 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG dbcounter [-] [81409] Writing DB stats keystone:SELECT=100 {{(pid=81409) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:08:45.446537 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-97c9770a-aa7a-425b-a1c1-e82c2e8eb4ac None admin] Authenticating user token {{(pid=81408) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:08:45.521799 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a7f63b96-6097-4b74-97a8-c65fe7fba23a None admin] Validating token access rules against request {{(pid=81409) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:08:45.576197 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cef2b086-d7b7-4abd-8cc8-41a25be27f75 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81409) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:08:45.576824 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cef2b086-d7b7-4abd-8cc8-41a25be27f75 None admin] REQUEST_METHOD: `GET` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:08:45.577104 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cef2b086-d7b7-4abd-8cc8-41a25be27f75 None admin] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:08:45.577710 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cef2b086-d7b7-4abd-8cc8-41a25be27f75 None admin] PATH_INFO: `/v3/roles` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:08:45.578459 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cef2b086-d7b7-4abd-8cc8-41a25be27f75 None admin] RBAC: Adding query filter params (name=anotherrole) {{(pid=81409) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 13 04:08:45.579057 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cef2b086-d7b7-4abd-8cc8-41a25be27f75 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:08:45.580454 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cef2b086-d7b7-4abd-8cc8-41a25be27f75 None admin] RBAC: Authorization granted {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:08:45.592050 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 166/326] 10.222.0.27 () {56 vars in 1111 bytes} [Thu Oct 13 04:08:45 2022] GET /identity/v3/roles?name=anotherrole => generated 334 bytes in 229 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 13 04:08:45.598719 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cef2b086-d7b7-4abd-8cc8-41a25be27f75 None admin] Authenticating user token {{(pid=81409) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:08:45.606933 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-97c9770a-aa7a-425b-a1c1-e82c2e8eb4ac None admin] Validating token access rules against request {{(pid=81408) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:08:45.634164 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG dbcounter [-] [81408] Writing DB stats keystone:SELECT=100 {{(pid=81408) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:08:45.650414 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fdc09e21-0d39-4a8a-945b-7dcd1a1120e7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81408) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:08:45.651376 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fdc09e21-0d39-4a8a-945b-7dcd1a1120e7 None admin] REQUEST_METHOD: `GET` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:08:45.651825 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fdc09e21-0d39-4a8a-945b-7dcd1a1120e7 None admin] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:08:45.652270 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fdc09e21-0d39-4a8a-945b-7dcd1a1120e7 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:08:45.665523 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fdc09e21-0d39-4a8a-945b-7dcd1a1120e7 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 13 04:08:45.671755 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application [None req-fdc09e21-0d39-4a8a-945b-7dcd1a1120e7 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Oct 13 04:08:45.671755 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 13 04:08:45.671755 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 13 04:08:45.671755 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 13 04:08:45.671755 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 13 04:08:45.671755 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 13 04:08:45.671755 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 13 04:08:45.671755 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 13 04:08:45.671755 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 13 04:08:45.671755 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 13 04:08:45.671755 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 13 04:08:45.671755 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 13 04:08:45.671755 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 13 04:08:45.671755 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 13 04:08:45.671755 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 13 04:08:45.671755 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 13 04:08:45.672662 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 13 04:08:45.672662 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 13 04:08:45.672662 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 13 04:08:45.672662 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 13 04:08:45.672662 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 13 04:08:45.672662 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 13 04:08:45.672662 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 13 04:08:45.672662 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 13 04:08:45.672662 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 13 04:08:45.672662 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application return self.get_or_create( Oct 13 04:08:45.672662 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 13 04:08:45.672662 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application with Lock( Oct 13 04:08:45.672662 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 13 04:08:45.672662 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application return self._enter() Oct 13 04:08:45.672662 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 13 04:08:45.672662 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 13 04:08:45.673453 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 13 04:08:45.673453 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application return self.creator() Oct 13 04:08:45.673453 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 13 04:08:45.673453 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application created_value = creator( Oct 13 04:08:45.673453 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 13 04:08:45.673453 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 13 04:08:45.673453 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 13 04:08:45.673453 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 13 04:08:45.673453 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 13 04:08:45.673453 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 13 04:08:45.673453 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Oct 13 04:08:45.673453 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application Oct 13 04:08:45.673453 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 161/327] 10.222.0.27 () {56 vars in 1096 bytes} [Thu Oct 13 04:08:45 2022] GET /identity/v3/roles/admin => generated 83 bytes in 228 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 13 04:08:45.679228 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fdc09e21-0d39-4a8a-945b-7dcd1a1120e7 None admin] Authenticating user token {{(pid=81408) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:08:45.717607 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG dbcounter [-] [81409] Writing DB stats keystone:SELECT=100 {{(pid=81409) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:08:45.745010 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cef2b086-d7b7-4abd-8cc8-41a25be27f75 None admin] Validating token access rules against request {{(pid=81409) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:08:45.786729 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e0909db7-9f7f-4e72-a4c0-045bab5ab705 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81409) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:08:45.787255 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0909db7-9f7f-4e72-a4c0-045bab5ab705 None admin] REQUEST_METHOD: `GET` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:08:45.787531 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0909db7-9f7f-4e72-a4c0-045bab5ab705 None admin] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:08:45.787799 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0909db7-9f7f-4e72-a4c0-045bab5ab705 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:08:45.804228 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fdc09e21-0d39-4a8a-945b-7dcd1a1120e7 None admin] Validating token access rules against request {{(pid=81408) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:08:45.846100 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7914bb40-9a99-4b8e-82ee-8956fa6dbfc1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81408) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:08:45.846841 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7914bb40-9a99-4b8e-82ee-8956fa6dbfc1 None admin] REQUEST_METHOD: `GET` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:08:45.847031 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7914bb40-9a99-4b8e-82ee-8956fa6dbfc1 None admin] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:08:45.847297 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7914bb40-9a99-4b8e-82ee-8956fa6dbfc1 None admin] PATH_INFO: `/v3/roles` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:08:45.848025 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7914bb40-9a99-4b8e-82ee-8956fa6dbfc1 None admin] RBAC: Adding query filter params (name=admin) {{(pid=81408) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 13 04:08:45.848253 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7914bb40-9a99-4b8e-82ee-8956fa6dbfc1 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:08:45.849884 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7914bb40-9a99-4b8e-82ee-8956fa6dbfc1 None admin] RBAC: Authorization granted {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:08:45.860769 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 162/328] 10.222.0.27 () {56 vars in 1099 bytes} [Thu Oct 13 04:08:45 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 182 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 13 04:08:45.866679 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7914bb40-9a99-4b8e-82ee-8956fa6dbfc1 None admin] Authenticating user token {{(pid=81408) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:08:45.875231 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e0909db7-9f7f-4e72-a4c0-045bab5ab705 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:08:45.876722 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e0909db7-9f7f-4e72-a4c0-045bab5ab705 None admin] RBAC: Authorization granted {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:08:45.919748 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG dbcounter [-] [81408] Writing DB stats keystone:SELECT=100 {{(pid=81408) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:08:45.968675 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG dbcounter [-] [81409] Writing DB stats keystone:SELECT=100 {{(pid=81409) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:08:45.986257 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7914bb40-9a99-4b8e-82ee-8956fa6dbfc1 None admin] Validating token access rules against request {{(pid=81408) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:08:45.998421 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 167/329] 10.222.0.27 () {58 vars in 1282 bytes} [Thu Oct 13 04:08:45 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 401 msecs (HTTP/1.1 200) 6 headers in 358 bytes (2 switches on core 0) Oct 13 04:08:46.003075 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e0909db7-9f7f-4e72-a4c0-045bab5ab705 None admin] Authenticating user token {{(pid=81409) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:08:46.022364 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-96e42906-e236-4c00-b893-75fcdff5b457 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81408) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:08:46.023390 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96e42906-e236-4c00-b893-75fcdff5b457 None admin] REQUEST_METHOD: `GET` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:08:46.023671 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96e42906-e236-4c00-b893-75fcdff5b457 None admin] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:08:46.023949 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96e42906-e236-4c00-b893-75fcdff5b457 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:08:46.108524 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-96e42906-e236-4c00-b893-75fcdff5b457 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:08:46.109918 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-96e42906-e236-4c00-b893-75fcdff5b457 None admin] RBAC: Authorization granted {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:08:46.124076 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e0909db7-9f7f-4e72-a4c0-045bab5ab705 None admin] Validating token access rules against request {{(pid=81409) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:08:46.160071 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG dbcounter [-] [81408] Writing DB stats keystone:SELECT=100 {{(pid=81408) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:08:46.163864 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-86b7a033-811f-4ec4-befc-621230361124 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81409) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:08:46.164553 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-86b7a033-811f-4ec4-befc-621230361124 None admin] REQUEST_METHOD: `GET` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:08:46.164910 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-86b7a033-811f-4ec4-befc-621230361124 None admin] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:08:46.165696 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-86b7a033-811f-4ec4-befc-621230361124 None admin] PATH_INFO: `/v3/projects/0114058aecc645ea8b47ae6d1301c17d` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:08:46.166853 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-86b7a033-811f-4ec4-befc-621230361124 None admin] RBAC: Authorizing `identity:get_project(project_id=0114058aecc645ea8b47ae6d1301c17d)` {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:08:46.168304 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-86b7a033-811f-4ec4-befc-621230361124 None admin] RBAC: Authorization granted {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:08:46.169953 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 168/330] 10.222.0.27 () {56 vars in 1186 bytes} [Thu Oct 13 04:08:46 2022] GET /identity/v3/projects/0114058aecc645ea8b47ae6d1301c17d => generated 302 bytes in 167 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 13 04:08:46.175380 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-86b7a033-811f-4ec4-befc-621230361124 None admin] Authenticating user token {{(pid=81409) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:08:46.218135 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 163/331] 10.222.0.27 () {58 vars in 1282 bytes} [Thu Oct 13 04:08:45 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 352 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Oct 13 04:08:46.222871 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG dbcounter [-] [81409] Writing DB stats keystone:SELECT=100 {{(pid=81409) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:08:46.223563 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-96e42906-e236-4c00-b893-75fcdff5b457 None admin] Authenticating user token {{(pid=81408) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:08:46.276502 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-86b7a033-811f-4ec4-befc-621230361124 None admin] Validating token access rules against request {{(pid=81409) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:08:46.309719 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dad7c1f5-c07c-41c6-918a-fab329082b6e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81409) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:08:46.310371 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dad7c1f5-c07c-41c6-918a-fab329082b6e None admin] REQUEST_METHOD: `GET` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:08:46.310646 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dad7c1f5-c07c-41c6-918a-fab329082b6e None admin] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:08:46.310914 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dad7c1f5-c07c-41c6-918a-fab329082b6e None admin] PATH_INFO: `/v3/groups/d232567016e94eeca7524d2c09a20c3b` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:08:46.312005 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dad7c1f5-c07c-41c6-918a-fab329082b6e None admin] RBAC: Authorizing `identity:get_group(group_id=d232567016e94eeca7524d2c09a20c3b)` {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:08:46.313425 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dad7c1f5-c07c-41c6-918a-fab329082b6e None admin] RBAC: Authorization granted {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:08:46.315032 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 169/332] 10.222.0.27 () {56 vars in 1180 bytes} [Thu Oct 13 04:08:46 2022] GET /identity/v3/groups/d232567016e94eeca7524d2c09a20c3b => generated 226 bytes in 140 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 13 04:08:46.320303 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-96e42906-e236-4c00-b893-75fcdff5b457 None admin] Validating token access rules against request {{(pid=81408) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:08:46.320645 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dad7c1f5-c07c-41c6-918a-fab329082b6e None admin] Authenticating user token {{(pid=81409) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:08:46.354726 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0463761c-2d8e-4044-a426-25d68df01b33 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81408) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:08:46.355473 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0463761c-2d8e-4044-a426-25d68df01b33 None admin] REQUEST_METHOD: `GET` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:08:46.355754 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0463761c-2d8e-4044-a426-25d68df01b33 None admin] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:08:46.356024 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0463761c-2d8e-4044-a426-25d68df01b33 None admin] PATH_INFO: `/v3/projects/8ba823f313144da3b683bacef8fd62e0` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:08:46.357142 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0463761c-2d8e-4044-a426-25d68df01b33 None admin] RBAC: Authorizing `identity:get_project(project_id=8ba823f313144da3b683bacef8fd62e0)` {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:08:46.358697 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0463761c-2d8e-4044-a426-25d68df01b33 None admin] RBAC: Authorization granted {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:08:46.360270 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 164/333] 10.222.0.27 () {56 vars in 1186 bytes} [Thu Oct 13 04:08:46 2022] GET /identity/v3/projects/8ba823f313144da3b683bacef8fd62e0 => generated 344 bytes in 137 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 13 04:08:46.365727 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0463761c-2d8e-4044-a426-25d68df01b33 None admin] Authenticating user token {{(pid=81408) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:08:46.377490 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG dbcounter [-] [81408] Writing DB stats keystone:SELECT=100 {{(pid=81408) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:08:46.423779 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dad7c1f5-c07c-41c6-918a-fab329082b6e None admin] Validating token access rules against request {{(pid=81409) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:08:46.441192 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG dbcounter [-] [81409] Writing DB stats keystone:SELECT=100 {{(pid=81409) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:08:46.460945 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-afdc4c69-3172-4211-8710-397f570465dc None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81409) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:08:46.461586 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-afdc4c69-3172-4211-8710-397f570465dc None admin] REQUEST_METHOD: `GET` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:08:46.462509 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-afdc4c69-3172-4211-8710-397f570465dc None admin] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:08:46.462774 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-afdc4c69-3172-4211-8710-397f570465dc None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:08:46.463495 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-afdc4c69-3172-4211-8710-397f570465dc None admin] RBAC: Adding query filter params (group.id=d232567016e94eeca7524d2c09a20c3b, role.id=743995e93fef47fd91204c3d701054b0, scope.project.id=0114058aecc645ea8b47ae6d1301c17d) {{(pid=81409) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 13 04:08:46.463792 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-afdc4c69-3172-4211-8710-397f570465dc None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:08:46.465214 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-afdc4c69-3172-4211-8710-397f570465dc None admin] RBAC: Authorization granted {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:08:46.471587 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 170/334] 10.222.0.27 () {56 vars in 1376 bytes} [Thu Oct 13 04:08:46 2022] GET /identity/v3/role_assignments?group.id=d232567016e94eeca7524d2c09a20c3b&scope.project.id=0114058aecc645ea8b47ae6d1301c17d&role.id=743995e93fef47fd91204c3d701054b0 => generated 611 bytes in 151 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 13 04:08:46.478674 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0463761c-2d8e-4044-a426-25d68df01b33 None admin] Validating token access rules against request {{(pid=81408) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:08:46.516572 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f0dd5b7-f773-4e5b-b816-ea738b135736 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81408) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:08:46.517216 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f0dd5b7-f773-4e5b-b816-ea738b135736 None admin] REQUEST_METHOD: `GET` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:08:46.517496 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f0dd5b7-f773-4e5b-b816-ea738b135736 None admin] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:08:46.517771 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f0dd5b7-f773-4e5b-b816-ea738b135736 None admin] PATH_INFO: `/v3/groups/63a8f28fb01c46fab610e79058cf47ae` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:08:46.518933 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5f0dd5b7-f773-4e5b-b816-ea738b135736 None admin] RBAC: Authorizing `identity:get_group(group_id=63a8f28fb01c46fab610e79058cf47ae)` {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:08:46.520413 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5f0dd5b7-f773-4e5b-b816-ea738b135736 None admin] RBAC: Authorization granted {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:08:46.522052 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 165/335] 10.222.0.27 () {56 vars in 1180 bytes} [Thu Oct 13 04:08:46 2022] GET /identity/v3/groups/63a8f28fb01c46fab610e79058cf47ae => generated 229 bytes in 157 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 13 04:08:46.529136 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-afdc4c69-3172-4211-8710-397f570465dc None admin] Authenticating user token {{(pid=81409) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:08:46.634958 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-afdc4c69-3172-4211-8710-397f570465dc None admin] Validating token access rules against request {{(pid=81409) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:08:46.669014 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-26f1b79e-cb43-4103-b3fc-0fd8b3ffa125 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81409) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:08:46.669886 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26f1b79e-cb43-4103-b3fc-0fd8b3ffa125 None admin] REQUEST_METHOD: `GET` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:08:46.670164 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26f1b79e-cb43-4103-b3fc-0fd8b3ffa125 None admin] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:08:46.670464 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26f1b79e-cb43-4103-b3fc-0fd8b3ffa125 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:08:46.671200 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-26f1b79e-cb43-4103-b3fc-0fd8b3ffa125 None admin] RBAC: Adding query filter params (group.id=63a8f28fb01c46fab610e79058cf47ae, role.id=ef55fbd94e1040be83695754075a7ed5, scope.project.id=8ba823f313144da3b683bacef8fd62e0) {{(pid=81409) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 13 04:08:46.671508 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-26f1b79e-cb43-4103-b3fc-0fd8b3ffa125 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:08:46.672941 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-26f1b79e-cb43-4103-b3fc-0fd8b3ffa125 None admin] RBAC: Authorization granted {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:08:46.678877 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 171/336] 10.222.0.27 () {56 vars in 1376 bytes} [Thu Oct 13 04:08:46 2022] GET /identity/v3/role_assignments?group.id=63a8f28fb01c46fab610e79058cf47ae&scope.project.id=8ba823f313144da3b683bacef8fd62e0&role.id=ef55fbd94e1040be83695754075a7ed5 => generated 611 bytes in 150 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 13 04:08:49.268483 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3756f616-16e2-46b2-b8d5-b914886c668d None None] REQUEST_METHOD: `GET` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:08:49.269104 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3756f616-16e2-46b2-b8d5-b914886c668d None None] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:08:49.269104 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3756f616-16e2-46b2-b8d5-b914886c668d None None] PATH_INFO: `/` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:08:49.270722 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 166/337] 10.222.0.27 () {54 vars in 927 bytes} [Thu Oct 13 04:08:49 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 13 04:08:49.276021 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07cbcf27-8931-4137-bf52-65b16aa66f00 None None] REQUEST_METHOD: `POST` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:08:49.276877 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07cbcf27-8931-4137-bf52-65b16aa66f00 None None] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:08:49.277439 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07cbcf27-8931-4137-bf52-65b16aa66f00 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:08:49.306421 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.auth.core [None req-07cbcf27-8931-4137-bf52-65b16aa66f00 None None] MFA Rules not processed for user `0e2d979563d6421995451a0e86bf3022`. Rule list: `[]` (Enabled: `True`). {{(pid=81409) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 13 04:08:49.307959 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.fernet_utils [None req-07cbcf27-8931-4137-bf52-65b16aa66f00 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81409) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:08:49.363416 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG dbcounter [-] [81409] Writing DB stats keystone:SELECT=100 {{(pid=81409) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:08:49.404195 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 172/338] 10.222.0.27 () {58 vars in 1024 bytes} [Thu Oct 13 04:08:49 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 129 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 13 04:08:49.612281 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c60874c0-bd0d-4125-9966-7a72e8589fb0 None None] REQUEST_METHOD: `GET` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:08:49.612944 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c60874c0-bd0d-4125-9966-7a72e8589fb0 None None] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:08:49.613431 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c60874c0-bd0d-4125-9966-7a72e8589fb0 None None] PATH_INFO: `/` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:08:49.615283 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 167/339] 10.222.0.27 () {54 vars in 927 bytes} [Thu Oct 13 04:08:49 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 13 04:08:49.620895 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1de8ea8-30f6-4de5-babd-4f35b1854ebf None None] REQUEST_METHOD: `GET` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:08:49.621110 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-670e9c1d-108f-4af9-8c72-d2824ee41aeb None None] REQUEST_METHOD: `POST` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:08:49.621393 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1de8ea8-30f6-4de5-babd-4f35b1854ebf None None] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:08:49.621668 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1de8ea8-30f6-4de5-babd-4f35b1854ebf None None] PATH_INFO: `/` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:08:49.622030 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-670e9c1d-108f-4af9-8c72-d2824ee41aeb None None] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:08:49.622516 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-670e9c1d-108f-4af9-8c72-d2824ee41aeb None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:08:49.623199 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 168/340] 10.222.0.27 () {54 vars in 927 bytes} [Thu Oct 13 04:08:49 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 13 04:08:49.628601 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5676a9e-ab0c-4d06-a660-2aabd862c2ef None None] REQUEST_METHOD: `POST` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:08:49.628882 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5676a9e-ab0c-4d06-a660-2aabd862c2ef None None] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:08:49.629158 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5676a9e-ab0c-4d06-a660-2aabd862c2ef None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:08:49.663195 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.auth.core [None req-670e9c1d-108f-4af9-8c72-d2824ee41aeb None None] MFA Rules not processed for user `0e2d979563d6421995451a0e86bf3022`. Rule list: `[]` (Enabled: `True`). {{(pid=81409) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 13 04:08:49.664418 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.fernet_utils [None req-670e9c1d-108f-4af9-8c72-d2824ee41aeb None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81409) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:08:49.669036 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.auth.core [None req-a5676a9e-ab0c-4d06-a660-2aabd862c2ef None None] MFA Rules not processed for user `0e2d979563d6421995451a0e86bf3022`. Rule list: `[]` (Enabled: `True`). {{(pid=81408) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 13 04:08:49.670226 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.fernet_utils [None req-a5676a9e-ab0c-4d06-a660-2aabd862c2ef None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81408) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:08:49.797091 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 173/341] 10.222.0.27 () {58 vars in 1024 bytes} [Thu Oct 13 04:08:49 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 177 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 13 04:08:49.803055 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-670e9c1d-108f-4af9-8c72-d2824ee41aeb None None] Authenticating user token {{(pid=81409) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:08:49.804673 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 169/342] 10.222.0.27 () {58 vars in 1024 bytes} [Thu Oct 13 04:08:49 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 177 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Oct 13 04:08:49.947476 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-670e9c1d-108f-4af9-8c72-d2824ee41aeb None None] Validating token access rules against request {{(pid=81409) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:08:49.955179 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG dbcounter [-] [81409] Writing DB stats keystone:SELECT=100 {{(pid=81409) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:08:50.004329 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-29b52880-668b-4d7a-93e5-29e1371516f8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81409) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:08:50.004971 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29b52880-668b-4d7a-93e5-29e1371516f8 None admin] REQUEST_METHOD: `GET` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:08:50.005424 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29b52880-668b-4d7a-93e5-29e1371516f8 None admin] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:08:50.005704 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29b52880-668b-4d7a-93e5-29e1371516f8 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:08:50.021196 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-29b52880-668b-4d7a-93e5-29e1371516f8 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:08:50.022730 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-29b52880-668b-4d7a-93e5-29e1371516f8 None admin] RBAC: Authorization granted {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:08:50.047336 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application [None req-29b52880-668b-4d7a-93e5-29e1371516f8 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 13 04:08:50.047336 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 13 04:08:50.047336 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 13 04:08:50.047336 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 13 04:08:50.047336 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 13 04:08:50.047336 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 13 04:08:50.047336 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 13 04:08:50.047336 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 13 04:08:50.047336 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 13 04:08:50.047336 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 13 04:08:50.047336 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 13 04:08:50.047336 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 13 04:08:50.047336 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 13 04:08:50.047336 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 13 04:08:50.047336 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 13 04:08:50.047336 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 13 04:08:50.048383 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 13 04:08:50.048383 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 13 04:08:50.048383 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 13 04:08:50.048383 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 13 04:08:50.048383 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 13 04:08:50.048383 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application return self.get_or_create( Oct 13 04:08:50.048383 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 13 04:08:50.048383 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application with Lock( Oct 13 04:08:50.048383 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 13 04:08:50.048383 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application return self._enter() Oct 13 04:08:50.048383 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 13 04:08:50.048383 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 13 04:08:50.048383 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 13 04:08:50.048383 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application return self.creator() Oct 13 04:08:50.048383 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 13 04:08:50.048383 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application created_value = creator( Oct 13 04:08:50.048383 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 13 04:08:50.049260 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 13 04:08:50.049260 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 13 04:08:50.049260 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application Oct 13 04:08:50.049260 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 174/343] 10.222.0.27 () {56 vars in 1108 bytes} [Thu Oct 13 04:08:49 2022] GET /identity/v3/domains/Default => generated 87 bytes in 251 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 13 04:08:50.056067 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a5676a9e-ab0c-4d06-a660-2aabd862c2ef None None] Authenticating user token {{(pid=81408) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:08:50.066899 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG dbcounter [-] [81408] Writing DB stats keystone:SELECT=100 {{(pid=81408) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:08:50.116909 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-29b52880-668b-4d7a-93e5-29e1371516f8 None admin] Authenticating user token {{(pid=81409) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:08:50.181282 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a5676a9e-ab0c-4d06-a660-2aabd862c2ef None None] Validating token access rules against request {{(pid=81408) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:08:50.218655 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-92c2eb50-fbcf-4526-9ae6-821f68b8e0b7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81408) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:08:50.219480 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92c2eb50-fbcf-4526-9ae6-821f68b8e0b7 None admin] REQUEST_METHOD: `GET` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:08:50.219921 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92c2eb50-fbcf-4526-9ae6-821f68b8e0b7 None admin] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:08:50.220185 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92c2eb50-fbcf-4526-9ae6-821f68b8e0b7 None admin] PATH_INFO: `/v3/domains` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:08:50.220826 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-92c2eb50-fbcf-4526-9ae6-821f68b8e0b7 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81408) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 13 04:08:50.221126 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-92c2eb50-fbcf-4526-9ae6-821f68b8e0b7 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:08:50.222512 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-92c2eb50-fbcf-4526-9ae6-821f68b8e0b7 None admin] RBAC: Authorization granted {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:08:50.229770 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-29b52880-668b-4d7a-93e5-29e1371516f8 None admin] Validating token access rules against request {{(pid=81409) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:08:50.236016 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 170/344] 10.222.0.27 () {56 vars in 1109 bytes} [Thu Oct 13 04:08:50 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 181 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 13 04:08:50.239082 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-92c2eb50-fbcf-4526-9ae6-821f68b8e0b7 None admin] Authenticating user token {{(pid=81408) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:08:50.267082 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9ecfd579-763b-4e07-b163-43724181fc47 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81409) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:08:50.267989 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ecfd579-763b-4e07-b163-43724181fc47 None admin] REQUEST_METHOD: `GET` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:08:50.268449 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ecfd579-763b-4e07-b163-43724181fc47 None admin] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:08:50.268708 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ecfd579-763b-4e07-b163-43724181fc47 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:08:50.279297 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9ecfd579-763b-4e07-b163-43724181fc47 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:08:50.280731 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9ecfd579-763b-4e07-b163-43724181fc47 None admin] RBAC: Authorization granted {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:08:50.295591 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application [None req-9ecfd579-763b-4e07-b163-43724181fc47 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 13 04:08:50.295591 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 13 04:08:50.295591 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 13 04:08:50.295591 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 13 04:08:50.295591 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 13 04:08:50.295591 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 13 04:08:50.295591 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 13 04:08:50.295591 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 13 04:08:50.295591 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 13 04:08:50.295591 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 13 04:08:50.295591 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 13 04:08:50.295591 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 13 04:08:50.295591 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 13 04:08:50.295591 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 13 04:08:50.295591 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 13 04:08:50.295591 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 13 04:08:50.296482 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 13 04:08:50.296482 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 13 04:08:50.296482 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 13 04:08:50.296482 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 13 04:08:50.296482 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 13 04:08:50.296482 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application return self.get_or_create( Oct 13 04:08:50.296482 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 13 04:08:50.296482 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application with Lock( Oct 13 04:08:50.296482 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 13 04:08:50.296482 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application return self._enter() Oct 13 04:08:50.296482 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 13 04:08:50.296482 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 13 04:08:50.296482 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 13 04:08:50.296482 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application return self.creator() Oct 13 04:08:50.296482 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 13 04:08:50.296482 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application created_value = creator( Oct 13 04:08:50.296482 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 13 04:08:50.299117 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 13 04:08:50.299117 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 13 04:08:50.299117 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application Oct 13 04:08:50.299117 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 175/345] 10.222.0.27 () {56 vars in 1108 bytes} [Thu Oct 13 04:08:50 2022] GET /identity/v3/domains/Default => generated 87 bytes in 180 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Oct 13 04:08:50.299117 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0eecbb48-0705-48d3-bc2c-5a7b1b8fa68c None None] REQUEST_METHOD: `GET` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:08:50.299117 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0eecbb48-0705-48d3-bc2c-5a7b1b8fa68c None None] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:08:50.299117 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0eecbb48-0705-48d3-bc2c-5a7b1b8fa68c None None] PATH_INFO: `/` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:08:50.303578 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 176/346] 10.222.0.27 () {54 vars in 927 bytes} [Thu Oct 13 04:08:50 2022] GET /identity => generated 270 bytes in 6 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 13 04:08:50.304259 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0eecbb48-0705-48d3-bc2c-5a7b1b8fa68c None None] Authenticating user token {{(pid=81409) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:08:50.315639 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG dbcounter [-] [81408] Writing DB stats keystone:SELECT=100 {{(pid=81408) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:08:50.316176 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG dbcounter [-] [81409] Writing DB stats keystone:SELECT=100 {{(pid=81409) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:08:50.355588 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-92c2eb50-fbcf-4526-9ae6-821f68b8e0b7 None admin] Validating token access rules against request {{(pid=81408) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:08:50.391446 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5fc0d50f-9ab6-402c-bdde-0f20a8315a9b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81408) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:08:50.392126 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5fc0d50f-9ab6-402c-bdde-0f20a8315a9b None admin] REQUEST_METHOD: `GET` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:08:50.392400 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5fc0d50f-9ab6-402c-bdde-0f20a8315a9b None admin] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:08:50.392665 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5fc0d50f-9ab6-402c-bdde-0f20a8315a9b None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:08:50.404624 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5fc0d50f-9ab6-402c-bdde-0f20a8315a9b None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:08:50.408773 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5fc0d50f-9ab6-402c-bdde-0f20a8315a9b None admin] RBAC: Authorization granted {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:08:50.417709 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0eecbb48-0705-48d3-bc2c-5a7b1b8fa68c None None] Validating token access rules against request {{(pid=81409) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:08:50.424896 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application [None req-5fc0d50f-9ab6-402c-bdde-0f20a8315a9b None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 13 04:08:50.424896 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 13 04:08:50.424896 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 13 04:08:50.424896 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 13 04:08:50.424896 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 13 04:08:50.424896 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 13 04:08:50.424896 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 13 04:08:50.424896 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 13 04:08:50.424896 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 13 04:08:50.424896 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 13 04:08:50.424896 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 13 04:08:50.424896 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 13 04:08:50.424896 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 13 04:08:50.424896 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 13 04:08:50.424896 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 13 04:08:50.424896 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 13 04:08:50.425990 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 13 04:08:50.425990 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 13 04:08:50.425990 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 13 04:08:50.425990 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 13 04:08:50.425990 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 13 04:08:50.425990 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application return self.get_or_create( Oct 13 04:08:50.425990 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 13 04:08:50.425990 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application with Lock( Oct 13 04:08:50.425990 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 13 04:08:50.425990 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application return self._enter() Oct 13 04:08:50.425990 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 13 04:08:50.425990 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 13 04:08:50.425990 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 13 04:08:50.425990 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application return self.creator() Oct 13 04:08:50.425990 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 13 04:08:50.425990 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application created_value = creator( Oct 13 04:08:50.425990 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 13 04:08:50.426856 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 13 04:08:50.426856 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 13 04:08:50.426856 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application Oct 13 04:08:50.426856 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 171/347] 10.222.0.27 () {56 vars in 1108 bytes} [Thu Oct 13 04:08:50 2022] GET /identity/v3/domains/Default => generated 87 bytes in 189 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 13 04:08:50.426856 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5fc0d50f-9ab6-402c-bdde-0f20a8315a9b None admin] Authenticating user token {{(pid=81408) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:08:50.458239 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-510fd693-c8da-4e6a-84c5-b39cfc8803b2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81409) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:08:50.458904 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-510fd693-c8da-4e6a-84c5-b39cfc8803b2 None admin] REQUEST_METHOD: `POST` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:08:50.459183 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-510fd693-c8da-4e6a-84c5-b39cfc8803b2 None admin] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:08:50.459446 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-510fd693-c8da-4e6a-84c5-b39cfc8803b2 None admin] PATH_INFO: `/v3/users` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:08:50.460078 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-510fd693-c8da-4e6a-84c5-b39cfc8803b2 None admin] RBAC: Authorizing `identity:create_user()` {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:08:50.463146 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-510fd693-c8da-4e6a-84c5-b39cfc8803b2 None admin] RBAC: Authorization granted {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:08:50.484718 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 177/348] 10.222.0.27 () {60 vars in 1132 bytes} [Thu Oct 13 04:08:50 2022] POST /identity/v3/users => generated 246 bytes in 181 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 13 04:08:50.491781 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13800c0d-940e-4cfc-a625-926cd67a13b2 None None] REQUEST_METHOD: `POST` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:08:50.492086 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13800c0d-940e-4cfc-a625-926cd67a13b2 None None] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:08:50.492336 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13800c0d-940e-4cfc-a625-926cd67a13b2 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:08:50.516876 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.auth.core [None req-13800c0d-940e-4cfc-a625-926cd67a13b2 None None] MFA Rules not processed for user `0e2d979563d6421995451a0e86bf3022`. Rule list: `[]` (Enabled: `True`). {{(pid=81409) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 13 04:08:50.518252 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.fernet_utils [None req-13800c0d-940e-4cfc-a625-926cd67a13b2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81409) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:08:50.553072 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5fc0d50f-9ab6-402c-bdde-0f20a8315a9b None admin] Validating token access rules against request {{(pid=81408) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:08:50.580203 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG dbcounter [-] [81408] Writing DB stats keystone:SELECT=100 {{(pid=81408) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:08:50.600004 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-14c15c7b-1af3-49e3-875b-0e9f221381ee None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81408) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:08:50.601121 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14c15c7b-1af3-49e3-875b-0e9f221381ee None admin] REQUEST_METHOD: `GET` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:08:50.601624 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14c15c7b-1af3-49e3-875b-0e9f221381ee None admin] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:08:50.602522 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG dbcounter [-] [81409] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81409) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:08:50.603132 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14c15c7b-1af3-49e3-875b-0e9f221381ee None admin] PATH_INFO: `/v3/domains` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:08:50.604834 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-14c15c7b-1af3-49e3-875b-0e9f221381ee None admin] RBAC: Adding query filter params (name=Default) {{(pid=81408) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 13 04:08:50.605433 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-14c15c7b-1af3-49e3-875b-0e9f221381ee None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:08:50.607391 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-14c15c7b-1af3-49e3-875b-0e9f221381ee None admin] RBAC: Authorization granted {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:08:50.617810 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 178/349] 10.222.0.27 () {58 vars in 1024 bytes} [Thu Oct 13 04:08:50 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 129 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 13 04:08:50.617810 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-13800c0d-940e-4cfc-a625-926cd67a13b2 None None] Authenticating user token {{(pid=81409) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:08:50.635444 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 172/350] 10.222.0.27 () {56 vars in 1109 bytes} [Thu Oct 13 04:08:50 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 209 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 13 04:08:50.640630 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-14c15c7b-1af3-49e3-875b-0e9f221381ee None admin] Authenticating user token {{(pid=81408) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:08:50.737681 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-13800c0d-940e-4cfc-a625-926cd67a13b2 None None] Validating token access rules against request {{(pid=81409) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:08:50.766101 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-14c15c7b-1af3-49e3-875b-0e9f221381ee None admin] Validating token access rules against request {{(pid=81408) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:08:50.775994 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6a23f31b-3115-436b-9a01-e6070ba02a49 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81409) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:08:50.776613 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a23f31b-3115-436b-9a01-e6070ba02a49 None admin] REQUEST_METHOD: `GET` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:08:50.776987 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a23f31b-3115-436b-9a01-e6070ba02a49 None admin] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:08:50.777268 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a23f31b-3115-436b-9a01-e6070ba02a49 None admin] PATH_INFO: `/v3/domains` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:08:50.777977 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6a23f31b-3115-436b-9a01-e6070ba02a49 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81409) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 13 04:08:50.778311 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6a23f31b-3115-436b-9a01-e6070ba02a49 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:08:50.779791 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6a23f31b-3115-436b-9a01-e6070ba02a49 None admin] RBAC: Authorization granted {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:08:50.799468 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 179/351] 10.222.0.27 () {56 vars in 1109 bytes} [Thu Oct 13 04:08:50 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 184 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 13 04:08:50.804341 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-df302985-dd5d-493f-9622-f46e77867fdd None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81408) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:08:50.804786 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6a23f31b-3115-436b-9a01-e6070ba02a49 None admin] Authenticating user token {{(pid=81409) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:08:50.804964 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df302985-dd5d-493f-9622-f46e77867fdd None admin] REQUEST_METHOD: `POST` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:08:50.805346 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df302985-dd5d-493f-9622-f46e77867fdd None admin] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:08:50.805681 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df302985-dd5d-493f-9622-f46e77867fdd None admin] PATH_INFO: `/v3/users` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:08:50.806510 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-df302985-dd5d-493f-9622-f46e77867fdd None admin] RBAC: Authorizing `identity:create_user()` {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:08:50.808038 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-df302985-dd5d-493f-9622-f46e77867fdd None admin] RBAC: Authorization granted {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:08:50.838300 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 173/352] 10.222.0.27 () {60 vars in 1132 bytes} [Thu Oct 13 04:08:50 2022] POST /identity/v3/users => generated 248 bytes in 195 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 13 04:08:50.898650 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG dbcounter [-] [81409] Writing DB stats keystone:SELECT=100 {{(pid=81409) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:08:50.943414 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-df302985-dd5d-493f-9622-f46e77867fdd None admin] Authenticating user token {{(pid=81408) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:08:50.945089 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6a23f31b-3115-436b-9a01-e6070ba02a49 None admin] Validating token access rules against request {{(pid=81409) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:08:51.018710 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1d262422-26db-4176-a630-20180da9110e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81409) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:08:51.019720 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d262422-26db-4176-a630-20180da9110e None admin] REQUEST_METHOD: `POST` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:08:51.020187 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d262422-26db-4176-a630-20180da9110e None admin] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:08:51.020660 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d262422-26db-4176-a630-20180da9110e None admin] PATH_INFO: `/v3/users` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:08:51.021524 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1d262422-26db-4176-a630-20180da9110e None admin] RBAC: Authorizing `identity:create_user()` {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:08:51.022480 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG dbcounter [-] [81408] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81408) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:08:51.023464 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1d262422-26db-4176-a630-20180da9110e None admin] RBAC: Authorization granted {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:08:51.050361 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 180/353] 10.222.0.27 () {60 vars in 1132 bytes} [Thu Oct 13 04:08:50 2022] POST /identity/v3/users => generated 248 bytes in 246 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 13 04:08:51.051875 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd076abb-04af-4a7d-a24e-15dab7f9e317 None None] REQUEST_METHOD: `GET` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:08:51.052154 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd076abb-04af-4a7d-a24e-15dab7f9e317 None None] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:08:51.053238 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd076abb-04af-4a7d-a24e-15dab7f9e317 None None] PATH_INFO: `/` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:08:51.054673 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 181/354] 10.222.0.27 () {54 vars in 927 bytes} [Thu Oct 13 04:08:51 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 13 04:08:51.061102 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e03034c-36b1-4c67-a048-0f97ab218034 None None] REQUEST_METHOD: `POST` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:08:51.061388 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e03034c-36b1-4c67-a048-0f97ab218034 None None] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:08:51.061708 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e03034c-36b1-4c67-a048-0f97ab218034 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:08:51.082033 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.auth.core [None req-3e03034c-36b1-4c67-a048-0f97ab218034 None None] MFA Rules not processed for user `0e2d979563d6421995451a0e86bf3022`. Rule list: `[]` (Enabled: `True`). {{(pid=81409) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 13 04:08:51.083341 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.fernet_utils [None req-3e03034c-36b1-4c67-a048-0f97ab218034 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81409) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:08:51.089115 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-df302985-dd5d-493f-9622-f46e77867fdd None admin] Validating token access rules against request {{(pid=81408) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:08:51.131541 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a7ecfeb6-844b-44b7-b917-eb3acd75fc45 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81408) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:08:51.132296 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7ecfeb6-844b-44b7-b917-eb3acd75fc45 None admin] REQUEST_METHOD: `GET` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:08:51.132588 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7ecfeb6-844b-44b7-b917-eb3acd75fc45 None admin] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:08:51.132861 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7ecfeb6-844b-44b7-b917-eb3acd75fc45 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:08:51.147071 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a7ecfeb6-844b-44b7-b917-eb3acd75fc45 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:08:51.148670 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a7ecfeb6-844b-44b7-b917-eb3acd75fc45 None admin] RBAC: Authorization granted {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:08:51.160922 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 182/355] 10.222.0.27 () {58 vars in 1024 bytes} [Thu Oct 13 04:08:51 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 101 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 13 04:08:51.165686 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application [None req-a7ecfeb6-844b-44b7-b917-eb3acd75fc45 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 13 04:08:51.165686 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 13 04:08:51.165686 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 13 04:08:51.165686 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 13 04:08:51.165686 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 13 04:08:51.165686 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 13 04:08:51.165686 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 13 04:08:51.165686 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 13 04:08:51.165686 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 13 04:08:51.165686 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 13 04:08:51.165686 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 13 04:08:51.165686 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 13 04:08:51.165686 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 13 04:08:51.165686 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 13 04:08:51.165686 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 13 04:08:51.165686 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 13 04:08:51.166644 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 13 04:08:51.166644 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 13 04:08:51.166644 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 13 04:08:51.166644 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 13 04:08:51.166644 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 13 04:08:51.166644 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application return self.get_or_create( Oct 13 04:08:51.166644 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 13 04:08:51.166644 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application with Lock( Oct 13 04:08:51.166644 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 13 04:08:51.166644 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application return self._enter() Oct 13 04:08:51.166644 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 13 04:08:51.166644 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 13 04:08:51.166644 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 13 04:08:51.166644 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application return self.creator() Oct 13 04:08:51.166644 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 13 04:08:51.166644 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application created_value = creator( Oct 13 04:08:51.166644 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 13 04:08:51.167482 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 13 04:08:51.167482 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 13 04:08:51.167482 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application Oct 13 04:08:51.167482 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 174/356] 10.222.0.27 () {56 vars in 1108 bytes} [Thu Oct 13 04:08:50 2022] GET /identity/v3/domains/Default => generated 87 bytes in 224 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Oct 13 04:08:51.173208 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3e03034c-36b1-4c67-a048-0f97ab218034 None None] Authenticating user token {{(pid=81409) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:08:51.222465 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG dbcounter [-] [81409] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81409) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:08:51.285474 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3e03034c-36b1-4c67-a048-0f97ab218034 None None] Validating token access rules against request {{(pid=81409) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:08:51.322352 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f5501b8d-4abe-4b99-b29d-6456878ff4e4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81409) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:08:51.322965 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5501b8d-4abe-4b99-b29d-6456878ff4e4 None admin] REQUEST_METHOD: `GET` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:08:51.323240 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5501b8d-4abe-4b99-b29d-6456878ff4e4 None admin] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:08:51.323506 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5501b8d-4abe-4b99-b29d-6456878ff4e4 None admin] PATH_INFO: `/v3/domains` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:08:51.324153 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f5501b8d-4abe-4b99-b29d-6456878ff4e4 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81409) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 13 04:08:51.324450 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f5501b8d-4abe-4b99-b29d-6456878ff4e4 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:08:51.325815 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f5501b8d-4abe-4b99-b29d-6456878ff4e4 None admin] RBAC: Authorization granted {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:08:51.338407 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 183/357] 10.222.0.27 () {56 vars in 1109 bytes} [Thu Oct 13 04:08:51 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 166 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 13 04:08:51.347286 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a7ecfeb6-844b-44b7-b917-eb3acd75fc45 None admin] Authenticating user token {{(pid=81408) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:08:51.448344 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG dbcounter [-] [81408] Writing DB stats keystone:SELECT=100 {{(pid=81408) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:08:51.463504 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a7ecfeb6-844b-44b7-b917-eb3acd75fc45 None admin] Validating token access rules against request {{(pid=81408) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:08:51.476914 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f5501b8d-4abe-4b99-b29d-6456878ff4e4 None admin] Authenticating user token {{(pid=81409) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:08:51.521428 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0b7dcae6-9287-452d-806e-f9388330b7f9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81408) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:08:51.522054 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b7dcae6-9287-452d-806e-f9388330b7f9 None admin] REQUEST_METHOD: `POST` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:08:51.522356 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b7dcae6-9287-452d-806e-f9388330b7f9 None admin] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:08:51.522624 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b7dcae6-9287-452d-806e-f9388330b7f9 None admin] PATH_INFO: `/v3/users` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:08:51.523278 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0b7dcae6-9287-452d-806e-f9388330b7f9 None admin] RBAC: Authorizing `identity:create_user()` {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:08:51.525381 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0b7dcae6-9287-452d-806e-f9388330b7f9 None admin] RBAC: Authorization granted {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:08:51.562328 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 175/358] 10.222.0.27 () {60 vars in 1132 bytes} [Thu Oct 13 04:08:51 2022] POST /identity/v3/users => generated 249 bytes in 215 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 13 04:08:51.597809 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG dbcounter [-] [81409] Writing DB stats keystone:SELECT=100 {{(pid=81409) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:08:51.601566 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f5501b8d-4abe-4b99-b29d-6456878ff4e4 None admin] Validating token access rules against request {{(pid=81409) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:08:51.641107 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0b7d5c5f-721c-4a33-a651-3c8356bb9737 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81409) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:08:51.641779 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b7d5c5f-721c-4a33-a651-3c8356bb9737 None admin] REQUEST_METHOD: `POST` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:08:51.642057 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b7d5c5f-721c-4a33-a651-3c8356bb9737 None admin] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:08:51.642358 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b7d5c5f-721c-4a33-a651-3c8356bb9737 None admin] PATH_INFO: `/v3/roles` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:08:51.643060 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0b7d5c5f-721c-4a33-a651-3c8356bb9737 None admin] RBAC: Authorizing `identity:create_role()` {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:08:51.644463 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0b7d5c5f-721c-4a33-a651-3c8356bb9737 None admin] RBAC: Authorization granted {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:08:51.653840 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.sql.core [None req-0b7d5c5f-721c-4a33-a651-3c8356bb9737 None admin] Conflict role: (pymysql.err.IntegrityError) (1062, "Duplicate entry 'anotherrole-<>' for key 'role.ixu_role_name_domain_id'") Oct 13 04:08:51.653840 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [SQL: INSERT INTO `role` (id, name, description, extra) VALUES (%(id)s, %(name)s, %(description)s, %(extra)s)] Oct 13 04:08:51.653840 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [parameters: {'id': '3607eea1e5ac4407889757064cf9b4b7', 'name': 'anotherrole', 'description': None, 'extra': '{}'}] Oct 13 04:08:51.653840 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: (Background on this error at: https://sqlalche.me/e/14/gkpj) {{(pid=81409) wrapper /opt/stack/keystone/keystone/common/sql/core.py:533}} Oct 13 04:08:51.672019 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application [None req-0b7d5c5f-721c-4a33-a651-3c8356bb9737 None admin] Conflict occurred attempting to store role - Duplicate entry found with name anotherrole.: keystone.exception.Conflict: Conflict occurred attempting to store role - Duplicate entry found with name anotherrole. Oct 13 04:08:51.672019 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 13 04:08:51.672019 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1900, in _execute_context Oct 13 04:08:51.672019 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application self.dialect.do_execute( Oct 13 04:08:51.672019 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/default.py", line 732, in do_execute Oct 13 04:08:51.672019 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application cursor.execute(statement, parameters) Oct 13 04:08:51.672019 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/cursors.py", line 148, in execute Oct 13 04:08:51.672019 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application result = self._query(query) Oct 13 04:08:51.672019 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/cursors.py", line 310, in _query Oct 13 04:08:51.672019 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application conn.query(q) Oct 13 04:08:51.672019 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 548, in query Oct 13 04:08:51.672019 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application self._affected_rows = self._read_query_result(unbuffered=unbuffered) Oct 13 04:08:51.672019 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 775, in _read_query_result Oct 13 04:08:51.672019 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application result.read() Oct 13 04:08:51.672019 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 1156, in read Oct 13 04:08:51.672942 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application first_packet = self.connection._read_packet() Oct 13 04:08:51.672942 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 725, in _read_packet Oct 13 04:08:51.672942 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application packet.raise_for_error() Oct 13 04:08:51.672942 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/protocol.py", line 221, in raise_for_error Oct 13 04:08:51.672942 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application err.raise_mysql_exception(self._data) Oct 13 04:08:51.672942 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/err.py", line 143, in raise_mysql_exception Oct 13 04:08:51.672942 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application raise errorclass(errno, errval) Oct 13 04:08:51.672942 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application pymysql.err.IntegrityError: (1062, "Duplicate entry 'anotherrole-<>' for key 'role.ixu_role_name_domain_id'") Oct 13 04:08:51.672942 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application Oct 13 04:08:51.672942 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application The above exception was the direct cause of the following exception: Oct 13 04:08:51.672942 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application Oct 13 04:08:51.672942 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 13 04:08:51.672942 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/sql/core.py", line 528, in wrapper Oct 13 04:08:51.672942 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application return method(*args, **kwargs) Oct 13 04:08:51.672942 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 33, in create_role Oct 13 04:08:51.672942 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application return ref.to_dict() Oct 13 04:08:51.672942 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ Oct 13 04:08:51.672942 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application next(self.gen) Oct 13 04:08:51.673859 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 1060, in _transaction_scope Oct 13 04:08:51.673859 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application yield resource Oct 13 04:08:51.673859 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ Oct 13 04:08:51.673859 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application next(self.gen) Oct 13 04:08:51.673859 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 662, in _session Oct 13 04:08:51.673859 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application self.session.rollback() Oct 13 04:08:51.673859 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_utils/excutils.py", line 227, in __exit__ Oct 13 04:08:51.673859 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application self.force_reraise() Oct 13 04:08:51.673859 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_utils/excutils.py", line 200, in force_reraise Oct 13 04:08:51.673859 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application raise self.value Oct 13 04:08:51.673859 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 659, in _session Oct 13 04:08:51.673859 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application self._end_session_transaction(self.session) Oct 13 04:08:51.673859 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 687, in _end_session_transaction Oct 13 04:08:51.673859 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application session.commit() Oct 13 04:08:51.673859 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 1451, in commit Oct 13 04:08:51.673859 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application self._transaction.commit(_to_root=self.future) Oct 13 04:08:51.673859 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 829, in commit Oct 13 04:08:51.674729 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application self._prepare_impl() Oct 13 04:08:51.674729 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 808, in _prepare_impl Oct 13 04:08:51.674729 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application self.session.flush() Oct 13 04:08:51.674729 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 3383, in flush Oct 13 04:08:51.674729 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application self._flush(objects) Oct 13 04:08:51.674729 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 3523, in _flush Oct 13 04:08:51.674729 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application transaction.rollback(_capture_exception=True) Oct 13 04:08:51.674729 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/util/langhelpers.py", line 70, in __exit__ Oct 13 04:08:51.674729 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application compat.raise_( Oct 13 04:08:51.674729 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/util/compat.py", line 208, in raise_ Oct 13 04:08:51.674729 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application raise exception Oct 13 04:08:51.674729 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 3483, in _flush Oct 13 04:08:51.674729 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application flush_context.execute() Oct 13 04:08:51.674729 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/unitofwork.py", line 456, in execute Oct 13 04:08:51.674729 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application rec.execute(self) Oct 13 04:08:51.674729 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/unitofwork.py", line 630, in execute Oct 13 04:08:51.674729 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application util.preloaded.orm_persistence.save_obj( Oct 13 04:08:51.675611 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/persistence.py", line 245, in save_obj Oct 13 04:08:51.675611 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application _emit_insert_statements( Oct 13 04:08:51.675611 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/persistence.py", line 1097, in _emit_insert_statements Oct 13 04:08:51.675611 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application c = connection._execute_20( Oct 13 04:08:51.675611 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1705, in _execute_20 Oct 13 04:08:51.675611 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application return meth(self, args_10style, kwargs_10style, execution_options) Oct 13 04:08:51.675611 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/sql/elements.py", line 333, in _execute_on_connection Oct 13 04:08:51.675611 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application return connection._execute_clauseelement( Oct 13 04:08:51.675611 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1572, in _execute_clauseelement Oct 13 04:08:51.675611 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application ret = self._execute_context( Oct 13 04:08:51.675611 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1943, in _execute_context Oct 13 04:08:51.675611 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application self._handle_dbapi_exception( Oct 13 04:08:51.675611 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 2122, in _handle_dbapi_exception Oct 13 04:08:51.675611 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application util.raise_(newraise, with_traceback=exc_info[2], from_=e) Oct 13 04:08:51.675611 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/util/compat.py", line 208, in raise_ Oct 13 04:08:51.675611 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application raise exception Oct 13 04:08:51.676410 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1900, in _execute_context Oct 13 04:08:51.676410 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application self.dialect.do_execute( Oct 13 04:08:51.676410 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/default.py", line 732, in do_execute Oct 13 04:08:51.676410 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application cursor.execute(statement, parameters) Oct 13 04:08:51.676410 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/cursors.py", line 148, in execute Oct 13 04:08:51.676410 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application result = self._query(query) Oct 13 04:08:51.676410 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/cursors.py", line 310, in _query Oct 13 04:08:51.676410 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application conn.query(q) Oct 13 04:08:51.676410 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 548, in query Oct 13 04:08:51.676410 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application self._affected_rows = self._read_query_result(unbuffered=unbuffered) Oct 13 04:08:51.676410 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 775, in _read_query_result Oct 13 04:08:51.676410 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application result.read() Oct 13 04:08:51.676410 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 1156, in read Oct 13 04:08:51.676410 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application first_packet = self.connection._read_packet() Oct 13 04:08:51.676410 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 725, in _read_packet Oct 13 04:08:51.676410 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application packet.raise_for_error() Oct 13 04:08:51.677266 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/protocol.py", line 221, in raise_for_error Oct 13 04:08:51.677266 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application err.raise_mysql_exception(self._data) Oct 13 04:08:51.677266 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/err.py", line 143, in raise_mysql_exception Oct 13 04:08:51.677266 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application raise errorclass(errno, errval) Oct 13 04:08:51.677266 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application oslo_db.exception.DBDuplicateEntry: (pymysql.err.IntegrityError) (1062, "Duplicate entry 'anotherrole-<>' for key 'role.ixu_role_name_domain_id'") Oct 13 04:08:51.677266 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application [SQL: INSERT INTO `role` (id, name, description, extra) VALUES (%(id)s, %(name)s, %(description)s, %(extra)s)] Oct 13 04:08:51.677266 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application [parameters: {'id': '3607eea1e5ac4407889757064cf9b4b7', 'name': 'anotherrole', 'description': None, 'extra': '{}'}] Oct 13 04:08:51.677266 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application (Background on this error at: https://sqlalche.me/e/14/gkpj) Oct 13 04:08:51.677266 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application Oct 13 04:08:51.677266 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Oct 13 04:08:51.677266 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application Oct 13 04:08:51.677266 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 13 04:08:51.677266 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 13 04:08:51.677266 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 13 04:08:51.677266 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 13 04:08:51.677266 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 13 04:08:51.678066 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 13 04:08:51.678066 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 13 04:08:51.678066 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 13 04:08:51.678066 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 13 04:08:51.678066 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 13 04:08:51.678066 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 13 04:08:51.678066 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 115, in post Oct 13 04:08:51.678066 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application ref = PROVIDERS.role_api.create_role( Oct 13 04:08:51.678066 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 13 04:08:51.678066 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 13 04:08:51.678066 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1298, in create_role Oct 13 04:08:51.678066 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application ret = self.driver.create_role(role_id, role) Oct 13 04:08:51.678066 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/sql/core.py", line 563, in wrapper Oct 13 04:08:51.678066 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application raise exception.Conflict(type=conflict_type, Oct 13 04:08:51.678066 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application keystone.exception.Conflict: Conflict occurred attempting to store role - Duplicate entry found with name anotherrole. Oct 13 04:08:51.678066 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application Oct 13 04:08:51.681150 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 184/359] 10.222.0.27 () {60 vars in 1131 bytes} [Thu Oct 13 04:08:51 2022] POST /identity/v3/roles => generated 144 bytes in 197 msecs (HTTP/1.1 409) 5 headers in 183 bytes (2 switches on core 0) Oct 13 04:08:51.686248 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0b7dcae6-9287-452d-806e-f9388330b7f9 None admin] Authenticating user token {{(pid=81408) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:08:51.854057 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0b7dcae6-9287-452d-806e-f9388330b7f9 None admin] Validating token access rules against request {{(pid=81408) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:08:51.889058 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-78681471-c82d-4bfe-8414-8b7977260364 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81408) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:08:51.889695 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78681471-c82d-4bfe-8414-8b7977260364 None admin] REQUEST_METHOD: `GET` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:08:51.889969 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78681471-c82d-4bfe-8414-8b7977260364 None admin] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:08:51.890261 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78681471-c82d-4bfe-8414-8b7977260364 None admin] PATH_INFO: `/v3/roles/anotherrole` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:08:51.901236 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-78681471-c82d-4bfe-8414-8b7977260364 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 13 04:08:51.906587 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application [None req-78681471-c82d-4bfe-8414-8b7977260364 None admin] Could not find role: anotherrole.: keystone.exception.RoleNotFound: Could not find role: anotherrole. Oct 13 04:08:51.906587 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 13 04:08:51.906587 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 13 04:08:51.906587 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 13 04:08:51.906587 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 13 04:08:51.906587 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 13 04:08:51.906587 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 13 04:08:51.906587 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 13 04:08:51.906587 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 13 04:08:51.906587 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 13 04:08:51.906587 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 13 04:08:51.906587 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 13 04:08:51.906587 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 13 04:08:51.906587 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 13 04:08:51.906587 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 13 04:08:51.906587 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 13 04:08:51.908331 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 13 04:08:51.908331 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 13 04:08:51.908331 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 13 04:08:51.908331 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 13 04:08:51.908331 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 13 04:08:51.908331 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 13 04:08:51.908331 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 13 04:08:51.908331 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 13 04:08:51.908331 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 13 04:08:51.908331 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application return self.get_or_create( Oct 13 04:08:51.908331 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 13 04:08:51.908331 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application with Lock( Oct 13 04:08:51.908331 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 13 04:08:51.908331 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application return self._enter() Oct 13 04:08:51.908331 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 13 04:08:51.908331 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 13 04:08:51.909301 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 13 04:08:51.909301 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application return self.creator() Oct 13 04:08:51.909301 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 13 04:08:51.909301 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application created_value = creator( Oct 13 04:08:51.909301 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 13 04:08:51.909301 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 13 04:08:51.909301 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 13 04:08:51.909301 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 13 04:08:51.909301 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 13 04:08:51.909301 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 13 04:08:51.909301 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: anotherrole. Oct 13 04:08:51.909301 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application Oct 13 04:08:51.909301 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 176/360] 10.222.0.27 () {56 vars in 1114 bytes} [Thu Oct 13 04:08:51 2022] GET /identity/v3/roles/anotherrole => generated 89 bytes in 225 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 13 04:08:51.913594 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0b7d5c5f-721c-4a33-a651-3c8356bb9737 None admin] Authenticating user token {{(pid=81409) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:08:52.028031 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0b7d5c5f-721c-4a33-a651-3c8356bb9737 None admin] Validating token access rules against request {{(pid=81409) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:08:52.068748 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f399e993-5362-48d2-814f-e55aac522190 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81409) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:08:52.069363 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f399e993-5362-48d2-814f-e55aac522190 None admin] REQUEST_METHOD: `GET` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:08:52.069636 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f399e993-5362-48d2-814f-e55aac522190 None admin] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:08:52.069900 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f399e993-5362-48d2-814f-e55aac522190 None admin] PATH_INFO: `/v3/roles` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:08:52.070588 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f399e993-5362-48d2-814f-e55aac522190 None admin] RBAC: Adding query filter params (name=anotherrole) {{(pid=81409) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 13 04:08:52.070887 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f399e993-5362-48d2-814f-e55aac522190 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:08:52.072490 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f399e993-5362-48d2-814f-e55aac522190 None admin] RBAC: Authorization granted {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:08:52.081095 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 185/361] 10.222.0.27 () {56 vars in 1111 bytes} [Thu Oct 13 04:08:51 2022] GET /identity/v3/roles?name=anotherrole => generated 334 bytes in 168 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 13 04:08:53.196580 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53cac0c9-e6cb-4e4f-b82d-54998f28f962 None None] REQUEST_METHOD: `GET` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:08:53.197437 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53cac0c9-e6cb-4e4f-b82d-54998f28f962 None None] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:08:53.197437 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53cac0c9-e6cb-4e4f-b82d-54998f28f962 None None] PATH_INFO: `/` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:08:53.198597 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 177/362] 10.222.0.27 () {54 vars in 927 bytes} [Thu Oct 13 04:08:53 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 13 04:08:53.203909 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49d53061-89f4-4179-9743-ba3ce4fa542b None None] REQUEST_METHOD: `POST` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:08:53.204401 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49d53061-89f4-4179-9743-ba3ce4fa542b None None] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:08:53.204557 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49d53061-89f4-4179-9743-ba3ce4fa542b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:08:53.227877 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.auth.core [None req-49d53061-89f4-4179-9743-ba3ce4fa542b None None] MFA Rules not processed for user `0e2d979563d6421995451a0e86bf3022`. Rule list: `[]` (Enabled: `True`). {{(pid=81409) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 13 04:08:53.229208 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.fernet_utils [None req-49d53061-89f4-4179-9743-ba3ce4fa542b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81409) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:08:53.250535 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG dbcounter [-] [81409] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81409) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:08:53.315010 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 186/363] 10.222.0.27 () {58 vars in 1024 bytes} [Thu Oct 13 04:08:53 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 112 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 13 04:08:53.532270 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7d1b2a4-d75a-4d94-af36-ac054631cc45 None None] REQUEST_METHOD: `GET` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:08:53.532598 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7d1b2a4-d75a-4d94-af36-ac054631cc45 None None] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:08:53.532804 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7d1b2a4-d75a-4d94-af36-ac054631cc45 None None] PATH_INFO: `/` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:08:53.534645 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 178/364] 10.222.0.27 () {54 vars in 927 bytes} [Thu Oct 13 04:08:53 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 13 04:08:53.540600 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07fe624d-6599-40e6-a0e7-788a146217c6 None None] REQUEST_METHOD: `POST` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:08:53.541117 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07fe624d-6599-40e6-a0e7-788a146217c6 None None] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:08:53.541580 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07fe624d-6599-40e6-a0e7-788a146217c6 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:08:53.568556 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.auth.core [None req-07fe624d-6599-40e6-a0e7-788a146217c6 None None] MFA Rules not processed for user `0e2d979563d6421995451a0e86bf3022`. Rule list: `[]` (Enabled: `True`). {{(pid=81409) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 13 04:08:53.570220 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.fernet_utils [None req-07fe624d-6599-40e6-a0e7-788a146217c6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81409) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:08:53.648671 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 187/365] 10.222.0.27 () {58 vars in 1024 bytes} [Thu Oct 13 04:08:53 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 110 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 13 04:08:53.710927 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a7d1b2a4-d75a-4d94-af36-ac054631cc45 None None] Authenticating user token {{(pid=81408) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:08:53.716558 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG dbcounter [-] [81408] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81408) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:08:53.850546 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a7d1b2a4-d75a-4d94-af36-ac054631cc45 None None] Validating token access rules against request {{(pid=81408) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:08:53.888873 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d002511b-0271-42e6-ae4c-8fcc50d96746 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81408) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:08:53.889501 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d002511b-0271-42e6-ae4c-8fcc50d96746 None admin] REQUEST_METHOD: `GET` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:08:53.889780 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d002511b-0271-42e6-ae4c-8fcc50d96746 None admin] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:08:53.890051 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d002511b-0271-42e6-ae4c-8fcc50d96746 None admin] PATH_INFO: `/v3/roles/service` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:08:53.901096 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d002511b-0271-42e6-ae4c-8fcc50d96746 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 13 04:08:53.905981 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application [None req-d002511b-0271-42e6-ae4c-8fcc50d96746 None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Oct 13 04:08:53.905981 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 13 04:08:53.905981 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 13 04:08:53.905981 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 13 04:08:53.905981 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 13 04:08:53.905981 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 13 04:08:53.905981 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 13 04:08:53.905981 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 13 04:08:53.905981 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 13 04:08:53.905981 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 13 04:08:53.905981 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 13 04:08:53.905981 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 13 04:08:53.905981 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 13 04:08:53.905981 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 13 04:08:53.905981 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 13 04:08:53.905981 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 13 04:08:53.906863 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 13 04:08:53.906863 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 13 04:08:53.906863 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 13 04:08:53.906863 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 13 04:08:53.906863 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 13 04:08:53.906863 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 13 04:08:53.906863 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 13 04:08:53.906863 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 13 04:08:53.906863 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 13 04:08:53.906863 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application return self.get_or_create( Oct 13 04:08:53.906863 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 13 04:08:53.906863 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application with Lock( Oct 13 04:08:53.906863 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 13 04:08:53.906863 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application return self._enter() Oct 13 04:08:53.906863 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 13 04:08:53.906863 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 13 04:08:53.907654 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 13 04:08:53.907654 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application return self.creator() Oct 13 04:08:53.907654 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 13 04:08:53.907654 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application created_value = creator( Oct 13 04:08:53.907654 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 13 04:08:53.907654 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 13 04:08:53.907654 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 13 04:08:53.907654 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 13 04:08:53.907654 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 13 04:08:53.907654 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 13 04:08:53.907654 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Oct 13 04:08:53.907654 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application Oct 13 04:08:53.907654 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 179/366] 10.222.0.27 () {56 vars in 1102 bytes} [Thu Oct 13 04:08:53 2022] GET /identity/v3/roles/service => generated 85 bytes in 197 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 13 04:08:53.913193 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-07fe624d-6599-40e6-a0e7-788a146217c6 None None] Authenticating user token {{(pid=81409) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:08:53.974043 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e625cf49-27fc-4ffd-994a-0d0a5711418f None None] REQUEST_METHOD: `GET` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:08:53.974368 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e625cf49-27fc-4ffd-994a-0d0a5711418f None None] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:08:53.974930 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e625cf49-27fc-4ffd-994a-0d0a5711418f None None] PATH_INFO: `/` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:08:53.976546 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 180/367] 10.222.0.27 () {54 vars in 927 bytes} [Thu Oct 13 04:08:53 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 13 04:08:53.984618 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f25b5586-84e3-45f5-aace-772c62a36605 None None] REQUEST_METHOD: `POST` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:08:53.984898 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f25b5586-84e3-45f5-aace-772c62a36605 None None] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:08:53.985175 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f25b5586-84e3-45f5-aace-772c62a36605 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:08:54.015575 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.auth.core [None req-f25b5586-84e3-45f5-aace-772c62a36605 None None] MFA Rules not processed for user `0e2d979563d6421995451a0e86bf3022`. Rule list: `[]` (Enabled: `True`). {{(pid=81408) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 13 04:08:54.016785 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.fernet_utils [None req-f25b5586-84e3-45f5-aace-772c62a36605 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81408) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:08:54.040636 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG dbcounter [-] [81409] Writing DB stats keystone:SELECT=100 {{(pid=81409) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:08:54.066788 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-07fe624d-6599-40e6-a0e7-788a146217c6 None None] Validating token access rules against request {{(pid=81409) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:08:54.103519 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG dbcounter [-] [81408] Writing DB stats keystone:SELECT=100 {{(pid=81408) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:08:54.110012 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eafe698f-1fcd-45ca-9696-45d87af7e2d5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81409) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:08:54.110945 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eafe698f-1fcd-45ca-9696-45d87af7e2d5 None admin] REQUEST_METHOD: `GET` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:08:54.111539 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eafe698f-1fcd-45ca-9696-45d87af7e2d5 None admin] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:08:54.112108 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eafe698f-1fcd-45ca-9696-45d87af7e2d5 None admin] PATH_INFO: `/v3/roles` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:08:54.113043 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eafe698f-1fcd-45ca-9696-45d87af7e2d5 None admin] RBAC: Adding query filter params (name=service) {{(pid=81409) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 13 04:08:54.113559 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eafe698f-1fcd-45ca-9696-45d87af7e2d5 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:08:54.113913 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 181/368] 10.222.0.27 () {58 vars in 1024 bytes} [Thu Oct 13 04:08:53 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 130 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 13 04:08:54.114855 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f25b5586-84e3-45f5-aace-772c62a36605 None None] Authenticating user token {{(pid=81408) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:08:54.115778 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eafe698f-1fcd-45ca-9696-45d87af7e2d5 None admin] RBAC: Authorization granted {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:08:54.126978 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 188/369] 10.222.0.27 () {56 vars in 1103 bytes} [Thu Oct 13 04:08:53 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 214 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 13 04:08:54.136213 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eafe698f-1fcd-45ca-9696-45d87af7e2d5 None admin] Authenticating user token {{(pid=81409) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:08:54.304442 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f25b5586-84e3-45f5-aace-772c62a36605 None None] Validating token access rules against request {{(pid=81408) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:08:54.327163 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eafe698f-1fcd-45ca-9696-45d87af7e2d5 None admin] Validating token access rules against request {{(pid=81409) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:08:54.367615 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-809c4221-1487-4c3e-80ed-f9897a7b7e32 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81409) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:08:54.368913 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-809c4221-1487-4c3e-80ed-f9897a7b7e32 None admin] REQUEST_METHOD: `GET` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:08:54.369195 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-809c4221-1487-4c3e-80ed-f9897a7b7e32 None admin] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:08:54.369469 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-809c4221-1487-4c3e-80ed-f9897a7b7e32 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:08:54.387773 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c4e58802-91b9-4b7b-aac1-82929316f873 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81408) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:08:54.388431 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4e58802-91b9-4b7b-aac1-82929316f873 None admin] REQUEST_METHOD: `GET` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:08:54.388706 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4e58802-91b9-4b7b-aac1-82929316f873 None admin] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:08:54.388976 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4e58802-91b9-4b7b-aac1-82929316f873 None admin] PATH_INFO: `/v3/roles/service` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:08:54.400436 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-809c4221-1487-4c3e-80ed-f9897a7b7e32 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:08:54.401916 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-809c4221-1487-4c3e-80ed-f9897a7b7e32 None admin] RBAC: Authorization granted {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:08:54.423245 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c4e58802-91b9-4b7b-aac1-82929316f873 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 13 04:08:54.426431 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG dbcounter [-] [81409] Writing DB stats keystone:SELECT=100 {{(pid=81409) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:08:54.429336 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application [None req-c4e58802-91b9-4b7b-aac1-82929316f873 None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Oct 13 04:08:54.429336 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 13 04:08:54.429336 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 13 04:08:54.429336 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 13 04:08:54.429336 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 13 04:08:54.429336 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 13 04:08:54.429336 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 13 04:08:54.429336 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 13 04:08:54.429336 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 13 04:08:54.429336 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 13 04:08:54.429336 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 13 04:08:54.429336 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 13 04:08:54.429336 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 13 04:08:54.429336 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 13 04:08:54.429336 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 13 04:08:54.429336 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 13 04:08:54.430267 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 13 04:08:54.430267 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 13 04:08:54.430267 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 13 04:08:54.430267 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 13 04:08:54.430267 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 13 04:08:54.430267 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 13 04:08:54.430267 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 13 04:08:54.430267 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 13 04:08:54.430267 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 13 04:08:54.430267 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application return self.get_or_create( Oct 13 04:08:54.430267 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 13 04:08:54.430267 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application with Lock( Oct 13 04:08:54.430267 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 13 04:08:54.430267 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application return self._enter() Oct 13 04:08:54.430267 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 13 04:08:54.430267 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 13 04:08:54.431591 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 13 04:08:54.431591 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application return self.creator() Oct 13 04:08:54.431591 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 13 04:08:54.431591 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application created_value = creator( Oct 13 04:08:54.431591 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 13 04:08:54.431591 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 13 04:08:54.431591 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 13 04:08:54.431591 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 13 04:08:54.431591 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 13 04:08:54.431591 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 13 04:08:54.431591 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Oct 13 04:08:54.431591 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application Oct 13 04:08:54.434148 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 182/370] 10.222.0.27 () {56 vars in 1102 bytes} [Thu Oct 13 04:08:54 2022] GET /identity/v3/roles/service => generated 85 bytes in 320 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 13 04:08:54.434294 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application [None req-809c4221-1487-4c3e-80ed-f9897a7b7e32 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 13 04:08:54.434294 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 13 04:08:54.434294 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 13 04:08:54.434294 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 13 04:08:54.434294 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 13 04:08:54.434294 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 13 04:08:54.434294 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 13 04:08:54.434294 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 13 04:08:54.434294 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 13 04:08:54.434294 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 13 04:08:54.434294 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 13 04:08:54.434294 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 13 04:08:54.434294 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 13 04:08:54.434294 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 13 04:08:54.434294 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 13 04:08:54.434294 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 13 04:08:54.435093 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 13 04:08:54.435093 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 13 04:08:54.435093 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 13 04:08:54.435093 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 13 04:08:54.435093 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 13 04:08:54.435093 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application return self.get_or_create( Oct 13 04:08:54.435093 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 13 04:08:54.435093 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application with Lock( Oct 13 04:08:54.435093 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 13 04:08:54.435093 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application return self._enter() Oct 13 04:08:54.435093 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 13 04:08:54.435093 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 13 04:08:54.435093 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 13 04:08:54.435093 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application return self.creator() Oct 13 04:08:54.435093 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 13 04:08:54.435093 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application created_value = creator( Oct 13 04:08:54.435093 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 13 04:08:54.435936 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 13 04:08:54.435936 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 13 04:08:54.435936 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application Oct 13 04:08:54.435936 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 189/371] 10.222.0.27 () {56 vars in 1108 bytes} [Thu Oct 13 04:08:54 2022] GET /identity/v3/domains/Default => generated 87 bytes in 300 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 13 04:08:54.436678 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-809c4221-1487-4c3e-80ed-f9897a7b7e32 None admin] Authenticating user token {{(pid=81409) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:08:54.439753 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c4e58802-91b9-4b7b-aac1-82929316f873 None admin] Authenticating user token {{(pid=81408) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:08:54.502303 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG dbcounter [-] [81408] Writing DB stats keystone:SELECT=100 {{(pid=81408) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:08:54.550612 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c4e58802-91b9-4b7b-aac1-82929316f873 None admin] Validating token access rules against request {{(pid=81408) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:08:54.552260 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-809c4221-1487-4c3e-80ed-f9897a7b7e32 None admin] Validating token access rules against request {{(pid=81409) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:08:54.587806 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1875bd75-8666-4c25-9d36-5c44b1c9c876 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81408) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:08:54.588409 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1875bd75-8666-4c25-9d36-5c44b1c9c876 None admin] REQUEST_METHOD: `GET` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:08:54.588680 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1875bd75-8666-4c25-9d36-5c44b1c9c876 None admin] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:08:54.589019 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1875bd75-8666-4c25-9d36-5c44b1c9c876 None admin] PATH_INFO: `/v3/roles` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:08:54.589687 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1875bd75-8666-4c25-9d36-5c44b1c9c876 None admin] RBAC: Adding query filter params (name=service) {{(pid=81408) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 13 04:08:54.589989 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1875bd75-8666-4c25-9d36-5c44b1c9c876 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:08:54.590091 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-537638f2-108c-4b61-a5d3-cbcf64a663d4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81409) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:08:54.590760 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-537638f2-108c-4b61-a5d3-cbcf64a663d4 None admin] REQUEST_METHOD: `GET` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:08:54.591035 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-537638f2-108c-4b61-a5d3-cbcf64a663d4 None admin] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:08:54.591305 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-537638f2-108c-4b61-a5d3-cbcf64a663d4 None admin] PATH_INFO: `/v3/roles/service` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:08:54.591394 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1875bd75-8666-4c25-9d36-5c44b1c9c876 None admin] RBAC: Authorization granted {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:08:54.605341 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 183/372] 10.222.0.27 () {56 vars in 1103 bytes} [Thu Oct 13 04:08:54 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 161 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 13 04:08:54.605341 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1875bd75-8666-4c25-9d36-5c44b1c9c876 None admin] Authenticating user token {{(pid=81408) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:08:54.607035 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-537638f2-108c-4b61-a5d3-cbcf64a663d4 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 13 04:08:54.612422 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application [None req-537638f2-108c-4b61-a5d3-cbcf64a663d4 None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Oct 13 04:08:54.612422 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 13 04:08:54.612422 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 13 04:08:54.612422 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 13 04:08:54.612422 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 13 04:08:54.612422 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 13 04:08:54.612422 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 13 04:08:54.612422 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 13 04:08:54.612422 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 13 04:08:54.612422 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 13 04:08:54.612422 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 13 04:08:54.612422 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 13 04:08:54.612422 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 13 04:08:54.612422 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 13 04:08:54.612422 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 13 04:08:54.612422 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 13 04:08:54.613345 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 13 04:08:54.613345 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 13 04:08:54.613345 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 13 04:08:54.613345 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 13 04:08:54.613345 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 13 04:08:54.613345 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 13 04:08:54.613345 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 13 04:08:54.613345 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 13 04:08:54.613345 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 13 04:08:54.613345 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application return self.get_or_create( Oct 13 04:08:54.613345 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 13 04:08:54.613345 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application with Lock( Oct 13 04:08:54.613345 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 13 04:08:54.613345 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application return self._enter() Oct 13 04:08:54.613345 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 13 04:08:54.613345 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 13 04:08:54.614473 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 13 04:08:54.614473 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application return self.creator() Oct 13 04:08:54.614473 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 13 04:08:54.614473 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application created_value = creator( Oct 13 04:08:54.614473 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 13 04:08:54.614473 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 13 04:08:54.614473 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 13 04:08:54.614473 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 13 04:08:54.614473 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 13 04:08:54.614473 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 13 04:08:54.614473 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Oct 13 04:08:54.614473 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application Oct 13 04:08:54.614473 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 190/373] 10.222.0.27 () {56 vars in 1102 bytes} [Thu Oct 13 04:08:54 2022] GET /identity/v3/roles/service => generated 85 bytes in 177 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 13 04:08:54.619721 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1028912-ea88-4142-94b3-f9faed978639 None None] REQUEST_METHOD: `GET` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:08:54.619941 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1028912-ea88-4142-94b3-f9faed978639 None None] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:08:54.620213 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1028912-ea88-4142-94b3-f9faed978639 None None] PATH_INFO: `/` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:08:54.621504 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 191/374] 10.222.0.27 () {54 vars in 927 bytes} [Thu Oct 13 04:08:54 2022] GET /identity => generated 270 bytes in 7 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 13 04:08:54.622222 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e1028912-ea88-4142-94b3-f9faed978639 None None] Authenticating user token {{(pid=81409) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:08:54.691485 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG dbcounter [-] [81409] Writing DB stats keystone:SELECT=100 {{(pid=81409) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:08:54.716804 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1875bd75-8666-4c25-9d36-5c44b1c9c876 None admin] Validating token access rules against request {{(pid=81408) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:08:54.736835 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e1028912-ea88-4142-94b3-f9faed978639 None None] Validating token access rules against request {{(pid=81409) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:08:54.738274 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG dbcounter [-] [81408] Writing DB stats keystone:SELECT=100 {{(pid=81408) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:08:54.765578 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-67a56e90-f240-41dd-9f8c-98f11eaf20b4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81408) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:08:54.766756 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67a56e90-f240-41dd-9f8c-98f11eaf20b4 None admin] REQUEST_METHOD: `GET` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:08:54.767197 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67a56e90-f240-41dd-9f8c-98f11eaf20b4 None admin] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:08:54.767797 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67a56e90-f240-41dd-9f8c-98f11eaf20b4 None admin] PATH_INFO: `/v3/domains` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:08:54.769176 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-67a56e90-f240-41dd-9f8c-98f11eaf20b4 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81408) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 13 04:08:54.769641 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-67a56e90-f240-41dd-9f8c-98f11eaf20b4 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:08:54.771198 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-67a56e90-f240-41dd-9f8c-98f11eaf20b4 None admin] RBAC: Authorization granted {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:08:54.772745 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b5cd9909-9fd3-42c1-89d3-d611068ddca2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81409) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:08:54.773371 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5cd9909-9fd3-42c1-89d3-d611068ddca2 None admin] REQUEST_METHOD: `GET` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:08:54.773642 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5cd9909-9fd3-42c1-89d3-d611068ddca2 None admin] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:08:54.773904 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5cd9909-9fd3-42c1-89d3-d611068ddca2 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:08:54.784561 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b5cd9909-9fd3-42c1-89d3-d611068ddca2 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:08:54.790787 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b5cd9909-9fd3-42c1-89d3-d611068ddca2 None admin] RBAC: Authorization granted {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:08:54.790920 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 184/375] 10.222.0.27 () {56 vars in 1109 bytes} [Thu Oct 13 04:08:54 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 188 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 13 04:08:54.790920 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-67a56e90-f240-41dd-9f8c-98f11eaf20b4 None admin] Authenticating user token {{(pid=81408) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:08:54.807816 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application [None req-b5cd9909-9fd3-42c1-89d3-d611068ddca2 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 13 04:08:54.807816 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 13 04:08:54.807816 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 13 04:08:54.807816 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 13 04:08:54.807816 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 13 04:08:54.807816 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 13 04:08:54.807816 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 13 04:08:54.807816 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 13 04:08:54.807816 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 13 04:08:54.807816 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 13 04:08:54.807816 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 13 04:08:54.807816 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 13 04:08:54.807816 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 13 04:08:54.807816 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 13 04:08:54.807816 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 13 04:08:54.807816 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 13 04:08:54.808753 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 13 04:08:54.808753 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 13 04:08:54.808753 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 13 04:08:54.808753 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 13 04:08:54.808753 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 13 04:08:54.808753 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application return self.get_or_create( Oct 13 04:08:54.808753 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 13 04:08:54.808753 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application with Lock( Oct 13 04:08:54.808753 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 13 04:08:54.808753 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application return self._enter() Oct 13 04:08:54.808753 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 13 04:08:54.808753 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 13 04:08:54.808753 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 13 04:08:54.808753 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application return self.creator() Oct 13 04:08:54.808753 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 13 04:08:54.808753 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application created_value = creator( Oct 13 04:08:54.808753 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 13 04:08:54.809680 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 13 04:08:54.809680 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 13 04:08:54.809680 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application Oct 13 04:08:54.809680 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 192/376] 10.222.0.27 () {56 vars in 1108 bytes} [Thu Oct 13 04:08:54 2022] GET /identity/v3/domains/Default => generated 87 bytes in 187 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 13 04:08:54.810454 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11b9b59a-4c5c-469d-8076-2a53086430ff None None] REQUEST_METHOD: `POST` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:08:54.810565 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11b9b59a-4c5c-469d-8076-2a53086430ff None None] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:08:54.810850 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11b9b59a-4c5c-469d-8076-2a53086430ff None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:08:54.830831 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.auth.core [None req-11b9b59a-4c5c-469d-8076-2a53086430ff None None] MFA Rules not processed for user `0e2d979563d6421995451a0e86bf3022`. Rule list: `[]` (Enabled: `True`). {{(pid=81409) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 13 04:08:54.832015 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.fernet_utils [None req-11b9b59a-4c5c-469d-8076-2a53086430ff None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81409) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:08:54.896767 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-67a56e90-f240-41dd-9f8c-98f11eaf20b4 None admin] Validating token access rules against request {{(pid=81408) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:08:54.910237 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 193/377] 10.222.0.27 () {58 vars in 1024 bytes} [Thu Oct 13 04:08:54 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 94 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 13 04:08:54.910237 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-11b9b59a-4c5c-469d-8076-2a53086430ff None None] Authenticating user token {{(pid=81409) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:08:54.944812 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1c987ec0-7a40-4da2-9f00-9f30a2abfcea None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81408) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:08:54.945522 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c987ec0-7a40-4da2-9f00-9f30a2abfcea None admin] REQUEST_METHOD: `GET` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:08:54.945872 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c987ec0-7a40-4da2-9f00-9f30a2abfcea None admin] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:08:54.946295 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c987ec0-7a40-4da2-9f00-9f30a2abfcea None admin] PATH_INFO: `/v3/roles` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:08:54.947405 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1c987ec0-7a40-4da2-9f00-9f30a2abfcea None admin] RBAC: Adding query filter params (name=service) {{(pid=81408) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 13 04:08:54.948014 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1c987ec0-7a40-4da2-9f00-9f30a2abfcea None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:08:54.949693 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1c987ec0-7a40-4da2-9f00-9f30a2abfcea None admin] RBAC: Authorization granted {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:08:54.951571 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG dbcounter [-] [81409] Writing DB stats keystone:SELECT=100 {{(pid=81409) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:08:54.960320 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 185/378] 10.222.0.27 () {56 vars in 1103 bytes} [Thu Oct 13 04:08:54 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 171 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 13 04:08:54.961332 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1c987ec0-7a40-4da2-9f00-9f30a2abfcea None admin] Authenticating user token {{(pid=81408) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:08:55.020044 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG dbcounter [-] [81408] Writing DB stats keystone:SELECT=100 {{(pid=81408) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:08:55.021719 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-11b9b59a-4c5c-469d-8076-2a53086430ff None None] Validating token access rules against request {{(pid=81409) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:08:55.058186 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ae82290b-0847-437f-8915-02f807c977a4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81409) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:08:55.058955 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae82290b-0847-437f-8915-02f807c977a4 None admin] REQUEST_METHOD: `GET` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:08:55.059234 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae82290b-0847-437f-8915-02f807c977a4 None admin] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:08:55.059505 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae82290b-0847-437f-8915-02f807c977a4 None admin] PATH_INFO: `/v3/users/nova` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:08:55.066983 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ae82290b-0847-437f-8915-02f807c977a4 None admin] RBAC: Authorizing `identity:get_user(user_id=nova)` {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:08:55.068567 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ae82290b-0847-437f-8915-02f807c977a4 None admin] RBAC: Authorization granted {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:08:55.082620 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application [None req-ae82290b-0847-437f-8915-02f807c977a4 None admin] Could not find user: nova.: keystone.exception.UserNotFound: Could not find user: nova. Oct 13 04:08:55.082620 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 13 04:08:55.082620 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 13 04:08:55.082620 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 13 04:08:55.082620 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 13 04:08:55.082620 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 13 04:08:55.082620 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 13 04:08:55.082620 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 13 04:08:55.082620 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 13 04:08:55.082620 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 13 04:08:55.082620 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 13 04:08:55.082620 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 13 04:08:55.082620 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Oct 13 04:08:55.082620 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application return self._get_user(user_id) Oct 13 04:08:55.082620 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Oct 13 04:08:55.082620 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Oct 13 04:08:55.083554 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 13 04:08:55.083554 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 13 04:08:55.083554 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Oct 13 04:08:55.083554 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 13 04:08:55.083554 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Oct 13 04:08:55.083554 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 13 04:08:55.083554 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 13 04:08:55.083554 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 13 04:08:55.083554 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 13 04:08:55.083554 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application return self.get_or_create( Oct 13 04:08:55.083554 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 13 04:08:55.083554 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application with Lock( Oct 13 04:08:55.083554 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 13 04:08:55.083554 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application return self._enter() Oct 13 04:08:55.083554 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 13 04:08:55.083554 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 13 04:08:55.083554 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 13 04:08:55.083554 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application return self.creator() Oct 13 04:08:55.084435 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 13 04:08:55.084435 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application created_value = creator( Oct 13 04:08:55.084435 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Oct 13 04:08:55.084435 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Oct 13 04:08:55.084435 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Oct 13 04:08:55.084435 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Oct 13 04:08:55.084435 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Oct 13 04:08:55.084435 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Oct 13 04:08:55.084435 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: nova. Oct 13 04:08:55.084435 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application Oct 13 04:08:55.084435 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 194/379] 10.222.0.27 () {56 vars in 1093 bytes} [Thu Oct 13 04:08:54 2022] GET /identity/v3/users/nova => generated 82 bytes in 179 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 13 04:08:55.085000 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ae82290b-0847-437f-8915-02f807c977a4 None admin] Authenticating user token {{(pid=81409) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:08:55.103955 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1c987ec0-7a40-4da2-9f00-9f30a2abfcea None admin] Validating token access rules against request {{(pid=81408) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:08:55.140852 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-da0d8811-43d3-42df-a25c-213cdc464890 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81408) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:08:55.141456 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da0d8811-43d3-42df-a25c-213cdc464890 None admin] REQUEST_METHOD: `GET` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:08:55.141731 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da0d8811-43d3-42df-a25c-213cdc464890 None admin] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:08:55.142344 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da0d8811-43d3-42df-a25c-213cdc464890 None admin] PATH_INFO: `/v3/domains` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:08:55.142686 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-da0d8811-43d3-42df-a25c-213cdc464890 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81408) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 13 04:08:55.142986 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-da0d8811-43d3-42df-a25c-213cdc464890 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:08:55.144353 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-da0d8811-43d3-42df-a25c-213cdc464890 None admin] RBAC: Authorization granted {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:08:55.160911 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 186/380] 10.222.0.27 () {56 vars in 1109 bytes} [Thu Oct 13 04:08:54 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 196 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 13 04:08:55.160911 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4998eb3e-bf68-492e-b3d0-6587039049c0 None None] REQUEST_METHOD: `GET` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:08:55.160911 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4998eb3e-bf68-492e-b3d0-6587039049c0 None None] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:08:55.160911 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4998eb3e-bf68-492e-b3d0-6587039049c0 None None] PATH_INFO: `/` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:08:55.160911 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 187/381] 10.222.0.27 () {54 vars in 927 bytes} [Thu Oct 13 04:08:55 2022] GET /identity => generated 270 bytes in 2 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 13 04:08:55.160911 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4998eb3e-bf68-492e-b3d0-6587039049c0 None None] Authenticating user token {{(pid=81408) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:08:55.201400 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ae82290b-0847-437f-8915-02f807c977a4 None admin] Validating token access rules against request {{(pid=81409) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:08:55.202970 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG dbcounter [-] [81409] Writing DB stats keystone:SELECT=100 {{(pid=81409) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:08:55.239085 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-623cba6e-9e3c-4d4f-966b-a6708776451c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81409) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:08:55.239712 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-623cba6e-9e3c-4d4f-966b-a6708776451c None admin] REQUEST_METHOD: `GET` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:08:55.239989 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-623cba6e-9e3c-4d4f-966b-a6708776451c None admin] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:08:55.240257 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-623cba6e-9e3c-4d4f-966b-a6708776451c None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:08:55.251359 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-623cba6e-9e3c-4d4f-966b-a6708776451c None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:08:55.252814 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-623cba6e-9e3c-4d4f-966b-a6708776451c None admin] RBAC: Authorization granted {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:08:55.262454 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG dbcounter [-] [81408] Writing DB stats keystone:SELECT=100 {{(pid=81408) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:08:55.267565 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application [None req-623cba6e-9e3c-4d4f-966b-a6708776451c None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 13 04:08:55.267565 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 13 04:08:55.267565 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 13 04:08:55.267565 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 13 04:08:55.267565 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 13 04:08:55.267565 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 13 04:08:55.267565 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 13 04:08:55.267565 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 13 04:08:55.267565 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 13 04:08:55.267565 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 13 04:08:55.267565 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 13 04:08:55.267565 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 13 04:08:55.267565 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 13 04:08:55.267565 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 13 04:08:55.267565 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 13 04:08:55.267565 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 13 04:08:55.268636 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 13 04:08:55.268636 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 13 04:08:55.268636 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 13 04:08:55.268636 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 13 04:08:55.268636 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 13 04:08:55.268636 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application return self.get_or_create( Oct 13 04:08:55.268636 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 13 04:08:55.268636 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application with Lock( Oct 13 04:08:55.268636 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 13 04:08:55.268636 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application return self._enter() Oct 13 04:08:55.268636 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 13 04:08:55.268636 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 13 04:08:55.268636 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 13 04:08:55.268636 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application return self.creator() Oct 13 04:08:55.268636 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 13 04:08:55.268636 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application created_value = creator( Oct 13 04:08:55.268636 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 13 04:08:55.274281 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 13 04:08:55.274281 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 13 04:08:55.274281 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application Oct 13 04:08:55.274522 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4998eb3e-bf68-492e-b3d0-6587039049c0 None None] Validating token access rules against request {{(pid=81408) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:08:55.274611 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 195/382] 10.222.0.27 () {56 vars in 1108 bytes} [Thu Oct 13 04:08:55 2022] GET /identity/v3/domains/Default => generated 87 bytes in 185 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 13 04:08:55.274611 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-623cba6e-9e3c-4d4f-966b-a6708776451c None admin] Authenticating user token {{(pid=81409) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:08:55.309386 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e2d33dae-c85c-4f5d-a155-d5d49c7ce72e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81408) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:08:55.309989 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2d33dae-c85c-4f5d-a155-d5d49c7ce72e None admin] REQUEST_METHOD: `GET` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:08:55.310304 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2d33dae-c85c-4f5d-a155-d5d49c7ce72e None admin] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:08:55.310575 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2d33dae-c85c-4f5d-a155-d5d49c7ce72e None admin] PATH_INFO: `/v3/users` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:08:55.311304 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e2d33dae-c85c-4f5d-a155-d5d49c7ce72e None admin] RBAC: Adding query filter params (domain_id=default, name=nova) {{(pid=81408) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 13 04:08:55.311606 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e2d33dae-c85c-4f5d-a155-d5d49c7ce72e None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:08:55.313037 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e2d33dae-c85c-4f5d-a155-d5d49c7ce72e None admin] RBAC: Authorization granted {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:08:55.348896 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 188/383] 10.222.0.27 () {56 vars in 1133 bytes} [Thu Oct 13 04:08:55 2022] GET /identity/v3/users?domain_id=default&name=nova => generated 368 bytes in 189 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 13 04:08:55.350537 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6d055cd-60f8-473d-8163-bbc85c3e0182 None None] REQUEST_METHOD: `POST` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:08:55.350848 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6d055cd-60f8-473d-8163-bbc85c3e0182 None None] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:08:55.351123 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6d055cd-60f8-473d-8163-bbc85c3e0182 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:08:55.369172 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.auth.core [None req-b6d055cd-60f8-473d-8163-bbc85c3e0182 None None] MFA Rules not processed for user `0e2d979563d6421995451a0e86bf3022`. Rule list: `[]` (Enabled: `True`). {{(pid=81408) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 13 04:08:55.370387 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.fernet_utils [None req-b6d055cd-60f8-473d-8163-bbc85c3e0182 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81408) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:08:55.373713 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-623cba6e-9e3c-4d4f-966b-a6708776451c None admin] Validating token access rules against request {{(pid=81409) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:08:55.407178 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-02115c3b-fd34-41b8-9319-951b3042acb0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81409) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:08:55.407796 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-02115c3b-fd34-41b8-9319-951b3042acb0 None admin] REQUEST_METHOD: `GET` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:08:55.408074 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-02115c3b-fd34-41b8-9319-951b3042acb0 None admin] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:08:55.408345 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-02115c3b-fd34-41b8-9319-951b3042acb0 None admin] PATH_INFO: `/v3/users/cinder` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:08:55.420324 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-02115c3b-fd34-41b8-9319-951b3042acb0 None admin] RBAC: Authorizing `identity:get_user(user_id=cinder)` {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:08:55.421823 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-02115c3b-fd34-41b8-9319-951b3042acb0 None admin] RBAC: Authorization granted {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:08:55.433150 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application [None req-02115c3b-fd34-41b8-9319-951b3042acb0 None admin] Could not find user: cinder.: keystone.exception.UserNotFound: Could not find user: cinder. Oct 13 04:08:55.433150 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 13 04:08:55.433150 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 13 04:08:55.433150 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 13 04:08:55.433150 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 13 04:08:55.433150 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 13 04:08:55.433150 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 13 04:08:55.433150 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 13 04:08:55.433150 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 13 04:08:55.433150 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 13 04:08:55.433150 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 13 04:08:55.433150 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 13 04:08:55.433150 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Oct 13 04:08:55.433150 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application return self._get_user(user_id) Oct 13 04:08:55.433150 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Oct 13 04:08:55.433150 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Oct 13 04:08:55.434156 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 13 04:08:55.434156 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 13 04:08:55.434156 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Oct 13 04:08:55.434156 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 13 04:08:55.434156 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Oct 13 04:08:55.434156 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 13 04:08:55.434156 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 13 04:08:55.434156 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 13 04:08:55.434156 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 13 04:08:55.434156 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application return self.get_or_create( Oct 13 04:08:55.434156 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 13 04:08:55.434156 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application with Lock( Oct 13 04:08:55.434156 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 13 04:08:55.434156 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application return self._enter() Oct 13 04:08:55.434156 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 13 04:08:55.434156 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 13 04:08:55.434156 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 13 04:08:55.434156 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application return self.creator() Oct 13 04:08:55.435073 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 13 04:08:55.435073 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application created_value = creator( Oct 13 04:08:55.435073 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Oct 13 04:08:55.435073 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Oct 13 04:08:55.435073 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Oct 13 04:08:55.435073 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Oct 13 04:08:55.435073 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Oct 13 04:08:55.435073 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Oct 13 04:08:55.435073 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: cinder. Oct 13 04:08:55.435073 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application Oct 13 04:08:55.435073 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 196/384] 10.222.0.27 () {56 vars in 1099 bytes} [Thu Oct 13 04:08:55 2022] GET /identity/v3/users/cinder => generated 84 bytes in 165 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 13 04:08:55.435073 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-02115c3b-fd34-41b8-9319-951b3042acb0 None admin] Authenticating user token {{(pid=81409) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:08:55.442981 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 189/385] 10.222.0.27 () {58 vars in 1024 bytes} [Thu Oct 13 04:08:55 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 94 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 13 04:08:55.443820 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b6d055cd-60f8-473d-8163-bbc85c3e0182 None None] Authenticating user token {{(pid=81408) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:08:55.460069 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG dbcounter [-] [81409] Writing DB stats keystone:SELECT=100 {{(pid=81409) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:08:55.531366 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG dbcounter [-] [81408] Writing DB stats keystone:SELECT=100 {{(pid=81408) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:08:55.548660 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b6d055cd-60f8-473d-8163-bbc85c3e0182 None None] Validating token access rules against request {{(pid=81408) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:08:55.555654 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-02115c3b-fd34-41b8-9319-951b3042acb0 None admin] Validating token access rules against request {{(pid=81409) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:08:55.585968 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3bc83786-edd7-4e18-8d20-d11f63e95f5e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81408) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:08:55.586545 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3bc83786-edd7-4e18-8d20-d11f63e95f5e None admin] REQUEST_METHOD: `GET` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:08:55.586989 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3bc83786-edd7-4e18-8d20-d11f63e95f5e None admin] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:08:55.587264 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3bc83786-edd7-4e18-8d20-d11f63e95f5e None admin] PATH_INFO: `/v3/domains` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:08:55.588127 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3bc83786-edd7-4e18-8d20-d11f63e95f5e None admin] RBAC: Adding query filter params (name=Default) {{(pid=81408) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 13 04:08:55.588242 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3bc83786-edd7-4e18-8d20-d11f63e95f5e None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:08:55.589566 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3bc83786-edd7-4e18-8d20-d11f63e95f5e None admin] RBAC: Authorization granted {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:08:55.592972 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c3581196-2e0f-4311-8687-68a6f2539028 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81409) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:08:55.593603 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c3581196-2e0f-4311-8687-68a6f2539028 None admin] REQUEST_METHOD: `GET` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:08:55.593880 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c3581196-2e0f-4311-8687-68a6f2539028 None admin] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:08:55.594151 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c3581196-2e0f-4311-8687-68a6f2539028 None admin] PATH_INFO: `/v3/roles/service` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:08:55.601498 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 190/386] 10.222.0.27 () {56 vars in 1109 bytes} [Thu Oct 13 04:08:55 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 158 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 13 04:08:55.603733 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3bc83786-edd7-4e18-8d20-d11f63e95f5e None admin] Authenticating user token {{(pid=81408) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:08:55.605703 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c3581196-2e0f-4311-8687-68a6f2539028 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 13 04:08:55.611135 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application [None req-c3581196-2e0f-4311-8687-68a6f2539028 None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Oct 13 04:08:55.611135 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 13 04:08:55.611135 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 13 04:08:55.611135 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 13 04:08:55.611135 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 13 04:08:55.611135 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 13 04:08:55.611135 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 13 04:08:55.611135 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 13 04:08:55.611135 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 13 04:08:55.611135 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 13 04:08:55.611135 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 13 04:08:55.611135 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 13 04:08:55.611135 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 13 04:08:55.611135 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 13 04:08:55.611135 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 13 04:08:55.611135 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 13 04:08:55.612045 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 13 04:08:55.612045 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 13 04:08:55.612045 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 13 04:08:55.612045 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 13 04:08:55.612045 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 13 04:08:55.612045 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 13 04:08:55.612045 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 13 04:08:55.612045 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 13 04:08:55.612045 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 13 04:08:55.612045 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application return self.get_or_create( Oct 13 04:08:55.612045 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 13 04:08:55.612045 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application with Lock( Oct 13 04:08:55.612045 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 13 04:08:55.612045 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application return self._enter() Oct 13 04:08:55.612045 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 13 04:08:55.612045 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 13 04:08:55.612826 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 13 04:08:55.612826 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application return self.creator() Oct 13 04:08:55.612826 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 13 04:08:55.612826 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application created_value = creator( Oct 13 04:08:55.612826 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 13 04:08:55.612826 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 13 04:08:55.612826 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 13 04:08:55.612826 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 13 04:08:55.612826 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 13 04:08:55.612826 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 13 04:08:55.612826 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Oct 13 04:08:55.612826 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application Oct 13 04:08:55.612826 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 197/387] 10.222.0.27 () {56 vars in 1102 bytes} [Thu Oct 13 04:08:55 2022] GET /identity/v3/roles/service => generated 85 bytes in 178 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 13 04:08:55.613499 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c3581196-2e0f-4311-8687-68a6f2539028 None admin] Authenticating user token {{(pid=81409) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:08:55.693447 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG dbcounter [-] [81409] Writing DB stats keystone:SELECT=100 {{(pid=81409) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:08:55.717815 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3bc83786-edd7-4e18-8d20-d11f63e95f5e None admin] Validating token access rules against request {{(pid=81408) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:08:55.722196 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c3581196-2e0f-4311-8687-68a6f2539028 None admin] Validating token access rules against request {{(pid=81409) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:08:55.751527 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-21ecd316-fafa-4ca7-aa1d-ca0c67a24c1a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81408) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:08:55.752154 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21ecd316-fafa-4ca7-aa1d-ca0c67a24c1a None admin] REQUEST_METHOD: `GET` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:08:55.752431 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21ecd316-fafa-4ca7-aa1d-ca0c67a24c1a None admin] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:08:55.752701 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21ecd316-fafa-4ca7-aa1d-ca0c67a24c1a None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:08:55.756774 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cc9d7f8d-689b-4ae2-b02c-602e3c7a6c21 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81409) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:08:55.757368 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc9d7f8d-689b-4ae2-b02c-602e3c7a6c21 None admin] REQUEST_METHOD: `GET` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:08:55.757643 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc9d7f8d-689b-4ae2-b02c-602e3c7a6c21 None admin] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:08:55.757913 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc9d7f8d-689b-4ae2-b02c-602e3c7a6c21 None admin] PATH_INFO: `/v3/users` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:08:55.759218 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cc9d7f8d-689b-4ae2-b02c-602e3c7a6c21 None admin] RBAC: Adding query filter params (domain_id=default, name=cinder) {{(pid=81409) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 13 04:08:55.759524 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cc9d7f8d-689b-4ae2-b02c-602e3c7a6c21 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:08:55.760955 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cc9d7f8d-689b-4ae2-b02c-602e3c7a6c21 None admin] RBAC: Authorization granted {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:08:55.764890 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-21ecd316-fafa-4ca7-aa1d-ca0c67a24c1a None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:08:55.766374 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-21ecd316-fafa-4ca7-aa1d-ca0c67a24c1a None admin] RBAC: Authorization granted {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:08:55.772631 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG dbcounter [-] [81408] Writing DB stats keystone:SELECT=100 {{(pid=81408) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:08:55.781172 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application [None req-21ecd316-fafa-4ca7-aa1d-ca0c67a24c1a None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 13 04:08:55.781172 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 13 04:08:55.781172 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 13 04:08:55.781172 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 13 04:08:55.781172 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 13 04:08:55.781172 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 13 04:08:55.781172 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 13 04:08:55.781172 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 13 04:08:55.781172 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 13 04:08:55.781172 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 13 04:08:55.781172 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 13 04:08:55.781172 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 13 04:08:55.781172 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 13 04:08:55.781172 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 13 04:08:55.781172 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 13 04:08:55.781172 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 13 04:08:55.782282 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 13 04:08:55.782282 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 13 04:08:55.782282 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 13 04:08:55.782282 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 13 04:08:55.782282 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 13 04:08:55.782282 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application return self.get_or_create( Oct 13 04:08:55.782282 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 13 04:08:55.782282 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application with Lock( Oct 13 04:08:55.782282 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 13 04:08:55.782282 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application return self._enter() Oct 13 04:08:55.782282 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 13 04:08:55.782282 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 13 04:08:55.782282 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 13 04:08:55.782282 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application return self.creator() Oct 13 04:08:55.782282 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 13 04:08:55.782282 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application created_value = creator( Oct 13 04:08:55.782282 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 13 04:08:55.783128 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 13 04:08:55.783128 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 13 04:08:55.783128 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application Oct 13 04:08:55.783128 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 191/388] 10.222.0.27 () {56 vars in 1108 bytes} [Thu Oct 13 04:08:55 2022] GET /identity/v3/domains/Default => generated 87 bytes in 180 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 13 04:08:55.783128 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-21ecd316-fafa-4ca7-aa1d-ca0c67a24c1a None admin] Authenticating user token {{(pid=81408) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:08:55.798739 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 198/389] 10.222.0.27 () {56 vars in 1137 bytes} [Thu Oct 13 04:08:55 2022] GET /identity/v3/users?domain_id=default&name=cinder => generated 372 bytes in 186 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 13 04:08:55.799570 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cc9d7f8d-689b-4ae2-b02c-602e3c7a6c21 None admin] Authenticating user token {{(pid=81409) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:08:55.885227 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-21ecd316-fafa-4ca7-aa1d-ca0c67a24c1a None admin] Validating token access rules against request {{(pid=81408) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:08:55.899682 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cc9d7f8d-689b-4ae2-b02c-602e3c7a6c21 None admin] Validating token access rules against request {{(pid=81409) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:08:55.918778 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-feb183e4-e5bf-4fa1-b6ad-d00b6f16dc0c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81408) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:08:55.919404 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-feb183e4-e5bf-4fa1-b6ad-d00b6f16dc0c None admin] REQUEST_METHOD: `GET` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:08:55.919681 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-feb183e4-e5bf-4fa1-b6ad-d00b6f16dc0c None admin] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:08:55.919955 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-feb183e4-e5bf-4fa1-b6ad-d00b6f16dc0c None admin] PATH_INFO: `/v3/users/glance` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:08:55.927111 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-feb183e4-e5bf-4fa1-b6ad-d00b6f16dc0c None admin] RBAC: Authorizing `identity:get_user(user_id=glance)` {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:08:55.928498 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG dbcounter [-] [81409] Writing DB stats keystone:SELECT=100 {{(pid=81409) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:08:55.928610 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-feb183e4-e5bf-4fa1-b6ad-d00b6f16dc0c None admin] RBAC: Authorization granted {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:08:55.934177 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1f805b53-a4f3-426f-b7f4-4aaa239f5eaa None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81409) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:08:55.934806 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f805b53-a4f3-426f-b7f4-4aaa239f5eaa None admin] REQUEST_METHOD: `GET` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:08:55.935084 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f805b53-a4f3-426f-b7f4-4aaa239f5eaa None admin] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:08:55.935353 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f805b53-a4f3-426f-b7f4-4aaa239f5eaa None admin] PATH_INFO: `/v3/roles` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:08:55.936008 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1f805b53-a4f3-426f-b7f4-4aaa239f5eaa None admin] RBAC: Adding query filter params (name=service) {{(pid=81409) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 13 04:08:55.936311 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1f805b53-a4f3-426f-b7f4-4aaa239f5eaa None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:08:55.937676 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1f805b53-a4f3-426f-b7f4-4aaa239f5eaa None admin] RBAC: Authorization granted {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:08:55.941514 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application [None req-feb183e4-e5bf-4fa1-b6ad-d00b6f16dc0c None admin] Could not find user: glance.: keystone.exception.UserNotFound: Could not find user: glance. Oct 13 04:08:55.941514 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 13 04:08:55.941514 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 13 04:08:55.941514 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 13 04:08:55.941514 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 13 04:08:55.941514 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 13 04:08:55.941514 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 13 04:08:55.941514 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 13 04:08:55.941514 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 13 04:08:55.941514 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 13 04:08:55.941514 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 13 04:08:55.941514 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 13 04:08:55.941514 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Oct 13 04:08:55.941514 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application return self._get_user(user_id) Oct 13 04:08:55.941514 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Oct 13 04:08:55.941514 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Oct 13 04:08:55.942591 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 13 04:08:55.942591 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 13 04:08:55.942591 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Oct 13 04:08:55.942591 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 13 04:08:55.942591 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Oct 13 04:08:55.942591 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 13 04:08:55.942591 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 13 04:08:55.942591 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 13 04:08:55.942591 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 13 04:08:55.942591 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application return self.get_or_create( Oct 13 04:08:55.942591 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 13 04:08:55.942591 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application with Lock( Oct 13 04:08:55.942591 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 13 04:08:55.942591 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application return self._enter() Oct 13 04:08:55.942591 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 13 04:08:55.942591 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 13 04:08:55.942591 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 13 04:08:55.942591 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application return self.creator() Oct 13 04:08:55.944326 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 13 04:08:55.944326 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application created_value = creator( Oct 13 04:08:55.944326 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Oct 13 04:08:55.944326 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Oct 13 04:08:55.944326 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Oct 13 04:08:55.944326 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Oct 13 04:08:55.944326 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Oct 13 04:08:55.944326 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Oct 13 04:08:55.944326 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: glance. Oct 13 04:08:55.944326 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application Oct 13 04:08:55.944326 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 192/390] 10.222.0.27 () {56 vars in 1099 bytes} [Thu Oct 13 04:08:55 2022] GET /identity/v3/users/glance => generated 84 bytes in 160 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 13 04:08:55.944326 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-feb183e4-e5bf-4fa1-b6ad-d00b6f16dc0c None admin] Authenticating user token {{(pid=81408) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:08:55.947575 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 199/391] 10.222.0.27 () {56 vars in 1103 bytes} [Thu Oct 13 04:08:55 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 148 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 13 04:08:55.948238 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1f805b53-a4f3-426f-b7f4-4aaa239f5eaa None admin] Authenticating user token {{(pid=81409) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:08:56.004996 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG dbcounter [-] [81408] Writing DB stats keystone:SELECT=100 {{(pid=81408) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:08:56.046646 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-feb183e4-e5bf-4fa1-b6ad-d00b6f16dc0c None admin] Validating token access rules against request {{(pid=81408) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:08:56.049557 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1f805b53-a4f3-426f-b7f4-4aaa239f5eaa None admin] Validating token access rules against request {{(pid=81409) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:08:56.080377 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bbfe532d-aee6-4ca9-b4fa-6c7c3b3b8320 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81408) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:08:56.081004 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bbfe532d-aee6-4ca9-b4fa-6c7c3b3b8320 None admin] REQUEST_METHOD: `GET` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:08:56.081282 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bbfe532d-aee6-4ca9-b4fa-6c7c3b3b8320 None admin] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:08:56.081552 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bbfe532d-aee6-4ca9-b4fa-6c7c3b3b8320 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:08:56.083505 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8560e1d8-7bb1-4fd2-b57a-f9e9af899f9a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81409) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:08:56.084107 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8560e1d8-7bb1-4fd2-b57a-f9e9af899f9a None admin] REQUEST_METHOD: `GET` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:08:56.084384 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8560e1d8-7bb1-4fd2-b57a-f9e9af899f9a None admin] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:08:56.084657 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8560e1d8-7bb1-4fd2-b57a-f9e9af899f9a None admin] PATH_INFO: `/v3/domains` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:08:56.085298 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8560e1d8-7bb1-4fd2-b57a-f9e9af899f9a None admin] RBAC: Adding query filter params (name=Default) {{(pid=81409) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 13 04:08:56.085601 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8560e1d8-7bb1-4fd2-b57a-f9e9af899f9a None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:08:56.087037 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8560e1d8-7bb1-4fd2-b57a-f9e9af899f9a None admin] RBAC: Authorization granted {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:08:56.092226 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bbfe532d-aee6-4ca9-b4fa-6c7c3b3b8320 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:08:56.093691 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bbfe532d-aee6-4ca9-b4fa-6c7c3b3b8320 None admin] RBAC: Authorization granted {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:08:56.100988 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 200/392] 10.222.0.27 () {56 vars in 1109 bytes} [Thu Oct 13 04:08:55 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 153 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 13 04:08:56.106147 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8560e1d8-7bb1-4fd2-b57a-f9e9af899f9a None admin] Authenticating user token {{(pid=81409) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:08:56.108672 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application [None req-bbfe532d-aee6-4ca9-b4fa-6c7c3b3b8320 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 13 04:08:56.108672 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 13 04:08:56.108672 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 13 04:08:56.108672 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 13 04:08:56.108672 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 13 04:08:56.108672 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 13 04:08:56.108672 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 13 04:08:56.108672 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 13 04:08:56.108672 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 13 04:08:56.108672 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 13 04:08:56.108672 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 13 04:08:56.108672 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 13 04:08:56.108672 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 13 04:08:56.108672 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 13 04:08:56.108672 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 13 04:08:56.108672 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 13 04:08:56.109670 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 13 04:08:56.109670 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 13 04:08:56.109670 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 13 04:08:56.109670 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 13 04:08:56.109670 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 13 04:08:56.109670 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application return self.get_or_create( Oct 13 04:08:56.109670 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 13 04:08:56.109670 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application with Lock( Oct 13 04:08:56.109670 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 13 04:08:56.109670 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application return self._enter() Oct 13 04:08:56.109670 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 13 04:08:56.109670 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 13 04:08:56.109670 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 13 04:08:56.109670 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application return self.creator() Oct 13 04:08:56.109670 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 13 04:08:56.109670 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application created_value = creator( Oct 13 04:08:56.109670 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 13 04:08:56.110543 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 13 04:08:56.110543 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 13 04:08:56.110543 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application Oct 13 04:08:56.110543 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 193/393] 10.222.0.27 () {56 vars in 1108 bytes} [Thu Oct 13 04:08:55 2022] GET /identity/v3/domains/Default => generated 87 bytes in 167 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 13 04:08:56.110543 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bbfe532d-aee6-4ca9-b4fa-6c7c3b3b8320 None admin] Authenticating user token {{(pid=81408) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:08:56.158337 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG dbcounter [-] [81409] Writing DB stats keystone:SELECT=100 {{(pid=81409) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:08:56.203989 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8560e1d8-7bb1-4fd2-b57a-f9e9af899f9a None admin] Validating token access rules against request {{(pid=81409) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:08:56.215348 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bbfe532d-aee6-4ca9-b4fa-6c7c3b3b8320 None admin] Validating token access rules against request {{(pid=81408) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:08:56.225597 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG dbcounter [-] [81408] Writing DB stats keystone:SELECT=100 {{(pid=81408) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:08:56.237280 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fc53ef08-2407-4530-aa67-bb41a7ebb484 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81409) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:08:56.237896 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc53ef08-2407-4530-aa67-bb41a7ebb484 None admin] REQUEST_METHOD: `GET` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:08:56.238166 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc53ef08-2407-4530-aa67-bb41a7ebb484 None admin] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:08:56.238465 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc53ef08-2407-4530-aa67-bb41a7ebb484 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:08:56.248165 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fc53ef08-2407-4530-aa67-bb41a7ebb484 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:08:56.249600 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fc53ef08-2407-4530-aa67-bb41a7ebb484 None admin] RBAC: Authorization granted {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:08:56.249716 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2656c72a-fca2-4043-ad42-f815942f6548 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81408) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:08:56.250355 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2656c72a-fca2-4043-ad42-f815942f6548 None admin] REQUEST_METHOD: `GET` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:08:56.250634 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2656c72a-fca2-4043-ad42-f815942f6548 None admin] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:08:56.250906 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2656c72a-fca2-4043-ad42-f815942f6548 None admin] PATH_INFO: `/v3/users` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:08:56.251634 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2656c72a-fca2-4043-ad42-f815942f6548 None admin] RBAC: Adding query filter params (domain_id=default, name=glance) {{(pid=81408) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 13 04:08:56.251939 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2656c72a-fca2-4043-ad42-f815942f6548 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:08:56.253373 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2656c72a-fca2-4043-ad42-f815942f6548 None admin] RBAC: Authorization granted {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:08:56.264894 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application [None req-fc53ef08-2407-4530-aa67-bb41a7ebb484 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 13 04:08:56.264894 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 13 04:08:56.264894 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 13 04:08:56.264894 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 13 04:08:56.264894 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 13 04:08:56.264894 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 13 04:08:56.264894 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 13 04:08:56.264894 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 13 04:08:56.264894 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 13 04:08:56.264894 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 13 04:08:56.264894 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 13 04:08:56.264894 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 13 04:08:56.264894 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 13 04:08:56.264894 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 13 04:08:56.264894 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 13 04:08:56.264894 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 13 04:08:56.265767 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 13 04:08:56.265767 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 13 04:08:56.265767 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 13 04:08:56.265767 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 13 04:08:56.265767 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 13 04:08:56.265767 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application return self.get_or_create( Oct 13 04:08:56.265767 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 13 04:08:56.265767 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application with Lock( Oct 13 04:08:56.265767 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 13 04:08:56.265767 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application return self._enter() Oct 13 04:08:56.265767 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 13 04:08:56.265767 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 13 04:08:56.265767 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 13 04:08:56.265767 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application return self.creator() Oct 13 04:08:56.265767 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 13 04:08:56.265767 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application created_value = creator( Oct 13 04:08:56.265767 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 13 04:08:56.266867 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 13 04:08:56.266867 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 13 04:08:56.266867 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application Oct 13 04:08:56.266867 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 201/394] 10.222.0.27 () {56 vars in 1108 bytes} [Thu Oct 13 04:08:56 2022] GET /identity/v3/domains/Default => generated 87 bytes in 164 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 13 04:08:56.266867 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fc53ef08-2407-4530-aa67-bb41a7ebb484 None admin] Authenticating user token {{(pid=81409) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:08:56.271312 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 194/395] 10.222.0.27 () {56 vars in 1137 bytes} [Thu Oct 13 04:08:56 2022] GET /identity/v3/users?domain_id=default&name=glance => generated 372 bytes in 161 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 13 04:08:56.272010 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2656c72a-fca2-4043-ad42-f815942f6548 None admin] Authenticating user token {{(pid=81408) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:08:56.369502 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2656c72a-fca2-4043-ad42-f815942f6548 None admin] Validating token access rules against request {{(pid=81408) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:08:56.372448 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fc53ef08-2407-4530-aa67-bb41a7ebb484 None admin] Validating token access rules against request {{(pid=81409) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:08:56.379194 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG dbcounter [-] [81409] Writing DB stats keystone:SELECT=100 {{(pid=81409) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:08:56.404019 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-85113ad9-b709-45ac-8a9e-a3bb49e47994 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81408) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:08:56.404643 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85113ad9-b709-45ac-8a9e-a3bb49e47994 None admin] REQUEST_METHOD: `GET` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:08:56.404917 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85113ad9-b709-45ac-8a9e-a3bb49e47994 None admin] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:08:56.405187 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85113ad9-b709-45ac-8a9e-a3bb49e47994 None admin] PATH_INFO: `/v3/projects/service` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:08:56.409551 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4b62b934-45d8-4b47-8c14-7f16999f38d5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81409) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:08:56.410183 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b62b934-45d8-4b47-8c14-7f16999f38d5 None admin] REQUEST_METHOD: `GET` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:08:56.410499 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b62b934-45d8-4b47-8c14-7f16999f38d5 None admin] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:08:56.413172 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b62b934-45d8-4b47-8c14-7f16999f38d5 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:08:56.413279 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-85113ad9-b709-45ac-8a9e-a3bb49e47994 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:08:56.413361 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-85113ad9-b709-45ac-8a9e-a3bb49e47994 None admin] RBAC: Authorization granted {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:08:56.421952 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4b62b934-45d8-4b47-8c14-7f16999f38d5 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:08:56.423433 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4b62b934-45d8-4b47-8c14-7f16999f38d5 None admin] RBAC: Authorization granted {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:08:56.425891 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application [None req-85113ad9-b709-45ac-8a9e-a3bb49e47994 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Oct 13 04:08:56.425891 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 13 04:08:56.425891 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 13 04:08:56.425891 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 13 04:08:56.425891 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 13 04:08:56.425891 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 13 04:08:56.425891 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 13 04:08:56.425891 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 13 04:08:56.425891 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 13 04:08:56.425891 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 13 04:08:56.425891 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 13 04:08:56.425891 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 13 04:08:56.425891 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Oct 13 04:08:56.425891 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application return self._get_project(project_id) Oct 13 04:08:56.425891 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Oct 13 04:08:56.426908 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Oct 13 04:08:56.426908 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 13 04:08:56.426908 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 13 04:08:56.426908 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 13 04:08:56.426908 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 13 04:08:56.426908 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 13 04:08:56.426908 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application return self.get_or_create( Oct 13 04:08:56.426908 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 13 04:08:56.426908 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application with Lock( Oct 13 04:08:56.426908 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 13 04:08:56.426908 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application return self._enter() Oct 13 04:08:56.426908 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 13 04:08:56.426908 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 13 04:08:56.426908 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 13 04:08:56.426908 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application return self.creator() Oct 13 04:08:56.426908 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 13 04:08:56.426908 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application created_value = creator( Oct 13 04:08:56.427987 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Oct 13 04:08:56.427987 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Oct 13 04:08:56.427987 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Oct 13 04:08:56.427987 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Oct 13 04:08:56.427987 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Oct 13 04:08:56.427987 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Oct 13 04:08:56.427987 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Oct 13 04:08:56.427987 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application Oct 13 04:08:56.427987 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 195/396] 10.222.0.27 () {56 vars in 1111 bytes} [Thu Oct 13 04:08:56 2022] GET /identity/v3/projects/service => generated 88 bytes in 155 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 13 04:08:56.428466 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-85113ad9-b709-45ac-8a9e-a3bb49e47994 None admin] Authenticating user token {{(pid=81408) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:08:56.441933 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application [None req-4b62b934-45d8-4b47-8c14-7f16999f38d5 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 13 04:08:56.441933 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 13 04:08:56.441933 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 13 04:08:56.441933 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 13 04:08:56.441933 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 13 04:08:56.441933 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 13 04:08:56.441933 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 13 04:08:56.441933 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 13 04:08:56.441933 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 13 04:08:56.441933 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 13 04:08:56.441933 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 13 04:08:56.441933 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 13 04:08:56.441933 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 13 04:08:56.441933 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 13 04:08:56.441933 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 13 04:08:56.441933 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 13 04:08:56.443069 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 13 04:08:56.443069 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 13 04:08:56.443069 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 13 04:08:56.443069 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 13 04:08:56.443069 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 13 04:08:56.443069 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application return self.get_or_create( Oct 13 04:08:56.443069 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 13 04:08:56.443069 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application with Lock( Oct 13 04:08:56.443069 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 13 04:08:56.443069 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application return self._enter() Oct 13 04:08:56.443069 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 13 04:08:56.443069 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 13 04:08:56.443069 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 13 04:08:56.443069 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application return self.creator() Oct 13 04:08:56.443069 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 13 04:08:56.443069 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application created_value = creator( Oct 13 04:08:56.443069 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 13 04:08:56.443887 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 13 04:08:56.443887 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 13 04:08:56.443887 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application Oct 13 04:08:56.443887 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 202/397] 10.222.0.27 () {56 vars in 1108 bytes} [Thu Oct 13 04:08:56 2022] GET /identity/v3/domains/Default => generated 87 bytes in 177 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 13 04:08:56.443887 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4b62b934-45d8-4b47-8c14-7f16999f38d5 None admin] Authenticating user token {{(pid=81409) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:08:56.460259 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG dbcounter [-] [81408] Writing DB stats keystone:SELECT=100 {{(pid=81408) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:08:56.531458 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-85113ad9-b709-45ac-8a9e-a3bb49e47994 None admin] Validating token access rules against request {{(pid=81408) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:08:56.546479 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4b62b934-45d8-4b47-8c14-7f16999f38d5 None admin] Validating token access rules against request {{(pid=81409) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:08:56.564329 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c679fc06-b3af-4120-8e21-a1f2688e5c85 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81408) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:08:56.564921 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c679fc06-b3af-4120-8e21-a1f2688e5c85 None admin] REQUEST_METHOD: `GET` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:08:56.565192 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c679fc06-b3af-4120-8e21-a1f2688e5c85 None admin] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:08:56.565459 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c679fc06-b3af-4120-8e21-a1f2688e5c85 None admin] PATH_INFO: `/v3/domains` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:08:56.566087 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c679fc06-b3af-4120-8e21-a1f2688e5c85 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81408) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 13 04:08:56.566413 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c679fc06-b3af-4120-8e21-a1f2688e5c85 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:08:56.567801 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c679fc06-b3af-4120-8e21-a1f2688e5c85 None admin] RBAC: Authorization granted {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:08:56.578802 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 196/398] 10.222.0.27 () {56 vars in 1109 bytes} [Thu Oct 13 04:08:56 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 151 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 13 04:08:56.579485 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c679fc06-b3af-4120-8e21-a1f2688e5c85 None admin] Authenticating user token {{(pid=81408) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:08:56.584377 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c590264a-f747-42ba-91ed-a0382681be25 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81409) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:08:56.584986 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c590264a-f747-42ba-91ed-a0382681be25 None admin] REQUEST_METHOD: `GET` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:08:56.585264 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c590264a-f747-42ba-91ed-a0382681be25 None admin] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:08:56.585535 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c590264a-f747-42ba-91ed-a0382681be25 None admin] PATH_INFO: `/v3/domains` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:08:56.586177 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c590264a-f747-42ba-91ed-a0382681be25 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81409) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 13 04:08:56.586505 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c590264a-f747-42ba-91ed-a0382681be25 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:08:56.588411 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c590264a-f747-42ba-91ed-a0382681be25 None admin] RBAC: Authorization granted {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:08:56.600388 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 203/399] 10.222.0.27 () {56 vars in 1109 bytes} [Thu Oct 13 04:08:56 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 157 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 13 04:08:56.601184 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c590264a-f747-42ba-91ed-a0382681be25 None admin] Authenticating user token {{(pid=81409) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:08:56.619470 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG dbcounter [-] [81409] Writing DB stats keystone:SELECT=100 {{(pid=81409) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:08:56.665469 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG dbcounter [-] [81408] Writing DB stats keystone:SELECT=100 {{(pid=81408) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:08:56.677496 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c679fc06-b3af-4120-8e21-a1f2688e5c85 None admin] Validating token access rules against request {{(pid=81408) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:08:56.700970 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c590264a-f747-42ba-91ed-a0382681be25 None admin] Validating token access rules against request {{(pid=81409) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:08:56.710136 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ad4ffb2c-081b-4dc1-9a96-53d4345ff8dc None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81408) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:08:56.710860 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad4ffb2c-081b-4dc1-9a96-53d4345ff8dc None admin] REQUEST_METHOD: `GET` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:08:56.711087 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad4ffb2c-081b-4dc1-9a96-53d4345ff8dc None admin] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:08:56.711367 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad4ffb2c-081b-4dc1-9a96-53d4345ff8dc None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:08:56.722226 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ad4ffb2c-081b-4dc1-9a96-53d4345ff8dc None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:08:56.723685 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ad4ffb2c-081b-4dc1-9a96-53d4345ff8dc None admin] RBAC: Authorization granted {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:08:56.736000 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6138b34a-5946-4231-98f0-8c9b730f6bad None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81409) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:08:56.736599 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6138b34a-5946-4231-98f0-8c9b730f6bad None admin] REQUEST_METHOD: `GET` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:08:56.736874 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6138b34a-5946-4231-98f0-8c9b730f6bad None admin] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:08:56.737144 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6138b34a-5946-4231-98f0-8c9b730f6bad None admin] PATH_INFO: `/v3/projects` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:08:56.737824 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6138b34a-5946-4231-98f0-8c9b730f6bad None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81409) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 13 04:08:56.737911 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application [None req-ad4ffb2c-081b-4dc1-9a96-53d4345ff8dc None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 13 04:08:56.737911 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 13 04:08:56.737911 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 13 04:08:56.737911 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 13 04:08:56.737911 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 13 04:08:56.737911 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 13 04:08:56.737911 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 13 04:08:56.737911 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 13 04:08:56.737911 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 13 04:08:56.737911 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 13 04:08:56.737911 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 13 04:08:56.737911 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 13 04:08:56.737911 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 13 04:08:56.737911 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 13 04:08:56.737911 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 13 04:08:56.737911 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 13 04:08:56.738788 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 13 04:08:56.738788 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 13 04:08:56.738788 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 13 04:08:56.738788 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 13 04:08:56.738788 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 13 04:08:56.738788 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application return self.get_or_create( Oct 13 04:08:56.738788 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 13 04:08:56.738788 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application with Lock( Oct 13 04:08:56.738788 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 13 04:08:56.738788 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application return self._enter() Oct 13 04:08:56.738788 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 13 04:08:56.738788 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 13 04:08:56.738788 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 13 04:08:56.738788 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application return self.creator() Oct 13 04:08:56.738788 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 13 04:08:56.738788 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application created_value = creator( Oct 13 04:08:56.738788 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 13 04:08:56.739625 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 13 04:08:56.739625 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 13 04:08:56.739625 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application Oct 13 04:08:56.739787 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6138b34a-5946-4231-98f0-8c9b730f6bad None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:08:56.739853 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 197/400] 10.222.0.27 () {56 vars in 1108 bytes} [Thu Oct 13 04:08:56 2022] GET /identity/v3/domains/Default => generated 87 bytes in 160 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 13 04:08:56.739917 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6138b34a-5946-4231-98f0-8c9b730f6bad None admin] RBAC: Authorization granted {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:08:56.740247 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ad4ffb2c-081b-4dc1-9a96-53d4345ff8dc None admin] Authenticating user token {{(pid=81408) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:08:56.758593 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 204/401] 10.222.0.27 () {56 vars in 1148 bytes} [Thu Oct 13 04:08:56 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 158 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 13 04:08:56.759547 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6138b34a-5946-4231-98f0-8c9b730f6bad None admin] Authenticating user token {{(pid=81409) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:08:56.835821 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG dbcounter [-] [81409] Writing DB stats keystone:SELECT=100 {{(pid=81409) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:08:56.845006 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ad4ffb2c-081b-4dc1-9a96-53d4345ff8dc None admin] Validating token access rules against request {{(pid=81408) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:08:56.859032 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6138b34a-5946-4231-98f0-8c9b730f6bad None admin] Validating token access rules against request {{(pid=81409) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:08:56.878433 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dcbb2c8b-1609-4747-96fb-83811b37bc51 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81408) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:08:56.879030 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dcbb2c8b-1609-4747-96fb-83811b37bc51 None admin] REQUEST_METHOD: `GET` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:08:56.879305 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dcbb2c8b-1609-4747-96fb-83811b37bc51 None admin] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:08:56.879575 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dcbb2c8b-1609-4747-96fb-83811b37bc51 None admin] PATH_INFO: `/v3/domains` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:08:56.880212 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dcbb2c8b-1609-4747-96fb-83811b37bc51 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81408) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 13 04:08:56.880513 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dcbb2c8b-1609-4747-96fb-83811b37bc51 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:08:56.881899 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dcbb2c8b-1609-4747-96fb-83811b37bc51 None admin] RBAC: Authorization granted {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:08:56.890427 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG dbcounter [-] [81408] Writing DB stats keystone:SELECT=100 {{(pid=81408) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:08:56.892852 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-49032241-2e3c-4e54-923c-ffd3823c3a86 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81409) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:08:56.893456 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49032241-2e3c-4e54-923c-ffd3823c3a86 None admin] REQUEST_METHOD: `POST` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:08:56.893735 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49032241-2e3c-4e54-923c-ffd3823c3a86 None admin] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:08:56.894003 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49032241-2e3c-4e54-923c-ffd3823c3a86 None admin] PATH_INFO: `/v3/users` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:08:56.894678 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-49032241-2e3c-4e54-923c-ffd3823c3a86 None admin] RBAC: Authorizing `identity:create_user()` {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:08:56.896103 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-49032241-2e3c-4e54-923c-ffd3823c3a86 None admin] RBAC: Authorization granted {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:08:56.898701 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 198/402] 10.222.0.27 () {56 vars in 1109 bytes} [Thu Oct 13 04:08:56 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 159 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 13 04:08:56.899438 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dcbb2c8b-1609-4747-96fb-83811b37bc51 None admin] Authenticating user token {{(pid=81408) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:08:56.917447 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 205/403] 10.222.0.27 () {60 vars in 1132 bytes} [Thu Oct 13 04:08:56 2022] POST /identity/v3/users => generated 247 bytes in 158 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 13 04:08:56.918333 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-49032241-2e3c-4e54-923c-ffd3823c3a86 None admin] Authenticating user token {{(pid=81409) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:08:57.010391 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dcbb2c8b-1609-4747-96fb-83811b37bc51 None admin] Validating token access rules against request {{(pid=81408) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:08:57.034846 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-49032241-2e3c-4e54-923c-ffd3823c3a86 None admin] Validating token access rules against request {{(pid=81409) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:08:57.047784 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3e60e766-08c1-431d-a2ea-781023cfcdf2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81408) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:08:57.048432 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e60e766-08c1-431d-a2ea-781023cfcdf2 None admin] REQUEST_METHOD: `GET` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:08:57.048708 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e60e766-08c1-431d-a2ea-781023cfcdf2 None admin] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:08:57.048976 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e60e766-08c1-431d-a2ea-781023cfcdf2 None admin] PATH_INFO: `/v3/projects/service` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:08:57.056957 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3e60e766-08c1-431d-a2ea-781023cfcdf2 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:08:57.058552 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3e60e766-08c1-431d-a2ea-781023cfcdf2 None admin] RBAC: Authorization granted {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:08:57.070579 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application [None req-3e60e766-08c1-431d-a2ea-781023cfcdf2 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Oct 13 04:08:57.070579 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 13 04:08:57.070579 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 13 04:08:57.070579 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 13 04:08:57.070579 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 13 04:08:57.070579 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 13 04:08:57.070579 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 13 04:08:57.070579 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 13 04:08:57.070579 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 13 04:08:57.070579 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 13 04:08:57.070579 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 13 04:08:57.070579 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 13 04:08:57.070579 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Oct 13 04:08:57.070579 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application return self._get_project(project_id) Oct 13 04:08:57.070579 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Oct 13 04:08:57.071944 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Oct 13 04:08:57.071944 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 13 04:08:57.071944 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 13 04:08:57.071944 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 13 04:08:57.071944 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 13 04:08:57.071944 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 13 04:08:57.071944 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application return self.get_or_create( Oct 13 04:08:57.071944 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 13 04:08:57.071944 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application with Lock( Oct 13 04:08:57.071944 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 13 04:08:57.071944 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application return self._enter() Oct 13 04:08:57.071944 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 13 04:08:57.071944 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 13 04:08:57.071944 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 13 04:08:57.071944 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application return self.creator() Oct 13 04:08:57.071944 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 13 04:08:57.071944 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application created_value = creator( Oct 13 04:08:57.072919 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Oct 13 04:08:57.072919 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Oct 13 04:08:57.072919 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Oct 13 04:08:57.072919 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Oct 13 04:08:57.072919 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Oct 13 04:08:57.072919 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Oct 13 04:08:57.072919 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Oct 13 04:08:57.072919 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application Oct 13 04:08:57.072919 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 199/404] 10.222.0.27 () {56 vars in 1111 bytes} [Thu Oct 13 04:08:56 2022] GET /identity/v3/projects/service => generated 88 bytes in 173 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 13 04:08:57.076718 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-10b10d4c-b25b-4870-a708-0859ceefca04 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81409) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:08:57.077435 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10b10d4c-b25b-4870-a708-0859ceefca04 None admin] REQUEST_METHOD: `GET` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:08:57.077713 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10b10d4c-b25b-4870-a708-0859ceefca04 None admin] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:08:57.077952 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3e60e766-08c1-431d-a2ea-781023cfcdf2 None admin] Authenticating user token {{(pid=81408) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:08:57.078045 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10b10d4c-b25b-4870-a708-0859ceefca04 None admin] PATH_INFO: `/v3/domains` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:08:57.078659 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-10b10d4c-b25b-4870-a708-0859ceefca04 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81409) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 13 04:08:57.078961 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-10b10d4c-b25b-4870-a708-0859ceefca04 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:08:57.080350 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-10b10d4c-b25b-4870-a708-0859ceefca04 None admin] RBAC: Authorization granted {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:08:57.082176 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG dbcounter [-] [81409] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81409) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:08:57.096005 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 206/405] 10.222.0.27 () {56 vars in 1109 bytes} [Thu Oct 13 04:08:56 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 178 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 13 04:08:57.101121 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-10b10d4c-b25b-4870-a708-0859ceefca04 None admin] Authenticating user token {{(pid=81409) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:08:57.151399 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG dbcounter [-] [81408] Writing DB stats keystone:SELECT=100 {{(pid=81408) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:08:57.194640 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3e60e766-08c1-431d-a2ea-781023cfcdf2 None admin] Validating token access rules against request {{(pid=81408) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:08:57.217602 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-10b10d4c-b25b-4870-a708-0859ceefca04 None admin] Validating token access rules against request {{(pid=81409) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:08:57.231550 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8d21edee-30b7-47c9-8bba-3b56f4798092 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81408) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:08:57.232214 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d21edee-30b7-47c9-8bba-3b56f4798092 None admin] REQUEST_METHOD: `PUT` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:08:57.232749 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d21edee-30b7-47c9-8bba-3b56f4798092 None admin] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:08:57.233028 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d21edee-30b7-47c9-8bba-3b56f4798092 None admin] PATH_INFO: `/v3/projects/dcec0431fb6849499d6aec35a234daa6/users/cb740ed7f7684b079e45f887be83f795/roles/4cfff1d3d68e49c9b04231a19cf76ede` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:08:57.251680 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8d21edee-30b7-47c9-8bba-3b56f4798092 None admin] RBAC: Authorizing `identity:create_grant(project_id=dcec0431fb6849499d6aec35a234daa6, user_id=cb740ed7f7684b079e45f887be83f795, role_id=4cfff1d3d68e49c9b04231a19cf76ede)` {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:08:57.253151 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8d21edee-30b7-47c9-8bba-3b56f4798092 None admin] RBAC: Authorization granted {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:08:57.253992 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ceb224d8-5d42-44b9-9551-0b81c3357247 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81409) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:08:57.254664 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ceb224d8-5d42-44b9-9551-0b81c3357247 None admin] REQUEST_METHOD: `GET` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:08:57.254941 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ceb224d8-5d42-44b9-9551-0b81c3357247 None admin] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:08:57.255375 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ceb224d8-5d42-44b9-9551-0b81c3357247 None admin] PATH_INFO: `/v3/users/neutron` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:08:57.263817 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ceb224d8-5d42-44b9-9551-0b81c3357247 None admin] RBAC: Authorizing `identity:get_user(user_id=neutron)` {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:08:57.265410 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ceb224d8-5d42-44b9-9551-0b81c3357247 None admin] RBAC: Authorization granted {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:08:57.267676 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 200/406] 10.222.0.27 () {58 vars in 1439 bytes} [Thu Oct 13 04:08:57 2022] PUT /identity/v3/projects/dcec0431fb6849499d6aec35a234daa6/users/cb740ed7f7684b079e45f887be83f795/roles/4cfff1d3d68e49c9b04231a19cf76ede => generated 0 bytes in 195 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 13 04:08:57.268534 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8d21edee-30b7-47c9-8bba-3b56f4798092 None admin] Authenticating user token {{(pid=81408) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:08:57.301155 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application [None req-ceb224d8-5d42-44b9-9551-0b81c3357247 None admin] Could not find user: neutron.: keystone.exception.UserNotFound: Could not find user: neutron. Oct 13 04:08:57.301155 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 13 04:08:57.301155 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 13 04:08:57.301155 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 13 04:08:57.301155 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 13 04:08:57.301155 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 13 04:08:57.301155 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 13 04:08:57.301155 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 13 04:08:57.301155 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 13 04:08:57.301155 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 13 04:08:57.301155 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 13 04:08:57.301155 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 13 04:08:57.301155 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Oct 13 04:08:57.301155 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application return self._get_user(user_id) Oct 13 04:08:57.301155 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Oct 13 04:08:57.301155 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Oct 13 04:08:57.302099 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 13 04:08:57.302099 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 13 04:08:57.302099 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Oct 13 04:08:57.302099 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 13 04:08:57.302099 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Oct 13 04:08:57.302099 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 13 04:08:57.302099 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 13 04:08:57.302099 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 13 04:08:57.302099 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 13 04:08:57.302099 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application return self.get_or_create( Oct 13 04:08:57.302099 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 13 04:08:57.302099 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application with Lock( Oct 13 04:08:57.302099 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 13 04:08:57.302099 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application return self._enter() Oct 13 04:08:57.302099 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 13 04:08:57.302099 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 13 04:08:57.302099 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 13 04:08:57.302099 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application return self.creator() Oct 13 04:08:57.303036 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 13 04:08:57.303036 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application created_value = creator( Oct 13 04:08:57.303036 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Oct 13 04:08:57.303036 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Oct 13 04:08:57.303036 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Oct 13 04:08:57.303036 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Oct 13 04:08:57.303036 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Oct 13 04:08:57.303036 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Oct 13 04:08:57.303036 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: neutron. Oct 13 04:08:57.303036 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application Oct 13 04:08:57.303036 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 207/407] 10.222.0.27 () {56 vars in 1102 bytes} [Thu Oct 13 04:08:57 2022] GET /identity/v3/users/neutron => generated 85 bytes in 206 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 13 04:08:57.303589 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ceb224d8-5d42-44b9-9551-0b81c3357247 None admin] Authenticating user token {{(pid=81409) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:08:57.392194 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG dbcounter [-] [81409] Writing DB stats keystone:SELECT=100 {{(pid=81409) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:08:57.409406 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8d21edee-30b7-47c9-8bba-3b56f4798092 None admin] Validating token access rules against request {{(pid=81408) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:08:57.423478 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG dbcounter [-] [81408] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81408) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:08:57.453846 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-08635f07-f795-448e-9c21-1c430c592169 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81408) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:08:57.454558 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08635f07-f795-448e-9c21-1c430c592169 None admin] REQUEST_METHOD: `GET` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:08:57.454831 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08635f07-f795-448e-9c21-1c430c592169 None admin] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:08:57.455103 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08635f07-f795-448e-9c21-1c430c592169 None admin] PATH_INFO: `/v3/projects` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:08:57.455803 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-08635f07-f795-448e-9c21-1c430c592169 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81408) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 13 04:08:57.456107 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-08635f07-f795-448e-9c21-1c430c592169 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:08:57.457870 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-08635f07-f795-448e-9c21-1c430c592169 None admin] RBAC: Authorization granted {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:08:57.460816 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ceb224d8-5d42-44b9-9551-0b81c3357247 None admin] Validating token access rules against request {{(pid=81409) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:08:57.480211 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 201/408] 10.222.0.27 () {56 vars in 1148 bytes} [Thu Oct 13 04:08:57 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 212 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 13 04:08:57.481094 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-08635f07-f795-448e-9c21-1c430c592169 None admin] Authenticating user token {{(pid=81408) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:08:57.505182 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e7c777b3-530f-4c46-a81d-47bcffa1961b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81409) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:08:57.506192 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7c777b3-530f-4c46-a81d-47bcffa1961b None admin] REQUEST_METHOD: `GET` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:08:57.506669 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7c777b3-530f-4c46-a81d-47bcffa1961b None admin] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:08:57.506944 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7c777b3-530f-4c46-a81d-47bcffa1961b None admin] PATH_INFO: `/v3/projects/service` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:08:57.517206 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e7c777b3-530f-4c46-a81d-47bcffa1961b None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:08:57.518857 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e7c777b3-530f-4c46-a81d-47bcffa1961b None admin] RBAC: Authorization granted {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:08:57.535116 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application [None req-e7c777b3-530f-4c46-a81d-47bcffa1961b None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Oct 13 04:08:57.535116 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 13 04:08:57.535116 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 13 04:08:57.535116 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 13 04:08:57.535116 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 13 04:08:57.535116 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 13 04:08:57.535116 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 13 04:08:57.535116 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 13 04:08:57.535116 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 13 04:08:57.535116 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 13 04:08:57.535116 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 13 04:08:57.535116 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 13 04:08:57.535116 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Oct 13 04:08:57.535116 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application return self._get_project(project_id) Oct 13 04:08:57.535116 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Oct 13 04:08:57.536101 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Oct 13 04:08:57.536101 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 13 04:08:57.536101 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 13 04:08:57.536101 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 13 04:08:57.536101 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 13 04:08:57.536101 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 13 04:08:57.536101 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application return self.get_or_create( Oct 13 04:08:57.536101 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 13 04:08:57.536101 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application with Lock( Oct 13 04:08:57.536101 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 13 04:08:57.536101 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application return self._enter() Oct 13 04:08:57.536101 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 13 04:08:57.536101 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 13 04:08:57.536101 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 13 04:08:57.536101 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application return self.creator() Oct 13 04:08:57.536101 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 13 04:08:57.536101 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application created_value = creator( Oct 13 04:08:57.537378 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Oct 13 04:08:57.537378 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Oct 13 04:08:57.537378 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Oct 13 04:08:57.537378 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Oct 13 04:08:57.537378 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Oct 13 04:08:57.537378 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Oct 13 04:08:57.537378 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Oct 13 04:08:57.537378 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application Oct 13 04:08:57.537378 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 208/409] 10.222.0.27 () {56 vars in 1111 bytes} [Thu Oct 13 04:08:57 2022] GET /identity/v3/projects/service => generated 88 bytes in 234 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 13 04:08:57.537980 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e7c777b3-530f-4c46-a81d-47bcffa1961b None admin] Authenticating user token {{(pid=81409) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:08:57.593509 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-08635f07-f795-448e-9c21-1c430c592169 None admin] Validating token access rules against request {{(pid=81408) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:08:57.640893 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f3181804-8061-4c30-9c70-03e4a0ec7886 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81408) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:08:57.642051 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3181804-8061-4c30-9c70-03e4a0ec7886 None admin] REQUEST_METHOD: `GET` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:08:57.642583 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3181804-8061-4c30-9c70-03e4a0ec7886 None admin] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:08:57.642854 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3181804-8061-4c30-9c70-03e4a0ec7886 None admin] PATH_INFO: `/v3/users` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:08:57.643600 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f3181804-8061-4c30-9c70-03e4a0ec7886 None admin] RBAC: Adding query filter params (domain_id=default, name=neutron) {{(pid=81408) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 13 04:08:57.643908 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f3181804-8061-4c30-9c70-03e4a0ec7886 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:08:57.645362 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f3181804-8061-4c30-9c70-03e4a0ec7886 None admin] RBAC: Authorization granted {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:08:57.666768 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e7c777b3-530f-4c46-a81d-47bcffa1961b None admin] Validating token access rules against request {{(pid=81409) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:08:57.671385 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 202/410] 10.222.0.27 () {56 vars in 1139 bytes} [Thu Oct 13 04:08:57 2022] GET /identity/v3/users?domain_id=default&name=neutron => generated 374 bytes in 191 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 13 04:08:57.677320 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f3181804-8061-4c30-9c70-03e4a0ec7886 None admin] Authenticating user token {{(pid=81408) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:08:57.684796 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG dbcounter [-] [81409] Writing DB stats keystone:SELECT=100 {{(pid=81409) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:08:57.708583 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0be85b3e-96a5-497f-a62d-98bb451a04ec None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81409) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:08:57.709342 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0be85b3e-96a5-497f-a62d-98bb451a04ec None admin] REQUEST_METHOD: `PUT` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:08:57.709737 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0be85b3e-96a5-497f-a62d-98bb451a04ec None admin] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:08:57.710079 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0be85b3e-96a5-497f-a62d-98bb451a04ec None admin] PATH_INFO: `/v3/projects/dcec0431fb6849499d6aec35a234daa6/users/6141b8a6c3d54e179d6879bb019f4d12/roles/4cfff1d3d68e49c9b04231a19cf76ede` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:08:57.717628 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG dbcounter [-] [81408] Writing DB stats keystone:SELECT=100 {{(pid=81408) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:08:57.731669 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0be85b3e-96a5-497f-a62d-98bb451a04ec None admin] RBAC: Authorizing `identity:create_grant(project_id=dcec0431fb6849499d6aec35a234daa6, user_id=6141b8a6c3d54e179d6879bb019f4d12, role_id=4cfff1d3d68e49c9b04231a19cf76ede)` {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:08:57.733322 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0be85b3e-96a5-497f-a62d-98bb451a04ec None admin] RBAC: Authorization granted {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:08:57.754236 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 209/411] 10.222.0.27 () {58 vars in 1439 bytes} [Thu Oct 13 04:08:57 2022] PUT /identity/v3/projects/dcec0431fb6849499d6aec35a234daa6/users/6141b8a6c3d54e179d6879bb019f4d12/roles/4cfff1d3d68e49c9b04231a19cf76ede => generated 0 bytes in 211 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 13 04:08:57.754236 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0be85b3e-96a5-497f-a62d-98bb451a04ec None admin] Authenticating user token {{(pid=81409) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:08:57.868872 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f3181804-8061-4c30-9c70-03e4a0ec7886 None admin] Validating token access rules against request {{(pid=81408) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:08:57.889821 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0be85b3e-96a5-497f-a62d-98bb451a04ec None admin] Validating token access rules against request {{(pid=81409) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:08:57.932890 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-acefd3b9-f1eb-4fb2-937d-cd61aa4619ab None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81409) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:08:57.933649 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-acefd3b9-f1eb-4fb2-937d-cd61aa4619ab None admin] REQUEST_METHOD: `GET` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:08:57.933931 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-acefd3b9-f1eb-4fb2-937d-cd61aa4619ab None admin] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:08:57.934220 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-acefd3b9-f1eb-4fb2-937d-cd61aa4619ab None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:08:57.946357 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-acefd3b9-f1eb-4fb2-937d-cd61aa4619ab None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:08:57.947888 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-acefd3b9-f1eb-4fb2-937d-cd61aa4619ab None admin] RBAC: Authorization granted {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:08:57.949402 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-85346165-8abe-498e-aec4-75b6db5bada0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81408) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:08:57.950136 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85346165-8abe-498e-aec4-75b6db5bada0 None admin] REQUEST_METHOD: `GET` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:08:57.950475 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85346165-8abe-498e-aec4-75b6db5bada0 None admin] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:08:57.950815 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85346165-8abe-498e-aec4-75b6db5bada0 None admin] PATH_INFO: `/v3/projects` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:08:57.951520 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-85346165-8abe-498e-aec4-75b6db5bada0 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81408) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 13 04:08:57.951824 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-85346165-8abe-498e-aec4-75b6db5bada0 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:08:57.953266 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-85346165-8abe-498e-aec4-75b6db5bada0 None admin] RBAC: Authorization granted {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:08:57.965213 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application [None req-acefd3b9-f1eb-4fb2-937d-cd61aa4619ab None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 13 04:08:57.965213 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 13 04:08:57.965213 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 13 04:08:57.965213 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 13 04:08:57.965213 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 13 04:08:57.965213 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 13 04:08:57.965213 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 13 04:08:57.965213 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 13 04:08:57.965213 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 13 04:08:57.965213 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 13 04:08:57.965213 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 13 04:08:57.965213 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 13 04:08:57.965213 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 13 04:08:57.965213 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 13 04:08:57.965213 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 13 04:08:57.965213 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 13 04:08:57.966168 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 13 04:08:57.966168 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 13 04:08:57.966168 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 13 04:08:57.966168 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 13 04:08:57.966168 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 13 04:08:57.966168 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application return self.get_or_create( Oct 13 04:08:57.966168 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 13 04:08:57.966168 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application with Lock( Oct 13 04:08:57.966168 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 13 04:08:57.966168 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application return self._enter() Oct 13 04:08:57.966168 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 13 04:08:57.966168 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 13 04:08:57.966168 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 13 04:08:57.966168 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application return self.creator() Oct 13 04:08:57.966168 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 13 04:08:57.966168 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application created_value = creator( Oct 13 04:08:57.966168 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 13 04:08:57.967604 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 13 04:08:57.967604 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 13 04:08:57.967604 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application Oct 13 04:08:57.967604 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 210/412] 10.222.0.27 () {56 vars in 1108 bytes} [Thu Oct 13 04:08:57 2022] GET /identity/v3/domains/Default => generated 87 bytes in 217 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 13 04:08:57.972781 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-acefd3b9-f1eb-4fb2-937d-cd61aa4619ab None admin] Authenticating user token {{(pid=81409) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:08:57.979913 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 203/413] 10.222.0.27 () {56 vars in 1148 bytes} [Thu Oct 13 04:08:57 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 303 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 13 04:08:57.986938 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-85346165-8abe-498e-aec4-75b6db5bada0 None admin] Authenticating user token {{(pid=81408) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:08:58.000373 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG dbcounter [-] [81409] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81409) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:08:58.121725 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG dbcounter [-] [81408] Writing DB stats keystone:SELECT=100 {{(pid=81408) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:08:58.125920 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-acefd3b9-f1eb-4fb2-937d-cd61aa4619ab None admin] Validating token access rules against request {{(pid=81409) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:08:58.135997 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-85346165-8abe-498e-aec4-75b6db5bada0 None admin] Validating token access rules against request {{(pid=81408) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:08:58.172310 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-26fa2b54-828f-476f-99b6-0412e32287c9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81409) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:08:58.172939 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26fa2b54-828f-476f-99b6-0412e32287c9 None admin] REQUEST_METHOD: `GET` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:08:58.173199 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26fa2b54-828f-476f-99b6-0412e32287c9 None admin] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:08:58.173474 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26fa2b54-828f-476f-99b6-0412e32287c9 None admin] PATH_INFO: `/v3/domains` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:08:58.174116 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-26fa2b54-828f-476f-99b6-0412e32287c9 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81409) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 13 04:08:58.174448 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-26fa2b54-828f-476f-99b6-0412e32287c9 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:08:58.175810 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-26fa2b54-828f-476f-99b6-0412e32287c9 None admin] RBAC: Authorization granted {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:08:58.177059 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-75979996-161f-400f-82ba-871940d87329 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81408) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:08:58.178000 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75979996-161f-400f-82ba-871940d87329 None admin] REQUEST_METHOD: `PUT` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:08:58.178534 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75979996-161f-400f-82ba-871940d87329 None admin] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:08:58.178998 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75979996-161f-400f-82ba-871940d87329 None admin] PATH_INFO: `/v3/projects/dcec0431fb6849499d6aec35a234daa6/users/a6700f49b86543f6a5b92cad8faaba42/roles/4cfff1d3d68e49c9b04231a19cf76ede` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:08:58.195122 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 211/414] 10.222.0.27 () {56 vars in 1109 bytes} [Thu Oct 13 04:08:57 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 223 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 13 04:08:58.198557 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-75979996-161f-400f-82ba-871940d87329 None admin] RBAC: Authorizing `identity:create_grant(project_id=dcec0431fb6849499d6aec35a234daa6, user_id=a6700f49b86543f6a5b92cad8faaba42, role_id=4cfff1d3d68e49c9b04231a19cf76ede)` {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:08:58.200306 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-75979996-161f-400f-82ba-871940d87329 None admin] RBAC: Authorization granted {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:08:58.201382 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-26fa2b54-828f-476f-99b6-0412e32287c9 None admin] Authenticating user token {{(pid=81409) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:08:58.235558 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 204/415] 10.222.0.27 () {58 vars in 1439 bytes} [Thu Oct 13 04:08:57 2022] PUT /identity/v3/projects/dcec0431fb6849499d6aec35a234daa6/users/a6700f49b86543f6a5b92cad8faaba42/roles/4cfff1d3d68e49c9b04231a19cf76ede => generated 0 bytes in 249 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 13 04:08:58.305739 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG dbcounter [-] [81409] Writing DB stats keystone:SELECT=100 {{(pid=81409) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:08:58.342104 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-26fa2b54-828f-476f-99b6-0412e32287c9 None admin] Validating token access rules against request {{(pid=81409) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:08:58.403073 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fd948ab9-66ee-4698-bd6f-371e671dc047 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81409) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:08:58.404484 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd948ab9-66ee-4698-bd6f-371e671dc047 None admin] REQUEST_METHOD: `GET` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:08:58.404950 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd948ab9-66ee-4698-bd6f-371e671dc047 None admin] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:08:58.405406 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd948ab9-66ee-4698-bd6f-371e671dc047 None admin] PATH_INFO: `/v3/projects/service` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:08:58.431683 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fd948ab9-66ee-4698-bd6f-371e671dc047 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:08:58.433537 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fd948ab9-66ee-4698-bd6f-371e671dc047 None admin] RBAC: Authorization granted {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:08:58.447469 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application [None req-fd948ab9-66ee-4698-bd6f-371e671dc047 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Oct 13 04:08:58.447469 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 13 04:08:58.447469 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 13 04:08:58.447469 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 13 04:08:58.447469 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 13 04:08:58.447469 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 13 04:08:58.447469 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 13 04:08:58.447469 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 13 04:08:58.447469 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 13 04:08:58.447469 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 13 04:08:58.447469 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 13 04:08:58.447469 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 13 04:08:58.447469 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Oct 13 04:08:58.447469 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application return self._get_project(project_id) Oct 13 04:08:58.447469 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Oct 13 04:08:58.448337 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Oct 13 04:08:58.448337 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 13 04:08:58.448337 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 13 04:08:58.448337 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 13 04:08:58.448337 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 13 04:08:58.448337 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 13 04:08:58.448337 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application return self.get_or_create( Oct 13 04:08:58.448337 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 13 04:08:58.448337 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application with Lock( Oct 13 04:08:58.448337 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 13 04:08:58.448337 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application return self._enter() Oct 13 04:08:58.448337 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 13 04:08:58.448337 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 13 04:08:58.448337 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 13 04:08:58.448337 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application return self.creator() Oct 13 04:08:58.448337 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 13 04:08:58.448337 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application created_value = creator( Oct 13 04:08:58.449579 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Oct 13 04:08:58.449579 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Oct 13 04:08:58.449579 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Oct 13 04:08:58.449579 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Oct 13 04:08:58.449579 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Oct 13 04:08:58.449579 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Oct 13 04:08:58.449579 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Oct 13 04:08:58.449579 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application Oct 13 04:08:58.449579 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 212/416] 10.222.0.27 () {56 vars in 1111 bytes} [Thu Oct 13 04:08:58 2022] GET /identity/v3/projects/service => generated 88 bytes in 248 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 13 04:08:58.454579 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-75979996-161f-400f-82ba-871940d87329 None admin] Authenticating user token {{(pid=81408) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:08:58.557542 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-75979996-161f-400f-82ba-871940d87329 None admin] Validating token access rules against request {{(pid=81408) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:08:58.593264 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fc9870ec-79b7-4290-97d7-e18f9ef24b85 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81408) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:08:58.593869 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc9870ec-79b7-4290-97d7-e18f9ef24b85 None admin] REQUEST_METHOD: `GET` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:08:58.594144 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc9870ec-79b7-4290-97d7-e18f9ef24b85 None admin] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:08:58.594447 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc9870ec-79b7-4290-97d7-e18f9ef24b85 None admin] PATH_INFO: `/v3/projects` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:08:58.595172 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fc9870ec-79b7-4290-97d7-e18f9ef24b85 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81408) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 13 04:08:58.595443 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fc9870ec-79b7-4290-97d7-e18f9ef24b85 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:08:58.596889 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fc9870ec-79b7-4290-97d7-e18f9ef24b85 None admin] RBAC: Authorization granted {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:08:58.604669 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG dbcounter [-] [81408] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81408) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:08:58.610655 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 205/417] 10.222.0.27 () {56 vars in 1148 bytes} [Thu Oct 13 04:08:58 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 156 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 13 04:08:58.616425 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fd948ab9-66ee-4698-bd6f-371e671dc047 None admin] Authenticating user token {{(pid=81409) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:08:58.746584 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fd948ab9-66ee-4698-bd6f-371e671dc047 None admin] Validating token access rules against request {{(pid=81409) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:08:58.791688 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-49e7d62c-51f8-489f-b378-e2b5bf13e332 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81409) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:08:58.792363 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49e7d62c-51f8-489f-b378-e2b5bf13e332 None admin] REQUEST_METHOD: `PUT` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:08:58.792644 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49e7d62c-51f8-489f-b378-e2b5bf13e332 None admin] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:08:58.792917 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49e7d62c-51f8-489f-b378-e2b5bf13e332 None admin] PATH_INFO: `/v3/projects/dcec0431fb6849499d6aec35a234daa6/users/d8812c3974f84871bb2d761cf0016257/roles/4cfff1d3d68e49c9b04231a19cf76ede` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:08:58.799325 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG dbcounter [-] [81409] Writing DB stats keystone:SELECT=100 {{(pid=81409) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:08:58.815964 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-49e7d62c-51f8-489f-b378-e2b5bf13e332 None admin] RBAC: Authorizing `identity:create_grant(project_id=dcec0431fb6849499d6aec35a234daa6, user_id=d8812c3974f84871bb2d761cf0016257, role_id=4cfff1d3d68e49c9b04231a19cf76ede)` {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:08:58.817428 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-49e7d62c-51f8-489f-b378-e2b5bf13e332 None admin] RBAC: Authorization granted {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:08:58.842066 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 213/418] 10.222.0.27 () {58 vars in 1439 bytes} [Thu Oct 13 04:08:58 2022] PUT /identity/v3/projects/dcec0431fb6849499d6aec35a234daa6/users/d8812c3974f84871bb2d761cf0016257/roles/4cfff1d3d68e49c9b04231a19cf76ede => generated 0 bytes in 226 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 13 04:08:59.422343 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00ea369d-48ad-4497-b34d-667dccecbab3 None None] REQUEST_METHOD: `GET` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:08:59.422902 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00ea369d-48ad-4497-b34d-667dccecbab3 None None] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:08:59.423369 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00ea369d-48ad-4497-b34d-667dccecbab3 None None] PATH_INFO: `/` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:08:59.425190 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 206/419] 10.222.0.27 () {54 vars in 927 bytes} [Thu Oct 13 04:08:59 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 13 04:08:59.430464 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e1cdf97-eea5-404f-a91e-66fc02b95eba None None] REQUEST_METHOD: `POST` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:08:59.430937 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e1cdf97-eea5-404f-a91e-66fc02b95eba None None] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:08:59.431011 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e1cdf97-eea5-404f-a91e-66fc02b95eba None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:08:59.456640 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.auth.core [None req-1e1cdf97-eea5-404f-a91e-66fc02b95eba None None] MFA Rules not processed for user `0e2d979563d6421995451a0e86bf3022`. Rule list: `[]` (Enabled: `True`). {{(pid=81409) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 13 04:08:59.457867 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.fernet_utils [None req-1e1cdf97-eea5-404f-a91e-66fc02b95eba None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81409) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:08:59.552787 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 214/420] 10.222.0.27 () {58 vars in 1024 bytes} [Thu Oct 13 04:08:59 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 123 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 13 04:08:59.879860 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f18e1b0d-3b65-4d86-9d2f-6559121d88aa None None] REQUEST_METHOD: `GET` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:08:59.880362 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f18e1b0d-3b65-4d86-9d2f-6559121d88aa None None] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:08:59.880520 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f18e1b0d-3b65-4d86-9d2f-6559121d88aa None None] PATH_INFO: `/` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:08:59.881984 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 207/421] 10.222.0.27 () {54 vars in 927 bytes} [Thu Oct 13 04:08:59 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 13 04:08:59.899061 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f52839e-54bb-40a3-b05f-5e4d586bd7a9 None None] REQUEST_METHOD: `POST` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:08:59.899539 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f18e1b0d-3b65-4d86-9d2f-6559121d88aa None None] Authenticating user token {{(pid=81408) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:08:59.899674 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f52839e-54bb-40a3-b05f-5e4d586bd7a9 None None] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:08:59.899940 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f52839e-54bb-40a3-b05f-5e4d586bd7a9 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:08:59.931641 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.auth.core [None req-0f52839e-54bb-40a3-b05f-5e4d586bd7a9 None None] MFA Rules not processed for user `0e2d979563d6421995451a0e86bf3022`. Rule list: `[]` (Enabled: `True`). {{(pid=81409) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 13 04:08:59.932844 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.fernet_utils [None req-0f52839e-54bb-40a3-b05f-5e4d586bd7a9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81409) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:09:00.007709 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f18e1b0d-3b65-4d86-9d2f-6559121d88aa None None] Validating token access rules against request {{(pid=81408) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:09:00.027392 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 215/422] 10.222.0.27 () {58 vars in 1024 bytes} [Thu Oct 13 04:08:59 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 129 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 13 04:09:00.077782 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-de469e9b-5571-443c-a2f8-1e34913de792 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81408) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:09:00.078458 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de469e9b-5571-443c-a2f8-1e34913de792 None admin] REQUEST_METHOD: `GET` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:09:00.078738 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de469e9b-5571-443c-a2f8-1e34913de792 None admin] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:09:00.079014 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de469e9b-5571-443c-a2f8-1e34913de792 None admin] PATH_INFO: `/v3/roles/service` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:09:00.098858 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-de469e9b-5571-443c-a2f8-1e34913de792 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 13 04:09:00.105258 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application [None req-de469e9b-5571-443c-a2f8-1e34913de792 None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Oct 13 04:09:00.105258 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 13 04:09:00.105258 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 13 04:09:00.105258 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 13 04:09:00.105258 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 13 04:09:00.105258 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 13 04:09:00.105258 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 13 04:09:00.105258 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 13 04:09:00.105258 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 13 04:09:00.105258 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 13 04:09:00.105258 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 13 04:09:00.105258 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 13 04:09:00.105258 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 13 04:09:00.105258 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 13 04:09:00.105258 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 13 04:09:00.105258 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 13 04:09:00.106115 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 13 04:09:00.106115 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 13 04:09:00.106115 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 13 04:09:00.106115 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 13 04:09:00.106115 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 13 04:09:00.106115 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 13 04:09:00.106115 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 13 04:09:00.106115 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 13 04:09:00.106115 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 13 04:09:00.106115 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application return self.get_or_create( Oct 13 04:09:00.106115 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 13 04:09:00.106115 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application with Lock( Oct 13 04:09:00.106115 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 13 04:09:00.106115 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application return self._enter() Oct 13 04:09:00.106115 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 13 04:09:00.106115 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 13 04:09:00.107523 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 13 04:09:00.107523 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application return self.creator() Oct 13 04:09:00.107523 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 13 04:09:00.107523 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application created_value = creator( Oct 13 04:09:00.107523 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 13 04:09:00.107523 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 13 04:09:00.107523 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 13 04:09:00.107523 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 13 04:09:00.107523 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 13 04:09:00.107523 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 13 04:09:00.107523 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Oct 13 04:09:00.107523 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application Oct 13 04:09:00.107523 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 208/423] 10.222.0.27 () {56 vars in 1102 bytes} [Thu Oct 13 04:08:59 2022] GET /identity/v3/roles/service => generated 85 bytes in 208 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Oct 13 04:09:00.112215 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0f52839e-54bb-40a3-b05f-5e4d586bd7a9 None None] Authenticating user token {{(pid=81409) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:09:00.162682 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG dbcounter [-] [81409] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81409) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:09:00.219570 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0f52839e-54bb-40a3-b05f-5e4d586bd7a9 None None] Validating token access rules against request {{(pid=81409) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:09:00.256002 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-58970ad3-24d1-4a91-9900-6190f87389eb None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81409) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:09:00.256611 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58970ad3-24d1-4a91-9900-6190f87389eb None admin] REQUEST_METHOD: `GET` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:09:00.256886 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58970ad3-24d1-4a91-9900-6190f87389eb None admin] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:09:00.257159 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58970ad3-24d1-4a91-9900-6190f87389eb None admin] PATH_INFO: `/v3/roles` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:09:00.257820 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-58970ad3-24d1-4a91-9900-6190f87389eb None admin] RBAC: Adding query filter params (name=service) {{(pid=81409) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 13 04:09:00.258123 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-58970ad3-24d1-4a91-9900-6190f87389eb None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:09:00.259788 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-58970ad3-24d1-4a91-9900-6190f87389eb None admin] RBAC: Authorization granted {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:09:00.268334 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 216/424] 10.222.0.27 () {56 vars in 1103 bytes} [Thu Oct 13 04:09:00 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 156 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 13 04:09:00.275745 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-de469e9b-5571-443c-a2f8-1e34913de792 None admin] Authenticating user token {{(pid=81408) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:09:00.368114 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG dbcounter [-] [81408] Writing DB stats keystone:SELECT=100 {{(pid=81408) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:09:00.370363 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-58970ad3-24d1-4a91-9900-6190f87389eb None admin] Authenticating user token {{(pid=81409) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:09:00.443998 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-de469e9b-5571-443c-a2f8-1e34913de792 None admin] Validating token access rules against request {{(pid=81408) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:09:00.501294 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-58970ad3-24d1-4a91-9900-6190f87389eb None admin] Validating token access rules against request {{(pid=81409) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:09:00.506730 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-35df5e91-73f9-4278-942e-66736836fdb6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81408) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:09:00.507696 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35df5e91-73f9-4278-942e-66736836fdb6 None admin] REQUEST_METHOD: `GET` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:09:00.508110 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35df5e91-73f9-4278-942e-66736836fdb6 None admin] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:09:00.508619 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35df5e91-73f9-4278-942e-66736836fdb6 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:09:00.521277 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-35df5e91-73f9-4278-942e-66736836fdb6 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:09:00.523030 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-35df5e91-73f9-4278-942e-66736836fdb6 None admin] RBAC: Authorization granted {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:09:00.530240 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG dbcounter [-] [81409] Writing DB stats keystone:SELECT=100 {{(pid=81409) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:09:00.549876 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application [None req-35df5e91-73f9-4278-942e-66736836fdb6 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 13 04:09:00.549876 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 13 04:09:00.549876 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 13 04:09:00.549876 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 13 04:09:00.549876 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 13 04:09:00.549876 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 13 04:09:00.549876 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 13 04:09:00.549876 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 13 04:09:00.549876 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 13 04:09:00.549876 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 13 04:09:00.549876 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 13 04:09:00.549876 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 13 04:09:00.549876 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 13 04:09:00.549876 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 13 04:09:00.549876 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 13 04:09:00.549876 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 13 04:09:00.550875 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 13 04:09:00.550875 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 13 04:09:00.550875 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 13 04:09:00.550875 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 13 04:09:00.550875 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 13 04:09:00.550875 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application return self.get_or_create( Oct 13 04:09:00.550875 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 13 04:09:00.550875 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application with Lock( Oct 13 04:09:00.550875 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 13 04:09:00.550875 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application return self._enter() Oct 13 04:09:00.550875 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 13 04:09:00.550875 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 13 04:09:00.550875 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 13 04:09:00.550875 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application return self.creator() Oct 13 04:09:00.550875 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 13 04:09:00.550875 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application created_value = creator( Oct 13 04:09:00.550875 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 13 04:09:00.551705 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 13 04:09:00.551705 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 13 04:09:00.551705 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application Oct 13 04:09:00.551864 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 209/425] 10.222.0.27 () {56 vars in 1108 bytes} [Thu Oct 13 04:09:00 2022] GET /identity/v3/domains/Default => generated 87 bytes in 277 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 13 04:09:00.558166 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-35df5e91-73f9-4278-942e-66736836fdb6 None admin] Authenticating user token {{(pid=81408) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:09:00.575403 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-30e414d3-9043-4dc7-be68-77ef5bc7f8ec None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81409) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:09:00.576039 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30e414d3-9043-4dc7-be68-77ef5bc7f8ec None admin] REQUEST_METHOD: `GET` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:09:00.576307 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30e414d3-9043-4dc7-be68-77ef5bc7f8ec None admin] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:09:00.576571 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30e414d3-9043-4dc7-be68-77ef5bc7f8ec None admin] PATH_INFO: `/v3/roles/service` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:09:00.596750 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-30e414d3-9043-4dc7-be68-77ef5bc7f8ec None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 13 04:09:00.604013 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application [None req-30e414d3-9043-4dc7-be68-77ef5bc7f8ec None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Oct 13 04:09:00.604013 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 13 04:09:00.604013 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 13 04:09:00.604013 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 13 04:09:00.604013 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 13 04:09:00.604013 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 13 04:09:00.604013 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 13 04:09:00.604013 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 13 04:09:00.604013 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 13 04:09:00.604013 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 13 04:09:00.604013 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 13 04:09:00.604013 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 13 04:09:00.604013 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 13 04:09:00.604013 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 13 04:09:00.604013 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 13 04:09:00.604013 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 13 04:09:00.605077 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 13 04:09:00.605077 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 13 04:09:00.605077 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 13 04:09:00.605077 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 13 04:09:00.605077 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 13 04:09:00.605077 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 13 04:09:00.605077 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 13 04:09:00.605077 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 13 04:09:00.605077 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 13 04:09:00.605077 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application return self.get_or_create( Oct 13 04:09:00.605077 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 13 04:09:00.605077 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application with Lock( Oct 13 04:09:00.605077 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 13 04:09:00.605077 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application return self._enter() Oct 13 04:09:00.605077 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 13 04:09:00.605077 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 13 04:09:00.606377 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 13 04:09:00.606377 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application return self.creator() Oct 13 04:09:00.606377 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 13 04:09:00.606377 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application created_value = creator( Oct 13 04:09:00.606377 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 13 04:09:00.606377 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 13 04:09:00.606377 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 13 04:09:00.606377 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 13 04:09:00.606377 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 13 04:09:00.606377 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 13 04:09:00.606377 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Oct 13 04:09:00.606377 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application Oct 13 04:09:00.606377 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 217/426] 10.222.0.27 () {56 vars in 1102 bytes} [Thu Oct 13 04:09:00 2022] GET /identity/v3/roles/service => generated 85 bytes in 238 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 13 04:09:00.613670 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-30e414d3-9043-4dc7-be68-77ef5bc7f8ec None admin] Authenticating user token {{(pid=81409) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:09:00.715144 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-35df5e91-73f9-4278-942e-66736836fdb6 None admin] Validating token access rules against request {{(pid=81408) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:09:00.726412 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG dbcounter [-] [81408] Writing DB stats keystone:SELECT=100 {{(pid=81408) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:09:00.753035 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7a226d94-aadd-473e-9797-34957289dbd1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81408) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:09:00.753642 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a226d94-aadd-473e-9797-34957289dbd1 None admin] REQUEST_METHOD: `GET` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:09:00.753918 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a226d94-aadd-473e-9797-34957289dbd1 None admin] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:09:00.754189 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a226d94-aadd-473e-9797-34957289dbd1 None admin] PATH_INFO: `/v3/domains` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:09:00.754850 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7a226d94-aadd-473e-9797-34957289dbd1 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81408) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 13 04:09:00.755151 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7a226d94-aadd-473e-9797-34957289dbd1 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:09:00.756525 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7a226d94-aadd-473e-9797-34957289dbd1 None admin] RBAC: Authorization granted {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:09:00.756862 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-30e414d3-9043-4dc7-be68-77ef5bc7f8ec None admin] Validating token access rules against request {{(pid=81409) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:09:00.774240 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 210/427] 10.222.0.27 () {56 vars in 1109 bytes} [Thu Oct 13 04:09:00 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 213 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 13 04:09:00.774240 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-309be1ad-6613-45b6-87d7-2fa35d55c4e3 None None] REQUEST_METHOD: `GET` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:09:00.774240 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-309be1ad-6613-45b6-87d7-2fa35d55c4e3 None None] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:09:00.774240 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-309be1ad-6613-45b6-87d7-2fa35d55c4e3 None None] PATH_INFO: `/` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:09:00.774240 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 211/428] 10.222.0.27 () {54 vars in 927 bytes} [Thu Oct 13 04:09:00 2022] GET /identity => generated 270 bytes in 2 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 13 04:09:00.776920 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-309be1ad-6613-45b6-87d7-2fa35d55c4e3 None None] Authenticating user token {{(pid=81408) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:09:00.811931 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9ee7e616-24bf-4ad3-87e9-5fb97632477d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81409) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:09:00.812570 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ee7e616-24bf-4ad3-87e9-5fb97632477d None admin] REQUEST_METHOD: `GET` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:09:00.812849 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ee7e616-24bf-4ad3-87e9-5fb97632477d None admin] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:09:00.813121 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ee7e616-24bf-4ad3-87e9-5fb97632477d None admin] PATH_INFO: `/v3/roles` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:09:00.813786 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9ee7e616-24bf-4ad3-87e9-5fb97632477d None admin] RBAC: Adding query filter params (name=service) {{(pid=81409) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 13 04:09:00.814089 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9ee7e616-24bf-4ad3-87e9-5fb97632477d None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:09:00.815698 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9ee7e616-24bf-4ad3-87e9-5fb97632477d None admin] RBAC: Authorization granted {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:09:00.825258 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 218/429] 10.222.0.27 () {56 vars in 1103 bytes} [Thu Oct 13 04:09:00 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 212 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 13 04:09:00.826500 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0e90e85-c0ab-4455-b39f-dd3b3bbf0763 None None] REQUEST_METHOD: `POST` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:09:00.826781 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0e90e85-c0ab-4455-b39f-dd3b3bbf0763 None None] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:09:00.827054 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0e90e85-c0ab-4455-b39f-dd3b3bbf0763 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:09:00.847407 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.auth.core [None req-b0e90e85-c0ab-4455-b39f-dd3b3bbf0763 None None] MFA Rules not processed for user `0e2d979563d6421995451a0e86bf3022`. Rule list: `[]` (Enabled: `True`). {{(pid=81409) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 13 04:09:00.848773 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.fernet_utils [None req-b0e90e85-c0ab-4455-b39f-dd3b3bbf0763 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81409) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:09:00.877641 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG dbcounter [-] [81409] Writing DB stats keystone:SELECT=100 {{(pid=81409) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:09:00.919150 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-309be1ad-6613-45b6-87d7-2fa35d55c4e3 None None] Validating token access rules against request {{(pid=81408) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:09:00.931664 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 219/430] 10.222.0.27 () {58 vars in 1024 bytes} [Thu Oct 13 04:09:00 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 106 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 13 04:09:00.932837 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b0e90e85-c0ab-4455-b39f-dd3b3bbf0763 None None] Authenticating user token {{(pid=81409) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:09:00.967252 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f702770-ae69-4b5a-a293-369c3b61ccfb None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81408) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:09:00.968786 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f702770-ae69-4b5a-a293-369c3b61ccfb None admin] REQUEST_METHOD: `GET` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:09:00.969213 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f702770-ae69-4b5a-a293-369c3b61ccfb None admin] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:09:00.969646 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f702770-ae69-4b5a-a293-369c3b61ccfb None admin] PATH_INFO: `/v3/users/swift` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:09:00.980010 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5f702770-ae69-4b5a-a293-369c3b61ccfb None admin] RBAC: Authorizing `identity:get_user(user_id=swift)` {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:09:00.982233 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5f702770-ae69-4b5a-a293-369c3b61ccfb None admin] RBAC: Authorization granted {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:09:00.999186 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application [None req-5f702770-ae69-4b5a-a293-369c3b61ccfb None admin] Could not find user: swift.: keystone.exception.UserNotFound: Could not find user: swift. Oct 13 04:09:00.999186 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 13 04:09:00.999186 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 13 04:09:00.999186 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 13 04:09:00.999186 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 13 04:09:00.999186 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 13 04:09:00.999186 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 13 04:09:00.999186 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 13 04:09:00.999186 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 13 04:09:00.999186 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 13 04:09:00.999186 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 13 04:09:00.999186 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 13 04:09:00.999186 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Oct 13 04:09:00.999186 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application return self._get_user(user_id) Oct 13 04:09:00.999186 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Oct 13 04:09:00.999186 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Oct 13 04:09:01.000119 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 13 04:09:01.000119 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 13 04:09:01.000119 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Oct 13 04:09:01.000119 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 13 04:09:01.000119 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Oct 13 04:09:01.000119 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 13 04:09:01.000119 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 13 04:09:01.000119 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 13 04:09:01.000119 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 13 04:09:01.000119 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application return self.get_or_create( Oct 13 04:09:01.000119 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 13 04:09:01.000119 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application with Lock( Oct 13 04:09:01.000119 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 13 04:09:01.000119 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application return self._enter() Oct 13 04:09:01.000119 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 13 04:09:01.000119 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 13 04:09:01.000119 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 13 04:09:01.000119 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application return self.creator() Oct 13 04:09:01.001256 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 13 04:09:01.001256 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application created_value = creator( Oct 13 04:09:01.001256 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Oct 13 04:09:01.001256 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Oct 13 04:09:01.001256 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Oct 13 04:09:01.001256 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Oct 13 04:09:01.001256 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Oct 13 04:09:01.001256 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Oct 13 04:09:01.001256 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: swift. Oct 13 04:09:01.001256 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application Oct 13 04:09:01.004091 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 212/431] 10.222.0.27 () {56 vars in 1096 bytes} [Thu Oct 13 04:09:00 2022] GET /identity/v3/users/swift => generated 83 bytes in 228 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 13 04:09:01.009723 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f702770-ae69-4b5a-a293-369c3b61ccfb None admin] Authenticating user token {{(pid=81408) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:09:01.043194 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG dbcounter [-] [81408] Writing DB stats keystone:SELECT=100 {{(pid=81408) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:09:01.059736 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b0e90e85-c0ab-4455-b39f-dd3b3bbf0763 None None] Validating token access rules against request {{(pid=81409) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:09:01.103925 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-74329d14-daae-41e8-ace3-7704e0749326 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81409) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:09:01.104790 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74329d14-daae-41e8-ace3-7704e0749326 None admin] REQUEST_METHOD: `GET` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:09:01.105180 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74329d14-daae-41e8-ace3-7704e0749326 None admin] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:09:01.105568 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74329d14-daae-41e8-ace3-7704e0749326 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:09:01.118848 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-74329d14-daae-41e8-ace3-7704e0749326 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:09:01.121140 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-74329d14-daae-41e8-ace3-7704e0749326 None admin] RBAC: Authorization granted {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:09:01.125244 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f702770-ae69-4b5a-a293-369c3b61ccfb None admin] Validating token access rules against request {{(pid=81408) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:09:01.147806 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application [None req-74329d14-daae-41e8-ace3-7704e0749326 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 13 04:09:01.147806 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 13 04:09:01.147806 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 13 04:09:01.147806 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 13 04:09:01.147806 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 13 04:09:01.147806 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 13 04:09:01.147806 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 13 04:09:01.147806 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 13 04:09:01.147806 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 13 04:09:01.147806 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 13 04:09:01.147806 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 13 04:09:01.147806 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 13 04:09:01.147806 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 13 04:09:01.147806 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 13 04:09:01.147806 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 13 04:09:01.147806 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 13 04:09:01.148954 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 13 04:09:01.148954 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 13 04:09:01.148954 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 13 04:09:01.148954 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 13 04:09:01.148954 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 13 04:09:01.148954 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application return self.get_or_create( Oct 13 04:09:01.148954 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 13 04:09:01.148954 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application with Lock( Oct 13 04:09:01.148954 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 13 04:09:01.148954 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application return self._enter() Oct 13 04:09:01.148954 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 13 04:09:01.148954 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 13 04:09:01.148954 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 13 04:09:01.148954 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application return self.creator() Oct 13 04:09:01.148954 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 13 04:09:01.148954 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application created_value = creator( Oct 13 04:09:01.148954 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 13 04:09:01.150238 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 13 04:09:01.150238 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 13 04:09:01.150238 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application Oct 13 04:09:01.152498 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 220/432] 10.222.0.27 () {56 vars in 1108 bytes} [Thu Oct 13 04:09:00 2022] GET /identity/v3/domains/Default => generated 87 bytes in 220 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 13 04:09:01.160598 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-74329d14-daae-41e8-ace3-7704e0749326 None admin] Authenticating user token {{(pid=81409) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:09:01.180166 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1f74f014-eab4-449e-87bd-0e097c3b87ae None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81408) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:09:01.181697 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f74f014-eab4-449e-87bd-0e097c3b87ae None admin] REQUEST_METHOD: `GET` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:09:01.181980 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f74f014-eab4-449e-87bd-0e097c3b87ae None admin] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:09:01.182290 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f74f014-eab4-449e-87bd-0e097c3b87ae None admin] PATH_INFO: `/v3/users` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:09:01.183077 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1f74f014-eab4-449e-87bd-0e097c3b87ae None admin] RBAC: Adding query filter params (domain_id=default, name=swift) {{(pid=81408) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 13 04:09:01.183372 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1f74f014-eab4-449e-87bd-0e097c3b87ae None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:09:01.184813 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1f74f014-eab4-449e-87bd-0e097c3b87ae None admin] RBAC: Authorization granted {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:09:01.190641 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG dbcounter [-] [81409] Writing DB stats keystone:SELECT=100 {{(pid=81409) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:09:01.205180 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 213/433] 10.222.0.27 () {56 vars in 1135 bytes} [Thu Oct 13 04:09:01 2022] GET /identity/v3/users?domain_id=default&name=swift => generated 370 bytes in 195 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 13 04:09:01.206662 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd0dfe9e-b406-4a80-8b75-7a82aa23cc87 None None] REQUEST_METHOD: `GET` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:09:01.206990 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd0dfe9e-b406-4a80-8b75-7a82aa23cc87 None None] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:09:01.207256 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd0dfe9e-b406-4a80-8b75-7a82aa23cc87 None None] PATH_INFO: `/` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:09:01.213190 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 214/434] 10.222.0.27 () {54 vars in 927 bytes} [Thu Oct 13 04:09:01 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 13 04:09:01.214161 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cd0dfe9e-b406-4a80-8b75-7a82aa23cc87 None None] Authenticating user token {{(pid=81408) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:09:01.297532 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-74329d14-daae-41e8-ace3-7704e0749326 None admin] Validating token access rules against request {{(pid=81409) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:09:01.328764 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG dbcounter [-] [81408] Writing DB stats keystone:SELECT=100 {{(pid=81408) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:09:01.336123 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6271f5c1-79fb-4490-b0dc-21fa7c6fd601 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81409) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:09:01.336794 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6271f5c1-79fb-4490-b0dc-21fa7c6fd601 None admin] REQUEST_METHOD: `GET` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:09:01.337076 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6271f5c1-79fb-4490-b0dc-21fa7c6fd601 None admin] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:09:01.337346 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6271f5c1-79fb-4490-b0dc-21fa7c6fd601 None admin] PATH_INFO: `/v3/domains` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:09:01.337990 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6271f5c1-79fb-4490-b0dc-21fa7c6fd601 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81409) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 13 04:09:01.338327 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6271f5c1-79fb-4490-b0dc-21fa7c6fd601 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:09:01.339692 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6271f5c1-79fb-4490-b0dc-21fa7c6fd601 None admin] RBAC: Authorization granted {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:09:01.342844 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cd0dfe9e-b406-4a80-8b75-7a82aa23cc87 None None] Validating token access rules against request {{(pid=81408) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:09:01.351993 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 221/435] 10.222.0.27 () {56 vars in 1109 bytes} [Thu Oct 13 04:09:01 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 192 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 13 04:09:01.355514 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8d84303-ea8b-499d-94b0-7a07c860374e None None] REQUEST_METHOD: `POST` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:09:01.357580 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8d84303-ea8b-499d-94b0-7a07c860374e None None] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:09:01.357984 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8d84303-ea8b-499d-94b0-7a07c860374e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:09:01.377084 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.auth.core [None req-c8d84303-ea8b-499d-94b0-7a07c860374e None None] MFA Rules not processed for user `0e2d979563d6421995451a0e86bf3022`. Rule list: `[]` (Enabled: `True`). {{(pid=81409) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 13 04:09:01.378086 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.fernet_utils [None req-c8d84303-ea8b-499d-94b0-7a07c860374e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81409) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:09:01.383051 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5e7ff614-b0b0-4f9d-a17e-afbf68b7ab01 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81408) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:09:01.384483 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e7ff614-b0b0-4f9d-a17e-afbf68b7ab01 None admin] REQUEST_METHOD: `GET` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:09:01.384988 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e7ff614-b0b0-4f9d-a17e-afbf68b7ab01 None admin] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:09:01.385450 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e7ff614-b0b0-4f9d-a17e-afbf68b7ab01 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:09:01.398742 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5e7ff614-b0b0-4f9d-a17e-afbf68b7ab01 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:09:01.401689 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5e7ff614-b0b0-4f9d-a17e-afbf68b7ab01 None admin] RBAC: Authorization granted {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:09:01.417153 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application [None req-5e7ff614-b0b0-4f9d-a17e-afbf68b7ab01 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 13 04:09:01.417153 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 13 04:09:01.417153 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 13 04:09:01.417153 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 13 04:09:01.417153 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 13 04:09:01.417153 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 13 04:09:01.417153 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 13 04:09:01.417153 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 13 04:09:01.417153 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 13 04:09:01.417153 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 13 04:09:01.417153 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 13 04:09:01.417153 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 13 04:09:01.417153 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 13 04:09:01.417153 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 13 04:09:01.417153 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 13 04:09:01.417153 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 13 04:09:01.418073 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 13 04:09:01.418073 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 13 04:09:01.418073 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 13 04:09:01.418073 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 13 04:09:01.418073 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 13 04:09:01.418073 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application return self.get_or_create( Oct 13 04:09:01.418073 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 13 04:09:01.418073 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application with Lock( Oct 13 04:09:01.418073 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 13 04:09:01.418073 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application return self._enter() Oct 13 04:09:01.418073 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 13 04:09:01.418073 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 13 04:09:01.418073 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 13 04:09:01.418073 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application return self.creator() Oct 13 04:09:01.418073 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 13 04:09:01.418073 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application created_value = creator( Oct 13 04:09:01.418073 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 13 04:09:01.424299 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 13 04:09:01.424299 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 13 04:09:01.424299 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application Oct 13 04:09:01.424299 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 215/436] 10.222.0.27 () {56 vars in 1108 bytes} [Thu Oct 13 04:09:01 2022] GET /identity/v3/domains/Default => generated 87 bytes in 205 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 13 04:09:01.424299 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5e7ff614-b0b0-4f9d-a17e-afbf68b7ab01 None admin] Authenticating user token {{(pid=81408) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:09:01.454195 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 222/437] 10.222.0.27 () {58 vars in 1024 bytes} [Thu Oct 13 04:09:01 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 99 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 13 04:09:01.454964 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c8d84303-ea8b-499d-94b0-7a07c860374e None None] Authenticating user token {{(pid=81409) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:09:01.457337 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG dbcounter [-] [81409] Writing DB stats keystone:SELECT=100 {{(pid=81409) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:09:01.572376 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5e7ff614-b0b0-4f9d-a17e-afbf68b7ab01 None admin] Validating token access rules against request {{(pid=81408) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:09:01.579052 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c8d84303-ea8b-499d-94b0-7a07c860374e None None] Validating token access rules against request {{(pid=81409) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:09:01.615329 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f42ac8b3-b5a5-4b25-ac9e-699ce2a6c7a8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81409) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:09:01.615964 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f42ac8b3-b5a5-4b25-ac9e-699ce2a6c7a8 None admin] REQUEST_METHOD: `GET` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:09:01.616239 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f42ac8b3-b5a5-4b25-ac9e-699ce2a6c7a8 None admin] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:09:01.616507 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f42ac8b3-b5a5-4b25-ac9e-699ce2a6c7a8 None admin] PATH_INFO: `/v3/users/nova` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:09:01.619237 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9b8077a1-1844-4eba-804b-43935576adfe None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81408) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:09:01.619872 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b8077a1-1844-4eba-804b-43935576adfe None admin] REQUEST_METHOD: `GET` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:09:01.620147 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b8077a1-1844-4eba-804b-43935576adfe None admin] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:09:01.620421 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b8077a1-1844-4eba-804b-43935576adfe None admin] PATH_INFO: `/v3/roles/service` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:09:01.624708 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f42ac8b3-b5a5-4b25-ac9e-699ce2a6c7a8 None admin] RBAC: Authorizing `identity:get_user(user_id=nova)` {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:09:01.626491 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f42ac8b3-b5a5-4b25-ac9e-699ce2a6c7a8 None admin] RBAC: Authorization granted {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:09:01.628625 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG dbcounter [-] [81408] Writing DB stats keystone:SELECT=100 {{(pid=81408) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:09:01.633631 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9b8077a1-1844-4eba-804b-43935576adfe None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 13 04:09:01.638659 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application [None req-9b8077a1-1844-4eba-804b-43935576adfe None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Oct 13 04:09:01.638659 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 13 04:09:01.638659 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 13 04:09:01.638659 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 13 04:09:01.638659 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 13 04:09:01.638659 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 13 04:09:01.638659 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 13 04:09:01.638659 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 13 04:09:01.638659 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 13 04:09:01.638659 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 13 04:09:01.638659 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 13 04:09:01.638659 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 13 04:09:01.638659 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 13 04:09:01.638659 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 13 04:09:01.638659 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 13 04:09:01.638659 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 13 04:09:01.639720 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 13 04:09:01.639720 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 13 04:09:01.639720 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 13 04:09:01.639720 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 13 04:09:01.639720 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 13 04:09:01.639720 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 13 04:09:01.639720 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 13 04:09:01.639720 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 13 04:09:01.639720 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 13 04:09:01.639720 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application return self.get_or_create( Oct 13 04:09:01.639720 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 13 04:09:01.639720 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application with Lock( Oct 13 04:09:01.639720 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 13 04:09:01.639720 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application return self._enter() Oct 13 04:09:01.639720 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 13 04:09:01.639720 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 13 04:09:01.640700 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 13 04:09:01.640700 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application return self.creator() Oct 13 04:09:01.640700 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 13 04:09:01.640700 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application created_value = creator( Oct 13 04:09:01.640700 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 13 04:09:01.640700 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 13 04:09:01.640700 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 13 04:09:01.640700 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 13 04:09:01.640700 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 13 04:09:01.640700 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 13 04:09:01.640700 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Oct 13 04:09:01.640700 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application Oct 13 04:09:01.642171 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application [None req-f42ac8b3-b5a5-4b25-ac9e-699ce2a6c7a8 None admin] Could not find user: nova.: keystone.exception.UserNotFound: Could not find user: nova. Oct 13 04:09:01.642171 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 13 04:09:01.642171 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 13 04:09:01.642171 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 13 04:09:01.642171 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 13 04:09:01.642171 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 13 04:09:01.642171 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 13 04:09:01.642171 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 13 04:09:01.642171 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 13 04:09:01.642171 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 13 04:09:01.642171 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 13 04:09:01.642171 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 13 04:09:01.642171 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Oct 13 04:09:01.642171 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application return self._get_user(user_id) Oct 13 04:09:01.642171 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Oct 13 04:09:01.642171 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Oct 13 04:09:01.643222 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 13 04:09:01.643222 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 13 04:09:01.643222 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Oct 13 04:09:01.643222 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 13 04:09:01.643222 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Oct 13 04:09:01.643222 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 13 04:09:01.643222 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 13 04:09:01.643222 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 13 04:09:01.643222 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 13 04:09:01.643222 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application return self.get_or_create( Oct 13 04:09:01.643222 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 13 04:09:01.643222 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application with Lock( Oct 13 04:09:01.643222 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 13 04:09:01.643222 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application return self._enter() Oct 13 04:09:01.643222 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 13 04:09:01.643222 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 13 04:09:01.643222 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 13 04:09:01.643222 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application return self.creator() Oct 13 04:09:01.644325 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 13 04:09:01.644325 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application created_value = creator( Oct 13 04:09:01.644325 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Oct 13 04:09:01.644325 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Oct 13 04:09:01.644325 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Oct 13 04:09:01.644325 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Oct 13 04:09:01.644325 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Oct 13 04:09:01.644325 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Oct 13 04:09:01.644325 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: nova. Oct 13 04:09:01.644325 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application Oct 13 04:09:01.644894 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 216/438] 10.222.0.27 () {56 vars in 1102 bytes} [Thu Oct 13 04:09:01 2022] GET /identity/v3/roles/service => generated 85 bytes in 226 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 13 04:09:01.645519 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9b8077a1-1844-4eba-804b-43935576adfe None admin] Authenticating user token {{(pid=81408) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:09:01.646311 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 223/439] 10.222.0.27 () {56 vars in 1093 bytes} [Thu Oct 13 04:09:01 2022] GET /identity/v3/users/nova => generated 82 bytes in 192 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 13 04:09:01.647710 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c727a6d7-63c8-4ab1-aab9-9f8ea8d4e333 None None] REQUEST_METHOD: `GET` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:09:01.648127 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c727a6d7-63c8-4ab1-aab9-9f8ea8d4e333 None None] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:09:01.648403 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c727a6d7-63c8-4ab1-aab9-9f8ea8d4e333 None None] PATH_INFO: `/` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:09:01.650327 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 224/440] 10.222.0.27 () {54 vars in 927 bytes} [Thu Oct 13 04:09:01 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 13 04:09:01.651352 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c727a6d7-63c8-4ab1-aab9-9f8ea8d4e333 None None] Authenticating user token {{(pid=81409) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:09:01.722825 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG dbcounter [-] [81409] Writing DB stats keystone:SELECT=100 {{(pid=81409) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:09:01.757542 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9b8077a1-1844-4eba-804b-43935576adfe None admin] Validating token access rules against request {{(pid=81408) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:09:01.765992 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c727a6d7-63c8-4ab1-aab9-9f8ea8d4e333 None None] Validating token access rules against request {{(pid=81409) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:09:01.791677 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e7cbddc8-8d1c-4808-9dd0-ce0d35feec09 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81408) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:09:01.792283 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7cbddc8-8d1c-4808-9dd0-ce0d35feec09 None admin] REQUEST_METHOD: `GET` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:09:01.792560 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7cbddc8-8d1c-4808-9dd0-ce0d35feec09 None admin] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:09:01.792833 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7cbddc8-8d1c-4808-9dd0-ce0d35feec09 None admin] PATH_INFO: `/v3/domains` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:09:01.793471 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e7cbddc8-8d1c-4808-9dd0-ce0d35feec09 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81408) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 13 04:09:01.793770 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e7cbddc8-8d1c-4808-9dd0-ce0d35feec09 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:09:01.795193 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e7cbddc8-8d1c-4808-9dd0-ce0d35feec09 None admin] RBAC: Authorization granted {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:09:01.799524 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5a39fd7a-41e2-4872-bd23-b04ea5c6ea76 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81409) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:09:01.800127 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a39fd7a-41e2-4872-bd23-b04ea5c6ea76 None admin] REQUEST_METHOD: `GET` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:09:01.800403 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a39fd7a-41e2-4872-bd23-b04ea5c6ea76 None admin] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:09:01.800670 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a39fd7a-41e2-4872-bd23-b04ea5c6ea76 None admin] PATH_INFO: `/v3/roles` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:09:01.801322 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5a39fd7a-41e2-4872-bd23-b04ea5c6ea76 None admin] RBAC: Adding query filter params (name=service) {{(pid=81409) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 13 04:09:01.801626 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5a39fd7a-41e2-4872-bd23-b04ea5c6ea76 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:09:01.803007 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5a39fd7a-41e2-4872-bd23-b04ea5c6ea76 None admin] RBAC: Authorization granted {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:09:01.807156 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 217/441] 10.222.0.27 () {56 vars in 1109 bytes} [Thu Oct 13 04:09:01 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 162 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 13 04:09:01.807956 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e7cbddc8-8d1c-4808-9dd0-ce0d35feec09 None admin] Authenticating user token {{(pid=81408) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:09:01.814468 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 225/442] 10.222.0.27 () {56 vars in 1103 bytes} [Thu Oct 13 04:09:01 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 164 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 13 04:09:01.815671 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39dfda35-42ea-4e8a-9361-c61ce561ddf5 None None] REQUEST_METHOD: `POST` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:09:01.816048 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39dfda35-42ea-4e8a-9361-c61ce561ddf5 None None] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:09:01.816321 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39dfda35-42ea-4e8a-9361-c61ce561ddf5 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:09:01.834491 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.auth.core [None req-39dfda35-42ea-4e8a-9361-c61ce561ddf5 None None] MFA Rules not processed for user `0e2d979563d6421995451a0e86bf3022`. Rule list: `[]` (Enabled: `True`). {{(pid=81409) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 13 04:09:01.835620 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.fernet_utils [None req-39dfda35-42ea-4e8a-9361-c61ce561ddf5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81409) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:09:01.868777 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG dbcounter [-] [81408] Writing DB stats keystone:SELECT=100 {{(pid=81408) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:09:01.901914 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 226/443] 10.222.0.27 () {58 vars in 1024 bytes} [Thu Oct 13 04:09:01 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 87 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 13 04:09:01.902773 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-39dfda35-42ea-4e8a-9361-c61ce561ddf5 None None] Authenticating user token {{(pid=81409) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:09:01.909644 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e7cbddc8-8d1c-4808-9dd0-ce0d35feec09 None admin] Validating token access rules against request {{(pid=81408) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:09:01.944857 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-34b465bc-bee5-4463-9a1d-b91ac49289a1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81408) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:09:01.945454 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34b465bc-bee5-4463-9a1d-b91ac49289a1 None admin] REQUEST_METHOD: `GET` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:09:01.945729 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34b465bc-bee5-4463-9a1d-b91ac49289a1 None admin] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:09:01.946082 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34b465bc-bee5-4463-9a1d-b91ac49289a1 None admin] PATH_INFO: `/v3/users` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:09:01.947344 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-34b465bc-bee5-4463-9a1d-b91ac49289a1 None admin] RBAC: Adding query filter params (domain_id=default, name=nova) {{(pid=81408) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 13 04:09:01.947645 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-34b465bc-bee5-4463-9a1d-b91ac49289a1 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:09:01.949072 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-34b465bc-bee5-4463-9a1d-b91ac49289a1 None admin] RBAC: Authorization granted {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:09:01.966589 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 218/444] 10.222.0.27 () {56 vars in 1133 bytes} [Thu Oct 13 04:09:01 2022] GET /identity/v3/users?domain_id=default&name=nova => generated 368 bytes in 159 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 13 04:09:01.967405 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-34b465bc-bee5-4463-9a1d-b91ac49289a1 None admin] Authenticating user token {{(pid=81408) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:09:01.980160 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG dbcounter [-] [81409] Writing DB stats keystone:SELECT=100 {{(pid=81409) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:09:02.028558 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-39dfda35-42ea-4e8a-9361-c61ce561ddf5 None None] Validating token access rules against request {{(pid=81409) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:09:02.066333 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-871f6f83-f8a9-4ab2-b872-2f88d7be6b5a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81409) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:09:02.067111 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-871f6f83-f8a9-4ab2-b872-2f88d7be6b5a None admin] REQUEST_METHOD: `GET` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:09:02.067393 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-871f6f83-f8a9-4ab2-b872-2f88d7be6b5a None admin] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:09:02.067708 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-871f6f83-f8a9-4ab2-b872-2f88d7be6b5a None admin] PATH_INFO: `/v3/roles/service` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:09:02.069600 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-34b465bc-bee5-4463-9a1d-b91ac49289a1 None admin] Validating token access rules against request {{(pid=81408) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:09:02.078166 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-871f6f83-f8a9-4ab2-b872-2f88d7be6b5a None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 13 04:09:02.083500 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application [None req-871f6f83-f8a9-4ab2-b872-2f88d7be6b5a None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Oct 13 04:09:02.083500 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 13 04:09:02.083500 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 13 04:09:02.083500 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 13 04:09:02.083500 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 13 04:09:02.083500 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 13 04:09:02.083500 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 13 04:09:02.083500 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 13 04:09:02.083500 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 13 04:09:02.083500 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 13 04:09:02.083500 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 13 04:09:02.083500 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 13 04:09:02.083500 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 13 04:09:02.083500 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 13 04:09:02.083500 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 13 04:09:02.083500 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 13 04:09:02.084349 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 13 04:09:02.084349 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 13 04:09:02.084349 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 13 04:09:02.084349 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 13 04:09:02.084349 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 13 04:09:02.084349 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 13 04:09:02.084349 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 13 04:09:02.084349 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 13 04:09:02.084349 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 13 04:09:02.084349 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application return self.get_or_create( Oct 13 04:09:02.084349 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 13 04:09:02.084349 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application with Lock( Oct 13 04:09:02.084349 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 13 04:09:02.084349 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application return self._enter() Oct 13 04:09:02.084349 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 13 04:09:02.084349 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 13 04:09:02.085203 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 13 04:09:02.085203 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application return self.creator() Oct 13 04:09:02.085203 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 13 04:09:02.085203 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application created_value = creator( Oct 13 04:09:02.085203 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 13 04:09:02.085203 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 13 04:09:02.085203 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 13 04:09:02.085203 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 13 04:09:02.085203 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 13 04:09:02.085203 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 13 04:09:02.085203 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Oct 13 04:09:02.085203 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application Oct 13 04:09:02.085203 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 227/445] 10.222.0.27 () {56 vars in 1102 bytes} [Thu Oct 13 04:09:01 2022] GET /identity/v3/roles/service => generated 85 bytes in 182 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 13 04:09:02.085883 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-871f6f83-f8a9-4ab2-b872-2f88d7be6b5a None admin] Authenticating user token {{(pid=81409) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:09:02.092074 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG dbcounter [-] [81408] Writing DB stats keystone:SELECT=100 {{(pid=81408) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:09:02.109958 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cbe56cd7-a9d2-4b93-9f49-19b7ded98adc None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81408) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:09:02.110633 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cbe56cd7-a9d2-4b93-9f49-19b7ded98adc None admin] REQUEST_METHOD: `GET` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:09:02.110914 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cbe56cd7-a9d2-4b93-9f49-19b7ded98adc None admin] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:09:02.111179 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cbe56cd7-a9d2-4b93-9f49-19b7ded98adc None admin] PATH_INFO: `/v3/projects/service` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:09:02.118733 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cbe56cd7-a9d2-4b93-9f49-19b7ded98adc None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:09:02.120196 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cbe56cd7-a9d2-4b93-9f49-19b7ded98adc None admin] RBAC: Authorization granted {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:09:02.131164 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application [None req-cbe56cd7-a9d2-4b93-9f49-19b7ded98adc None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Oct 13 04:09:02.131164 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 13 04:09:02.131164 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 13 04:09:02.131164 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 13 04:09:02.131164 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 13 04:09:02.131164 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 13 04:09:02.131164 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 13 04:09:02.131164 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 13 04:09:02.131164 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 13 04:09:02.131164 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 13 04:09:02.131164 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 13 04:09:02.131164 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 13 04:09:02.131164 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Oct 13 04:09:02.131164 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application return self._get_project(project_id) Oct 13 04:09:02.131164 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Oct 13 04:09:02.132182 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Oct 13 04:09:02.132182 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 13 04:09:02.132182 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 13 04:09:02.132182 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 13 04:09:02.132182 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 13 04:09:02.132182 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 13 04:09:02.132182 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application return self.get_or_create( Oct 13 04:09:02.132182 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 13 04:09:02.132182 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application with Lock( Oct 13 04:09:02.132182 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 13 04:09:02.132182 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application return self._enter() Oct 13 04:09:02.132182 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 13 04:09:02.132182 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 13 04:09:02.132182 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 13 04:09:02.132182 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application return self.creator() Oct 13 04:09:02.132182 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 13 04:09:02.132182 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application created_value = creator( Oct 13 04:09:02.133046 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Oct 13 04:09:02.133046 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Oct 13 04:09:02.133046 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Oct 13 04:09:02.133046 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Oct 13 04:09:02.133046 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Oct 13 04:09:02.133046 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Oct 13 04:09:02.133046 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Oct 13 04:09:02.133046 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application Oct 13 04:09:02.133046 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 219/446] 10.222.0.27 () {56 vars in 1111 bytes} [Thu Oct 13 04:09:01 2022] GET /identity/v3/projects/service => generated 88 bytes in 165 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 13 04:09:02.133594 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cbe56cd7-a9d2-4b93-9f49-19b7ded98adc None admin] Authenticating user token {{(pid=81408) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:09:02.199296 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-871f6f83-f8a9-4ab2-b872-2f88d7be6b5a None admin] Validating token access rules against request {{(pid=81409) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:09:02.213042 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG dbcounter [-] [81409] Writing DB stats keystone:SELECT=100 {{(pid=81409) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:09:02.235539 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-41ef3e5f-a7d3-4805-b8d2-63a460880d29 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81409) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:09:02.236164 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41ef3e5f-a7d3-4805-b8d2-63a460880d29 None admin] REQUEST_METHOD: `GET` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:09:02.236441 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41ef3e5f-a7d3-4805-b8d2-63a460880d29 None admin] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:09:02.236713 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41ef3e5f-a7d3-4805-b8d2-63a460880d29 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:09:02.238273 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cbe56cd7-a9d2-4b93-9f49-19b7ded98adc None admin] Validating token access rules against request {{(pid=81408) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:09:02.247703 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-41ef3e5f-a7d3-4805-b8d2-63a460880d29 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:09:02.249181 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-41ef3e5f-a7d3-4805-b8d2-63a460880d29 None admin] RBAC: Authorization granted {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:09:02.265155 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application [None req-41ef3e5f-a7d3-4805-b8d2-63a460880d29 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 13 04:09:02.265155 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 13 04:09:02.265155 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 13 04:09:02.265155 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 13 04:09:02.265155 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 13 04:09:02.265155 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 13 04:09:02.265155 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 13 04:09:02.265155 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 13 04:09:02.265155 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 13 04:09:02.265155 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 13 04:09:02.265155 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 13 04:09:02.265155 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 13 04:09:02.265155 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 13 04:09:02.265155 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 13 04:09:02.265155 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 13 04:09:02.265155 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 13 04:09:02.266275 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 13 04:09:02.266275 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 13 04:09:02.266275 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 13 04:09:02.266275 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 13 04:09:02.266275 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 13 04:09:02.266275 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application return self.get_or_create( Oct 13 04:09:02.266275 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 13 04:09:02.266275 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application with Lock( Oct 13 04:09:02.266275 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 13 04:09:02.266275 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application return self._enter() Oct 13 04:09:02.266275 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 13 04:09:02.266275 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 13 04:09:02.266275 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 13 04:09:02.266275 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application return self.creator() Oct 13 04:09:02.266275 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 13 04:09:02.266275 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application created_value = creator( Oct 13 04:09:02.266275 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 13 04:09:02.267305 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 13 04:09:02.267305 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 13 04:09:02.267305 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application Oct 13 04:09:02.267305 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 228/447] 10.222.0.27 () {56 vars in 1108 bytes} [Thu Oct 13 04:09:02 2022] GET /identity/v3/domains/Default => generated 87 bytes in 181 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 13 04:09:02.267566 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-41ef3e5f-a7d3-4805-b8d2-63a460880d29 None admin] Authenticating user token {{(pid=81409) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:09:02.275306 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fab81c8a-bc15-4bab-8f6f-076e3efe355e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81408) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:09:02.276004 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fab81c8a-bc15-4bab-8f6f-076e3efe355e None admin] REQUEST_METHOD: `GET` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:09:02.276344 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fab81c8a-bc15-4bab-8f6f-076e3efe355e None admin] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:09:02.276619 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fab81c8a-bc15-4bab-8f6f-076e3efe355e None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:09:02.324107 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG dbcounter [-] [81408] Writing DB stats keystone:SELECT=100 {{(pid=81408) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:09:02.345379 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fab81c8a-bc15-4bab-8f6f-076e3efe355e None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:09:02.346835 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fab81c8a-bc15-4bab-8f6f-076e3efe355e None admin] RBAC: Authorization granted {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:09:02.367881 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-41ef3e5f-a7d3-4805-b8d2-63a460880d29 None admin] Validating token access rules against request {{(pid=81409) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:09:02.401692 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6e84dd8b-6dd4-4598-9d46-39134906ed63 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81409) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:09:02.402325 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e84dd8b-6dd4-4598-9d46-39134906ed63 None admin] REQUEST_METHOD: `GET` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:09:02.402603 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e84dd8b-6dd4-4598-9d46-39134906ed63 None admin] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:09:02.402871 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e84dd8b-6dd4-4598-9d46-39134906ed63 None admin] PATH_INFO: `/v3/roles` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:09:02.403522 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6e84dd8b-6dd4-4598-9d46-39134906ed63 None admin] RBAC: Adding query filter params (name=service) {{(pid=81409) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 13 04:09:02.403822 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6e84dd8b-6dd4-4598-9d46-39134906ed63 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:09:02.405182 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6e84dd8b-6dd4-4598-9d46-39134906ed63 None admin] RBAC: Authorization granted {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:09:02.413150 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 229/448] 10.222.0.27 () {56 vars in 1103 bytes} [Thu Oct 13 04:09:02 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 146 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 13 04:09:02.413968 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6e84dd8b-6dd4-4598-9d46-39134906ed63 None admin] Authenticating user token {{(pid=81409) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:09:02.443907 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG dbcounter [-] [81409] Writing DB stats keystone:SELECT=100 {{(pid=81409) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:09:02.450611 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 220/449] 10.222.0.27 () {58 vars in 1282 bytes} [Thu Oct 13 04:09:02 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 317 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Oct 13 04:09:02.451392 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fab81c8a-bc15-4bab-8f6f-076e3efe355e None admin] Authenticating user token {{(pid=81408) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:09:02.517403 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6e84dd8b-6dd4-4598-9d46-39134906ed63 None admin] Validating token access rules against request {{(pid=81409) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:09:02.527110 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG dbcounter [-] [81408] Writing DB stats keystone:SELECT=100 {{(pid=81408) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:09:02.549097 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fab81c8a-bc15-4bab-8f6f-076e3efe355e None admin] Validating token access rules against request {{(pid=81408) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:09:02.550901 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-90cc292e-b166-4790-8ad1-75dae01fce1c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81409) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:09:02.551500 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90cc292e-b166-4790-8ad1-75dae01fce1c None admin] REQUEST_METHOD: `GET` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:09:02.551776 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90cc292e-b166-4790-8ad1-75dae01fce1c None admin] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:09:02.552042 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90cc292e-b166-4790-8ad1-75dae01fce1c None admin] PATH_INFO: `/v3/projects` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:09:02.552717 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-90cc292e-b166-4790-8ad1-75dae01fce1c None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81409) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 13 04:09:02.553018 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-90cc292e-b166-4790-8ad1-75dae01fce1c None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:09:02.554457 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-90cc292e-b166-4790-8ad1-75dae01fce1c None admin] RBAC: Authorization granted {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:09:02.566729 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 230/450] 10.222.0.27 () {56 vars in 1148 bytes} [Thu Oct 13 04:09:02 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 153 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 13 04:09:02.567518 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-90cc292e-b166-4790-8ad1-75dae01fce1c None admin] Authenticating user token {{(pid=81409) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:09:02.582436 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f5623c9c-96c2-4be1-a06a-45f468a302ff None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81408) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:09:02.583051 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5623c9c-96c2-4be1-a06a-45f468a302ff None admin] REQUEST_METHOD: `GET` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:09:02.583319 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5623c9c-96c2-4be1-a06a-45f468a302ff None admin] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:09:02.583584 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5623c9c-96c2-4be1-a06a-45f468a302ff None admin] PATH_INFO: `/v3/roles/service` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:09:02.593884 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f5623c9c-96c2-4be1-a06a-45f468a302ff None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 13 04:09:02.598667 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application [None req-f5623c9c-96c2-4be1-a06a-45f468a302ff None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Oct 13 04:09:02.598667 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 13 04:09:02.598667 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 13 04:09:02.598667 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 13 04:09:02.598667 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 13 04:09:02.598667 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 13 04:09:02.598667 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 13 04:09:02.598667 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 13 04:09:02.598667 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 13 04:09:02.598667 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 13 04:09:02.598667 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 13 04:09:02.598667 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 13 04:09:02.598667 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 13 04:09:02.598667 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 13 04:09:02.598667 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 13 04:09:02.598667 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 13 04:09:02.599714 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 13 04:09:02.599714 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 13 04:09:02.599714 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 13 04:09:02.599714 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 13 04:09:02.599714 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 13 04:09:02.599714 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 13 04:09:02.599714 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 13 04:09:02.599714 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 13 04:09:02.599714 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 13 04:09:02.599714 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application return self.get_or_create( Oct 13 04:09:02.599714 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 13 04:09:02.599714 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application with Lock( Oct 13 04:09:02.599714 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 13 04:09:02.599714 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application return self._enter() Oct 13 04:09:02.599714 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 13 04:09:02.599714 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 13 04:09:02.600962 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 13 04:09:02.600962 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application return self.creator() Oct 13 04:09:02.600962 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 13 04:09:02.600962 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application created_value = creator( Oct 13 04:09:02.600962 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 13 04:09:02.600962 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 13 04:09:02.600962 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 13 04:09:02.600962 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 13 04:09:02.600962 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 13 04:09:02.600962 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 13 04:09:02.600962 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Oct 13 04:09:02.600962 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application Oct 13 04:09:02.600962 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 221/451] 10.222.0.27 () {56 vars in 1102 bytes} [Thu Oct 13 04:09:02 2022] GET /identity/v3/roles/service => generated 85 bytes in 149 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 13 04:09:02.600962 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f5623c9c-96c2-4be1-a06a-45f468a302ff None admin] Authenticating user token {{(pid=81408) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:09:02.658120 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG dbcounter [-] [81409] Writing DB stats keystone:SELECT=100 {{(pid=81409) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:09:02.670575 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-90cc292e-b166-4790-8ad1-75dae01fce1c None admin] Validating token access rules against request {{(pid=81409) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:09:02.702143 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f5623c9c-96c2-4be1-a06a-45f468a302ff None admin] Validating token access rules against request {{(pid=81408) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:09:02.703571 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d2d846f0-1822-4886-930e-55104e136c5b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81409) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:09:02.704169 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2d846f0-1822-4886-930e-55104e136c5b None admin] REQUEST_METHOD: `GET` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:09:02.704445 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2d846f0-1822-4886-930e-55104e136c5b None admin] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:09:02.704708 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2d846f0-1822-4886-930e-55104e136c5b None admin] PATH_INFO: `/v3/domains` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:09:02.705336 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d2d846f0-1822-4886-930e-55104e136c5b None admin] RBAC: Adding query filter params (name=Default) {{(pid=81409) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 13 04:09:02.705635 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d2d846f0-1822-4886-930e-55104e136c5b None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:09:02.707027 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d2d846f0-1822-4886-930e-55104e136c5b None admin] RBAC: Authorization granted {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:09:02.718793 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 231/452] 10.222.0.27 () {56 vars in 1109 bytes} [Thu Oct 13 04:09:02 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 152 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 13 04:09:02.719598 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d2d846f0-1822-4886-930e-55104e136c5b None admin] Authenticating user token {{(pid=81409) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:09:02.737239 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9079143e-c361-4a6f-8345-bc08cd086899 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81408) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:09:02.737861 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9079143e-c361-4a6f-8345-bc08cd086899 None admin] REQUEST_METHOD: `GET` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:09:02.738135 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9079143e-c361-4a6f-8345-bc08cd086899 None admin] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:09:02.738436 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9079143e-c361-4a6f-8345-bc08cd086899 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:09:02.745302 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG dbcounter [-] [81408] Writing DB stats keystone:SELECT=100 {{(pid=81408) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:09:02.749614 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9079143e-c361-4a6f-8345-bc08cd086899 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:09:02.751089 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9079143e-c361-4a6f-8345-bc08cd086899 None admin] RBAC: Authorization granted {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:09:02.766259 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application [None req-9079143e-c361-4a6f-8345-bc08cd086899 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 13 04:09:02.766259 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 13 04:09:02.766259 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 13 04:09:02.766259 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 13 04:09:02.766259 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 13 04:09:02.766259 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 13 04:09:02.766259 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 13 04:09:02.766259 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 13 04:09:02.766259 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 13 04:09:02.766259 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 13 04:09:02.766259 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 13 04:09:02.766259 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 13 04:09:02.766259 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 13 04:09:02.766259 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 13 04:09:02.766259 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 13 04:09:02.766259 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 13 04:09:02.767251 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 13 04:09:02.767251 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 13 04:09:02.767251 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 13 04:09:02.767251 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 13 04:09:02.767251 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 13 04:09:02.767251 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application return self.get_or_create( Oct 13 04:09:02.767251 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 13 04:09:02.767251 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application with Lock( Oct 13 04:09:02.767251 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 13 04:09:02.767251 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application return self._enter() Oct 13 04:09:02.767251 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 13 04:09:02.767251 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 13 04:09:02.767251 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 13 04:09:02.767251 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application return self.creator() Oct 13 04:09:02.767251 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 13 04:09:02.767251 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application created_value = creator( Oct 13 04:09:02.767251 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 13 04:09:02.768098 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 13 04:09:02.768098 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 13 04:09:02.768098 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application Oct 13 04:09:02.768098 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 222/453] 10.222.0.27 () {56 vars in 1108 bytes} [Thu Oct 13 04:09:02 2022] GET /identity/v3/domains/Default => generated 87 bytes in 167 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 13 04:09:02.768098 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9079143e-c361-4a6f-8345-bc08cd086899 None admin] Authenticating user token {{(pid=81408) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:09:02.822690 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d2d846f0-1822-4886-930e-55104e136c5b None admin] Validating token access rules against request {{(pid=81409) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:09:02.856597 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7b56eb49-7371-479e-b05f-3aec2de8a346 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81409) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:09:02.857157 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b56eb49-7371-479e-b05f-3aec2de8a346 None admin] REQUEST_METHOD: `GET` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:09:02.857434 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b56eb49-7371-479e-b05f-3aec2de8a346 None admin] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:09:02.857709 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b56eb49-7371-479e-b05f-3aec2de8a346 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:09:02.868077 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7b56eb49-7371-479e-b05f-3aec2de8a346 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:09:02.868683 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9079143e-c361-4a6f-8345-bc08cd086899 None admin] Validating token access rules against request {{(pid=81408) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:09:02.869527 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7b56eb49-7371-479e-b05f-3aec2de8a346 None admin] RBAC: Authorization granted {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:09:02.877533 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG dbcounter [-] [81409] Writing DB stats keystone:SELECT=100 {{(pid=81409) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:09:02.884844 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application [None req-7b56eb49-7371-479e-b05f-3aec2de8a346 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 13 04:09:02.884844 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 13 04:09:02.884844 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 13 04:09:02.884844 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 13 04:09:02.884844 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 13 04:09:02.884844 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 13 04:09:02.884844 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 13 04:09:02.884844 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 13 04:09:02.884844 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 13 04:09:02.884844 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 13 04:09:02.884844 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 13 04:09:02.884844 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 13 04:09:02.884844 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 13 04:09:02.884844 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 13 04:09:02.884844 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 13 04:09:02.884844 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 13 04:09:02.885939 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 13 04:09:02.885939 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 13 04:09:02.885939 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 13 04:09:02.885939 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 13 04:09:02.885939 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 13 04:09:02.885939 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application return self.get_or_create( Oct 13 04:09:02.885939 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 13 04:09:02.885939 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application with Lock( Oct 13 04:09:02.885939 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 13 04:09:02.885939 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application return self._enter() Oct 13 04:09:02.885939 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 13 04:09:02.885939 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 13 04:09:02.885939 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 13 04:09:02.885939 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application return self.creator() Oct 13 04:09:02.885939 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 13 04:09:02.885939 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application created_value = creator( Oct 13 04:09:02.885939 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 13 04:09:02.886933 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 13 04:09:02.886933 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 13 04:09:02.886933 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application Oct 13 04:09:02.887685 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 232/454] 10.222.0.27 () {56 vars in 1108 bytes} [Thu Oct 13 04:09:02 2022] GET /identity/v3/domains/Default => generated 87 bytes in 168 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 13 04:09:02.888476 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7b56eb49-7371-479e-b05f-3aec2de8a346 None admin] Authenticating user token {{(pid=81409) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:09:02.903160 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8c901e59-7ab9-4a13-bc8c-bec94e3ddeea None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81408) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:09:02.903805 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c901e59-7ab9-4a13-bc8c-bec94e3ddeea None admin] REQUEST_METHOD: `PUT` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:09:02.904083 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c901e59-7ab9-4a13-bc8c-bec94e3ddeea None admin] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:09:02.904347 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c901e59-7ab9-4a13-bc8c-bec94e3ddeea None admin] PATH_INFO: `/v3/projects/dcec0431fb6849499d6aec35a234daa6/users/90a0f466b664458898cf9dc4112ae651/roles/4cfff1d3d68e49c9b04231a19cf76ede` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:09:02.918707 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8c901e59-7ab9-4a13-bc8c-bec94e3ddeea None admin] RBAC: Authorizing `identity:create_grant(project_id=dcec0431fb6849499d6aec35a234daa6, user_id=90a0f466b664458898cf9dc4112ae651, role_id=4cfff1d3d68e49c9b04231a19cf76ede)` {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:09:02.920153 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8c901e59-7ab9-4a13-bc8c-bec94e3ddeea None admin] RBAC: Authorization granted {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:09:02.932220 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 223/455] 10.222.0.27 () {58 vars in 1439 bytes} [Thu Oct 13 04:09:02 2022] PUT /identity/v3/projects/dcec0431fb6849499d6aec35a234daa6/users/90a0f466b664458898cf9dc4112ae651/roles/4cfff1d3d68e49c9b04231a19cf76ede => generated 0 bytes in 164 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 13 04:09:02.934373 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8c901e59-7ab9-4a13-bc8c-bec94e3ddeea None admin] Authenticating user token {{(pid=81408) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:09:02.979353 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG dbcounter [-] [81408] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81408) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:09:03.010516 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7b56eb49-7371-479e-b05f-3aec2de8a346 None admin] Validating token access rules against request {{(pid=81409) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:09:03.049295 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8c901e59-7ab9-4a13-bc8c-bec94e3ddeea None admin] Validating token access rules against request {{(pid=81408) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:09:03.051259 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-18b0f614-4966-4c77-a3a3-0dba92eda2ab None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81409) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:09:03.051873 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18b0f614-4966-4c77-a3a3-0dba92eda2ab None admin] REQUEST_METHOD: `GET` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:09:03.052870 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18b0f614-4966-4c77-a3a3-0dba92eda2ab None admin] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:09:03.052870 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18b0f614-4966-4c77-a3a3-0dba92eda2ab None admin] PATH_INFO: `/v3/roles` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:09:03.053039 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-18b0f614-4966-4c77-a3a3-0dba92eda2ab None admin] RBAC: Adding query filter params (name=service) {{(pid=81409) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 13 04:09:03.053336 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-18b0f614-4966-4c77-a3a3-0dba92eda2ab None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:09:03.054753 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-18b0f614-4966-4c77-a3a3-0dba92eda2ab None admin] RBAC: Authorization granted {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:09:03.064475 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 233/456] 10.222.0.27 () {56 vars in 1103 bytes} [Thu Oct 13 04:09:02 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 176 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 13 04:09:03.065450 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-18b0f614-4966-4c77-a3a3-0dba92eda2ab None admin] Authenticating user token {{(pid=81409) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:09:03.092457 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-13603e58-21f5-4325-af34-1576cd8ec0d3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81408) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:09:03.093096 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13603e58-21f5-4325-af34-1576cd8ec0d3 None admin] REQUEST_METHOD: `GET` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:09:03.093379 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13603e58-21f5-4325-af34-1576cd8ec0d3 None admin] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:09:03.093647 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13603e58-21f5-4325-af34-1576cd8ec0d3 None admin] PATH_INFO: `/v3/users/cinder` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:09:03.101779 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-13603e58-21f5-4325-af34-1576cd8ec0d3 None admin] RBAC: Authorizing `identity:get_user(user_id=cinder)` {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:09:03.103814 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-13603e58-21f5-4325-af34-1576cd8ec0d3 None admin] RBAC: Authorization granted {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:09:03.117796 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application [None req-13603e58-21f5-4325-af34-1576cd8ec0d3 None admin] Could not find user: cinder.: keystone.exception.UserNotFound: Could not find user: cinder. Oct 13 04:09:03.117796 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 13 04:09:03.117796 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 13 04:09:03.117796 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 13 04:09:03.117796 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 13 04:09:03.117796 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 13 04:09:03.117796 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 13 04:09:03.117796 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 13 04:09:03.117796 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 13 04:09:03.117796 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 13 04:09:03.117796 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 13 04:09:03.117796 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 13 04:09:03.117796 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Oct 13 04:09:03.117796 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application return self._get_user(user_id) Oct 13 04:09:03.117796 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Oct 13 04:09:03.117796 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Oct 13 04:09:03.119315 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 13 04:09:03.119315 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 13 04:09:03.119315 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Oct 13 04:09:03.119315 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 13 04:09:03.119315 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Oct 13 04:09:03.119315 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 13 04:09:03.119315 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 13 04:09:03.119315 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 13 04:09:03.119315 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 13 04:09:03.119315 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application return self.get_or_create( Oct 13 04:09:03.119315 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 13 04:09:03.119315 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application with Lock( Oct 13 04:09:03.119315 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 13 04:09:03.119315 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application return self._enter() Oct 13 04:09:03.119315 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 13 04:09:03.119315 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 13 04:09:03.119315 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 13 04:09:03.119315 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application return self.creator() Oct 13 04:09:03.120227 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 13 04:09:03.120227 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application created_value = creator( Oct 13 04:09:03.120227 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Oct 13 04:09:03.120227 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Oct 13 04:09:03.120227 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Oct 13 04:09:03.120227 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Oct 13 04:09:03.120227 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Oct 13 04:09:03.120227 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Oct 13 04:09:03.120227 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: cinder. Oct 13 04:09:03.120227 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application Oct 13 04:09:03.120227 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 224/457] 10.222.0.27 () {56 vars in 1099 bytes} [Thu Oct 13 04:09:02 2022] GET /identity/v3/users/cinder => generated 84 bytes in 187 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 13 04:09:03.120782 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-13603e58-21f5-4325-af34-1576cd8ec0d3 None admin] Authenticating user token {{(pid=81408) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:09:03.139349 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG dbcounter [-] [81409] Writing DB stats keystone:SELECT=100 {{(pid=81409) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:09:03.185615 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-18b0f614-4966-4c77-a3a3-0dba92eda2ab None admin] Validating token access rules against request {{(pid=81409) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:09:03.221515 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9a4fbe57-368d-40e3-8292-ad751d07f4cb None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81409) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:09:03.222122 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a4fbe57-368d-40e3-8292-ad751d07f4cb None admin] REQUEST_METHOD: `GET` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:09:03.222425 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a4fbe57-368d-40e3-8292-ad751d07f4cb None admin] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:09:03.222699 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a4fbe57-368d-40e3-8292-ad751d07f4cb None admin] PATH_INFO: `/v3/domains` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:09:03.223343 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9a4fbe57-368d-40e3-8292-ad751d07f4cb None admin] RBAC: Adding query filter params (name=Default) {{(pid=81409) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 13 04:09:03.223657 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9a4fbe57-368d-40e3-8292-ad751d07f4cb None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:09:03.225284 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9a4fbe57-368d-40e3-8292-ad751d07f4cb None admin] RBAC: Authorization granted {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:09:03.233827 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG dbcounter [-] [81408] Writing DB stats keystone:SELECT=100 {{(pid=81408) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:09:03.241809 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 234/458] 10.222.0.27 () {56 vars in 1109 bytes} [Thu Oct 13 04:09:03 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 173 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 13 04:09:03.241964 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-13603e58-21f5-4325-af34-1576cd8ec0d3 None admin] Validating token access rules against request {{(pid=81408) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:09:03.242040 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9a4fbe57-368d-40e3-8292-ad751d07f4cb None admin] Authenticating user token {{(pid=81409) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:09:03.281139 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-62a9dd40-3dc5-4c03-a602-41395a1786c6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81408) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:09:03.281749 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62a9dd40-3dc5-4c03-a602-41395a1786c6 None admin] REQUEST_METHOD: `GET` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:09:03.282028 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62a9dd40-3dc5-4c03-a602-41395a1786c6 None admin] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:09:03.282326 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62a9dd40-3dc5-4c03-a602-41395a1786c6 None admin] PATH_INFO: `/v3/domains` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:09:03.282968 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-62a9dd40-3dc5-4c03-a602-41395a1786c6 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81408) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 13 04:09:03.283272 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-62a9dd40-3dc5-4c03-a602-41395a1786c6 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:09:03.284644 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-62a9dd40-3dc5-4c03-a602-41395a1786c6 None admin] RBAC: Authorization granted {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:09:03.296753 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 225/459] 10.222.0.27 () {56 vars in 1109 bytes} [Thu Oct 13 04:09:03 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 176 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 13 04:09:03.297595 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-62a9dd40-3dc5-4c03-a602-41395a1786c6 None admin] Authenticating user token {{(pid=81408) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:09:03.354854 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9a4fbe57-368d-40e3-8292-ad751d07f4cb None admin] Validating token access rules against request {{(pid=81409) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:09:03.381446 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG dbcounter [-] [81409] Writing DB stats keystone:SELECT=100 {{(pid=81409) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:09:03.394464 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-415a9eb1-08e1-40fc-9fcf-0be0e1d65e5d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81409) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:09:03.395096 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-415a9eb1-08e1-40fc-9fcf-0be0e1d65e5d None admin] REQUEST_METHOD: `GET` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:09:03.395361 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-415a9eb1-08e1-40fc-9fcf-0be0e1d65e5d None admin] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:09:03.395628 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-415a9eb1-08e1-40fc-9fcf-0be0e1d65e5d None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:09:03.402684 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-62a9dd40-3dc5-4c03-a602-41395a1786c6 None admin] Validating token access rules against request {{(pid=81408) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:09:03.410490 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-415a9eb1-08e1-40fc-9fcf-0be0e1d65e5d None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:09:03.414618 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-415a9eb1-08e1-40fc-9fcf-0be0e1d65e5d None admin] RBAC: Authorization granted {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:09:03.431594 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application [None req-415a9eb1-08e1-40fc-9fcf-0be0e1d65e5d None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 13 04:09:03.431594 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 13 04:09:03.431594 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 13 04:09:03.431594 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 13 04:09:03.431594 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 13 04:09:03.431594 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 13 04:09:03.431594 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 13 04:09:03.431594 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 13 04:09:03.431594 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 13 04:09:03.431594 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 13 04:09:03.431594 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 13 04:09:03.431594 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 13 04:09:03.431594 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 13 04:09:03.431594 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 13 04:09:03.431594 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 13 04:09:03.431594 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 13 04:09:03.432739 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 13 04:09:03.432739 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 13 04:09:03.432739 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 13 04:09:03.432739 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 13 04:09:03.432739 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 13 04:09:03.432739 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application return self.get_or_create( Oct 13 04:09:03.432739 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 13 04:09:03.432739 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application with Lock( Oct 13 04:09:03.432739 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 13 04:09:03.432739 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application return self._enter() Oct 13 04:09:03.432739 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 13 04:09:03.432739 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 13 04:09:03.432739 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 13 04:09:03.432739 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application return self.creator() Oct 13 04:09:03.432739 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 13 04:09:03.432739 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application created_value = creator( Oct 13 04:09:03.432739 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 13 04:09:03.433619 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 13 04:09:03.433619 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 13 04:09:03.433619 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application Oct 13 04:09:03.433619 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 235/460] 10.222.0.27 () {56 vars in 1108 bytes} [Thu Oct 13 04:09:03 2022] GET /identity/v3/domains/Default => generated 87 bytes in 195 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 13 04:09:03.433619 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-415a9eb1-08e1-40fc-9fcf-0be0e1d65e5d None admin] Authenticating user token {{(pid=81409) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:09:03.444275 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-806b149d-191a-43a6-8ce5-26f4b85f2a59 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81408) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:09:03.445218 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-806b149d-191a-43a6-8ce5-26f4b85f2a59 None admin] REQUEST_METHOD: `GET` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:09:03.445509 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-806b149d-191a-43a6-8ce5-26f4b85f2a59 None admin] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:09:03.445779 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-806b149d-191a-43a6-8ce5-26f4b85f2a59 None admin] PATH_INFO: `/v3/users` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:09:03.446620 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-806b149d-191a-43a6-8ce5-26f4b85f2a59 None admin] RBAC: Adding query filter params (domain_id=default, name=cinder) {{(pid=81408) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 13 04:09:03.447109 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-806b149d-191a-43a6-8ce5-26f4b85f2a59 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:09:03.448567 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-806b149d-191a-43a6-8ce5-26f4b85f2a59 None admin] RBAC: Authorization granted {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:09:03.470894 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 226/461] 10.222.0.27 () {56 vars in 1137 bytes} [Thu Oct 13 04:09:03 2022] GET /identity/v3/users?domain_id=default&name=cinder => generated 372 bytes in 169 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 13 04:09:03.470894 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-806b149d-191a-43a6-8ce5-26f4b85f2a59 None admin] Authenticating user token {{(pid=81408) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:09:03.489955 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG dbcounter [-] [81408] Writing DB stats keystone:SELECT=100 {{(pid=81408) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:09:03.543717 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-415a9eb1-08e1-40fc-9fcf-0be0e1d65e5d None admin] Validating token access rules against request {{(pid=81409) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:09:03.574799 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-806b149d-191a-43a6-8ce5-26f4b85f2a59 None admin] Validating token access rules against request {{(pid=81408) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:09:03.578054 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-efceaf5a-ffb6-4bde-acfd-33467b89b2e2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81409) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:09:03.578708 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-efceaf5a-ffb6-4bde-acfd-33467b89b2e2 None admin] REQUEST_METHOD: `GET` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:09:03.578989 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-efceaf5a-ffb6-4bde-acfd-33467b89b2e2 None admin] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:09:03.579255 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-efceaf5a-ffb6-4bde-acfd-33467b89b2e2 None admin] PATH_INFO: `/v3/users/glance` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:09:03.587561 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-efceaf5a-ffb6-4bde-acfd-33467b89b2e2 None admin] RBAC: Authorizing `identity:get_user(user_id=glance)` {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:09:03.589057 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-efceaf5a-ffb6-4bde-acfd-33467b89b2e2 None admin] RBAC: Authorization granted {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:09:03.600846 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application [None req-efceaf5a-ffb6-4bde-acfd-33467b89b2e2 None admin] Could not find user: glance.: keystone.exception.UserNotFound: Could not find user: glance. Oct 13 04:09:03.600846 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 13 04:09:03.600846 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 13 04:09:03.600846 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 13 04:09:03.600846 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 13 04:09:03.600846 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 13 04:09:03.600846 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 13 04:09:03.600846 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 13 04:09:03.600846 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 13 04:09:03.600846 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 13 04:09:03.600846 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 13 04:09:03.600846 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 13 04:09:03.600846 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Oct 13 04:09:03.600846 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application return self._get_user(user_id) Oct 13 04:09:03.600846 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Oct 13 04:09:03.600846 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Oct 13 04:09:03.601767 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 13 04:09:03.601767 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 13 04:09:03.601767 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Oct 13 04:09:03.601767 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 13 04:09:03.601767 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Oct 13 04:09:03.601767 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 13 04:09:03.601767 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 13 04:09:03.601767 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 13 04:09:03.601767 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 13 04:09:03.601767 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application return self.get_or_create( Oct 13 04:09:03.601767 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 13 04:09:03.601767 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application with Lock( Oct 13 04:09:03.601767 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 13 04:09:03.601767 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application return self._enter() Oct 13 04:09:03.601767 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 13 04:09:03.601767 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 13 04:09:03.601767 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 13 04:09:03.601767 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application return self.creator() Oct 13 04:09:03.602734 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 13 04:09:03.602734 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application created_value = creator( Oct 13 04:09:03.602734 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Oct 13 04:09:03.602734 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Oct 13 04:09:03.602734 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Oct 13 04:09:03.602734 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Oct 13 04:09:03.602734 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Oct 13 04:09:03.602734 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Oct 13 04:09:03.602734 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: glance. Oct 13 04:09:03.602734 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application Oct 13 04:09:03.602734 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 236/462] 10.222.0.27 () {56 vars in 1099 bytes} [Thu Oct 13 04:09:03 2022] GET /identity/v3/users/glance => generated 84 bytes in 169 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 13 04:09:03.604119 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-efceaf5a-ffb6-4bde-acfd-33467b89b2e2 None admin] Authenticating user token {{(pid=81409) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:09:03.619293 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0c5c80a5-3f40-4d6d-9e5f-b32aaf904ee4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81408) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:09:03.619925 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c5c80a5-3f40-4d6d-9e5f-b32aaf904ee4 None admin] REQUEST_METHOD: `GET` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:09:03.620201 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c5c80a5-3f40-4d6d-9e5f-b32aaf904ee4 None admin] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:09:03.620472 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c5c80a5-3f40-4d6d-9e5f-b32aaf904ee4 None admin] PATH_INFO: `/v3/projects/service` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:09:03.627384 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0c5c80a5-3f40-4d6d-9e5f-b32aaf904ee4 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:09:03.628906 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0c5c80a5-3f40-4d6d-9e5f-b32aaf904ee4 None admin] RBAC: Authorization granted {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:09:03.639520 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG dbcounter [-] [81409] Writing DB stats keystone:SELECT=100 {{(pid=81409) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:09:03.641414 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application [None req-0c5c80a5-3f40-4d6d-9e5f-b32aaf904ee4 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Oct 13 04:09:03.641414 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 13 04:09:03.641414 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 13 04:09:03.641414 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 13 04:09:03.641414 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 13 04:09:03.641414 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 13 04:09:03.641414 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 13 04:09:03.641414 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 13 04:09:03.641414 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 13 04:09:03.641414 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 13 04:09:03.641414 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 13 04:09:03.641414 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 13 04:09:03.641414 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Oct 13 04:09:03.641414 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application return self._get_project(project_id) Oct 13 04:09:03.641414 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Oct 13 04:09:03.642692 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Oct 13 04:09:03.642692 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 13 04:09:03.642692 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 13 04:09:03.642692 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 13 04:09:03.642692 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 13 04:09:03.642692 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 13 04:09:03.642692 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application return self.get_or_create( Oct 13 04:09:03.642692 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 13 04:09:03.642692 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application with Lock( Oct 13 04:09:03.642692 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 13 04:09:03.642692 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application return self._enter() Oct 13 04:09:03.642692 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 13 04:09:03.642692 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 13 04:09:03.642692 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 13 04:09:03.642692 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application return self.creator() Oct 13 04:09:03.642692 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 13 04:09:03.642692 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application created_value = creator( Oct 13 04:09:03.643727 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Oct 13 04:09:03.643727 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Oct 13 04:09:03.643727 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Oct 13 04:09:03.643727 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Oct 13 04:09:03.643727 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Oct 13 04:09:03.643727 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Oct 13 04:09:03.643727 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Oct 13 04:09:03.643727 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application Oct 13 04:09:03.647351 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 227/463] 10.222.0.27 () {56 vars in 1111 bytes} [Thu Oct 13 04:09:03 2022] GET /identity/v3/projects/service => generated 88 bytes in 180 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 13 04:09:03.648076 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0c5c80a5-3f40-4d6d-9e5f-b32aaf904ee4 None admin] Authenticating user token {{(pid=81408) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:09:03.713060 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-efceaf5a-ffb6-4bde-acfd-33467b89b2e2 None admin] Validating token access rules against request {{(pid=81409) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:09:03.730557 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG dbcounter [-] [81408] Writing DB stats keystone:SELECT=100 {{(pid=81408) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:09:03.751340 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3b5ab94b-b6e6-45a8-8417-9326aafb3a47 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81409) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:09:03.752258 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b5ab94b-b6e6-45a8-8417-9326aafb3a47 None admin] REQUEST_METHOD: `GET` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:09:03.752531 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b5ab94b-b6e6-45a8-8417-9326aafb3a47 None admin] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:09:03.752801 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b5ab94b-b6e6-45a8-8417-9326aafb3a47 None admin] PATH_INFO: `/v3/domains` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:09:03.753441 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3b5ab94b-b6e6-45a8-8417-9326aafb3a47 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81409) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 13 04:09:03.753741 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3b5ab94b-b6e6-45a8-8417-9326aafb3a47 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:09:03.755148 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3b5ab94b-b6e6-45a8-8417-9326aafb3a47 None admin] RBAC: Authorization granted {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:09:03.755417 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0c5c80a5-3f40-4d6d-9e5f-b32aaf904ee4 None admin] Validating token access rules against request {{(pid=81408) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:09:03.767680 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 237/464] 10.222.0.27 () {56 vars in 1109 bytes} [Thu Oct 13 04:09:03 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 165 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 13 04:09:03.768490 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3b5ab94b-b6e6-45a8-8417-9326aafb3a47 None admin] Authenticating user token {{(pid=81409) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:09:03.797419 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f458945e-ddc0-4275-8920-6b601223ae42 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81408) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:09:03.798027 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f458945e-ddc0-4275-8920-6b601223ae42 None admin] REQUEST_METHOD: `GET` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:09:03.798331 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f458945e-ddc0-4275-8920-6b601223ae42 None admin] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:09:03.798853 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f458945e-ddc0-4275-8920-6b601223ae42 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:09:03.872370 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG dbcounter [-] [81409] Writing DB stats keystone:SELECT=100 {{(pid=81409) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:09:03.879840 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3b5ab94b-b6e6-45a8-8417-9326aafb3a47 None admin] Validating token access rules against request {{(pid=81409) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:09:03.886377 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f458945e-ddc0-4275-8920-6b601223ae42 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:09:03.887854 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f458945e-ddc0-4275-8920-6b601223ae42 None admin] RBAC: Authorization granted {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:09:03.918376 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-266142c6-574f-4072-88df-c8da11a7c0e6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81409) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:09:03.919277 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-266142c6-574f-4072-88df-c8da11a7c0e6 None admin] REQUEST_METHOD: `GET` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:09:03.919742 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-266142c6-574f-4072-88df-c8da11a7c0e6 None admin] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:09:03.920205 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-266142c6-574f-4072-88df-c8da11a7c0e6 None admin] PATH_INFO: `/v3/users` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:09:03.921102 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-266142c6-574f-4072-88df-c8da11a7c0e6 None admin] RBAC: Adding query filter params (domain_id=default, name=glance) {{(pid=81409) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 13 04:09:03.921622 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-266142c6-574f-4072-88df-c8da11a7c0e6 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:09:03.923167 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-266142c6-574f-4072-88df-c8da11a7c0e6 None admin] RBAC: Authorization granted {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:09:03.940832 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 238/465] 10.222.0.27 () {56 vars in 1137 bytes} [Thu Oct 13 04:09:03 2022] GET /identity/v3/users?domain_id=default&name=glance => generated 372 bytes in 172 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 13 04:09:03.941757 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-266142c6-574f-4072-88df-c8da11a7c0e6 None admin] Authenticating user token {{(pid=81409) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:09:03.970912 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG dbcounter [-] [81408] Writing DB stats keystone:SELECT=100 {{(pid=81408) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:09:03.996033 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 228/466] 10.222.0.27 () {58 vars in 1282 bytes} [Thu Oct 13 04:09:03 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 348 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Oct 13 04:09:03.997634 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f458945e-ddc0-4275-8920-6b601223ae42 None admin] Authenticating user token {{(pid=81408) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:09:04.053850 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-266142c6-574f-4072-88df-c8da11a7c0e6 None admin] Validating token access rules against request {{(pid=81409) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:09:04.090300 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cca59a05-75c3-47b5-b313-674cf7c4e585 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81409) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:09:04.090904 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cca59a05-75c3-47b5-b313-674cf7c4e585 None admin] REQUEST_METHOD: `GET` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:09:04.091172 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cca59a05-75c3-47b5-b313-674cf7c4e585 None admin] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:09:04.091441 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cca59a05-75c3-47b5-b313-674cf7c4e585 None admin] PATH_INFO: `/v3/projects` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:09:04.092114 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cca59a05-75c3-47b5-b313-674cf7c4e585 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81409) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 13 04:09:04.092448 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cca59a05-75c3-47b5-b313-674cf7c4e585 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:09:04.093875 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cca59a05-75c3-47b5-b313-674cf7c4e585 None admin] RBAC: Authorization granted {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:09:04.098829 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f458945e-ddc0-4275-8920-6b601223ae42 None admin] Validating token access rules against request {{(pid=81408) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:09:04.111790 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 239/467] 10.222.0.27 () {56 vars in 1148 bytes} [Thu Oct 13 04:09:03 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 166 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 13 04:09:04.111790 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cca59a05-75c3-47b5-b313-674cf7c4e585 None admin] Authenticating user token {{(pid=81409) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:09:04.126086 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG dbcounter [-] [81409] Writing DB stats keystone:SELECT=100 {{(pid=81409) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:09:04.141436 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7202b36b-d5d7-43e8-9976-ad9ec4ba066e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81408) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:09:04.142068 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7202b36b-d5d7-43e8-9976-ad9ec4ba066e None admin] REQUEST_METHOD: `GET` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:09:04.142373 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7202b36b-d5d7-43e8-9976-ad9ec4ba066e None admin] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:09:04.142668 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7202b36b-d5d7-43e8-9976-ad9ec4ba066e None admin] PATH_INFO: `/v3/users/neutron` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:09:04.150778 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7202b36b-d5d7-43e8-9976-ad9ec4ba066e None admin] RBAC: Authorizing `identity:get_user(user_id=neutron)` {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:09:04.152243 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7202b36b-d5d7-43e8-9976-ad9ec4ba066e None admin] RBAC: Authorization granted {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:09:04.164232 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application [None req-7202b36b-d5d7-43e8-9976-ad9ec4ba066e None admin] Could not find user: neutron.: keystone.exception.UserNotFound: Could not find user: neutron. Oct 13 04:09:04.164232 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 13 04:09:04.164232 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 13 04:09:04.164232 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 13 04:09:04.164232 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 13 04:09:04.164232 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 13 04:09:04.164232 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 13 04:09:04.164232 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 13 04:09:04.164232 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 13 04:09:04.164232 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 13 04:09:04.164232 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 13 04:09:04.164232 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 13 04:09:04.164232 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Oct 13 04:09:04.164232 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application return self._get_user(user_id) Oct 13 04:09:04.164232 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Oct 13 04:09:04.164232 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Oct 13 04:09:04.166114 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 13 04:09:04.166114 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 13 04:09:04.166114 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Oct 13 04:09:04.166114 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 13 04:09:04.166114 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Oct 13 04:09:04.166114 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 13 04:09:04.166114 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 13 04:09:04.166114 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 13 04:09:04.166114 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 13 04:09:04.166114 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application return self.get_or_create( Oct 13 04:09:04.166114 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 13 04:09:04.166114 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application with Lock( Oct 13 04:09:04.166114 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 13 04:09:04.166114 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application return self._enter() Oct 13 04:09:04.166114 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 13 04:09:04.166114 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 13 04:09:04.166114 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 13 04:09:04.166114 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application return self.creator() Oct 13 04:09:04.167084 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 13 04:09:04.167084 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application created_value = creator( Oct 13 04:09:04.167084 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Oct 13 04:09:04.167084 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Oct 13 04:09:04.167084 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Oct 13 04:09:04.167084 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Oct 13 04:09:04.167084 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Oct 13 04:09:04.167084 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Oct 13 04:09:04.167084 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: neutron. Oct 13 04:09:04.167084 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application Oct 13 04:09:04.167084 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 229/468] 10.222.0.27 () {56 vars in 1102 bytes} [Thu Oct 13 04:09:03 2022] GET /identity/v3/users/neutron => generated 85 bytes in 169 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 13 04:09:04.167626 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7202b36b-d5d7-43e8-9976-ad9ec4ba066e None admin] Authenticating user token {{(pid=81408) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:09:04.209755 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG dbcounter [-] [81408] Writing DB stats keystone:SELECT=100 {{(pid=81408) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:09:04.224092 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cca59a05-75c3-47b5-b313-674cf7c4e585 None admin] Validating token access rules against request {{(pid=81409) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:09:04.260707 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e010689f-6287-4841-89b8-6c09bd10737b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81409) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:09:04.261526 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e010689f-6287-4841-89b8-6c09bd10737b None admin] REQUEST_METHOD: `GET` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:09:04.261805 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e010689f-6287-4841-89b8-6c09bd10737b None admin] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:09:04.262404 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e010689f-6287-4841-89b8-6c09bd10737b None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:09:04.281918 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7202b36b-d5d7-43e8-9976-ad9ec4ba066e None admin] Validating token access rules against request {{(pid=81408) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:09:04.323521 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0f826e6d-bc87-4696-8b4b-7cafb97d054a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81408) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:09:04.324167 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f826e6d-bc87-4696-8b4b-7cafb97d054a None admin] REQUEST_METHOD: `GET` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:09:04.324449 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f826e6d-bc87-4696-8b4b-7cafb97d054a None admin] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:09:04.324719 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f826e6d-bc87-4696-8b4b-7cafb97d054a None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:09:04.335500 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0f826e6d-bc87-4696-8b4b-7cafb97d054a None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:09:04.337043 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0f826e6d-bc87-4696-8b4b-7cafb97d054a None admin] RBAC: Authorization granted {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:09:04.345724 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e010689f-6287-4841-89b8-6c09bd10737b None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:09:04.347459 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e010689f-6287-4841-89b8-6c09bd10737b None admin] RBAC: Authorization granted {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:09:04.351143 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application [None req-0f826e6d-bc87-4696-8b4b-7cafb97d054a None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 13 04:09:04.351143 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 13 04:09:04.351143 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 13 04:09:04.351143 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 13 04:09:04.351143 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 13 04:09:04.351143 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 13 04:09:04.351143 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 13 04:09:04.351143 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 13 04:09:04.351143 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 13 04:09:04.351143 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 13 04:09:04.351143 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 13 04:09:04.351143 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 13 04:09:04.351143 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 13 04:09:04.351143 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 13 04:09:04.351143 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 13 04:09:04.351143 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 13 04:09:04.352780 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 13 04:09:04.352780 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 13 04:09:04.352780 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 13 04:09:04.352780 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 13 04:09:04.352780 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 13 04:09:04.352780 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application return self.get_or_create( Oct 13 04:09:04.352780 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 13 04:09:04.352780 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application with Lock( Oct 13 04:09:04.352780 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 13 04:09:04.352780 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application return self._enter() Oct 13 04:09:04.352780 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 13 04:09:04.352780 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 13 04:09:04.352780 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 13 04:09:04.352780 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application return self.creator() Oct 13 04:09:04.352780 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 13 04:09:04.352780 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application created_value = creator( Oct 13 04:09:04.352780 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 13 04:09:04.353607 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 13 04:09:04.353607 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 13 04:09:04.353607 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application Oct 13 04:09:04.353607 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 230/469] 10.222.0.27 () {56 vars in 1108 bytes} [Thu Oct 13 04:09:04 2022] GET /identity/v3/domains/Default => generated 87 bytes in 186 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 13 04:09:04.353825 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0f826e6d-bc87-4696-8b4b-7cafb97d054a None admin] Authenticating user token {{(pid=81408) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:09:04.365025 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG dbcounter [-] [81409] Writing DB stats keystone:SELECT=100 {{(pid=81409) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:09:04.454448 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG dbcounter [-] [81408] Writing DB stats keystone:SELECT=100 {{(pid=81408) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:09:04.466535 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0f826e6d-bc87-4696-8b4b-7cafb97d054a None admin] Validating token access rules against request {{(pid=81408) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:09:04.471993 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 240/470] 10.222.0.27 () {58 vars in 1282 bytes} [Thu Oct 13 04:09:04 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 360 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Oct 13 04:09:04.471993 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e010689f-6287-4841-89b8-6c09bd10737b None admin] Authenticating user token {{(pid=81409) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:09:04.509477 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e0153b2a-a0d8-4d3d-9f7c-90c2989414b8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81408) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:09:04.510085 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0153b2a-a0d8-4d3d-9f7c-90c2989414b8 None admin] REQUEST_METHOD: `GET` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:09:04.510417 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0153b2a-a0d8-4d3d-9f7c-90c2989414b8 None admin] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:09:04.510693 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0153b2a-a0d8-4d3d-9f7c-90c2989414b8 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:09:04.511423 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e0153b2a-a0d8-4d3d-9f7c-90c2989414b8 None admin] RBAC: Adding query filter params (role.id=4cfff1d3d68e49c9b04231a19cf76ede, scope.project.id=dcec0431fb6849499d6aec35a234daa6, user.id=cb740ed7f7684b079e45f887be83f795) {{(pid=81408) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 13 04:09:04.511730 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e0153b2a-a0d8-4d3d-9f7c-90c2989414b8 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:09:04.513159 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e0153b2a-a0d8-4d3d-9f7c-90c2989414b8 None admin] RBAC: Authorization granted {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:09:04.530253 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 231/471] 10.222.0.27 () {56 vars in 1374 bytes} [Thu Oct 13 04:09:04 2022] GET /identity/v3/role_assignments?user.id=cb740ed7f7684b079e45f887be83f795&scope.project.id=dcec0431fb6849499d6aec35a234daa6&role.id=4cfff1d3d68e49c9b04231a19cf76ede => generated 608 bytes in 166 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 13 04:09:04.530253 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e0153b2a-a0d8-4d3d-9f7c-90c2989414b8 None admin] Authenticating user token {{(pid=81408) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:09:04.605401 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e010689f-6287-4841-89b8-6c09bd10737b None admin] Validating token access rules against request {{(pid=81409) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:09:04.626647 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG dbcounter [-] [81409] Writing DB stats keystone:SELECT=100 {{(pid=81409) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:09:04.658990 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e0153b2a-a0d8-4d3d-9f7c-90c2989414b8 None admin] Validating token access rules against request {{(pid=81408) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:09:04.659933 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-00940a67-95e6-485f-a9bc-a7756d41df50 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81409) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:09:04.660972 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00940a67-95e6-485f-a9bc-a7756d41df50 None admin] REQUEST_METHOD: `GET` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:09:04.661343 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00940a67-95e6-485f-a9bc-a7756d41df50 None admin] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:09:04.662599 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00940a67-95e6-485f-a9bc-a7756d41df50 None admin] PATH_INFO: `/v3/users` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:09:04.663348 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-00940a67-95e6-485f-a9bc-a7756d41df50 None admin] RBAC: Adding query filter params (domain_id=default, name=neutron) {{(pid=81409) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 13 04:09:04.663652 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-00940a67-95e6-485f-a9bc-a7756d41df50 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:09:04.667591 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-00940a67-95e6-485f-a9bc-a7756d41df50 None admin] RBAC: Authorization granted {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:09:04.697183 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 241/472] 10.222.0.27 () {56 vars in 1139 bytes} [Thu Oct 13 04:09:04 2022] GET /identity/v3/users?domain_id=default&name=neutron => generated 374 bytes in 229 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 13 04:09:04.698694 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-00940a67-95e6-485f-a9bc-a7756d41df50 None admin] Authenticating user token {{(pid=81409) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:09:04.699675 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-47641b78-7795-4c87-ba34-d7dedc91160d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81408) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:09:04.700329 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47641b78-7795-4c87-ba34-d7dedc91160d None admin] REQUEST_METHOD: `GET` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:09:04.700599 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47641b78-7795-4c87-ba34-d7dedc91160d None admin] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:09:04.700865 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47641b78-7795-4c87-ba34-d7dedc91160d None admin] PATH_INFO: `/v3/domains` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:09:04.701558 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-47641b78-7795-4c87-ba34-d7dedc91160d None admin] RBAC: Adding query filter params (name=Default) {{(pid=81408) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 13 04:09:04.702146 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-47641b78-7795-4c87-ba34-d7dedc91160d None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:09:04.703705 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-47641b78-7795-4c87-ba34-d7dedc91160d None admin] RBAC: Authorization granted {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:09:04.718928 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 232/473] 10.222.0.27 () {56 vars in 1109 bytes} [Thu Oct 13 04:09:04 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 199 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 13 04:09:04.719534 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-47641b78-7795-4c87-ba34-d7dedc91160d None admin] Authenticating user token {{(pid=81408) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:09:04.737390 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG dbcounter [-] [81408] Writing DB stats keystone:SELECT=100 {{(pid=81408) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:09:04.840118 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-00940a67-95e6-485f-a9bc-a7756d41df50 None admin] Validating token access rules against request {{(pid=81409) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:09:04.848272 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-47641b78-7795-4c87-ba34-d7dedc91160d None admin] Validating token access rules against request {{(pid=81408) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:09:04.881545 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e5f16de7-da8b-4080-b8d3-165e02b45b7c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81409) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:09:04.883191 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e5f16de7-da8b-4080-b8d3-165e02b45b7c None admin] REQUEST_METHOD: `GET` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:09:04.883864 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e5f16de7-da8b-4080-b8d3-165e02b45b7c None admin] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:09:04.884276 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e5f16de7-da8b-4080-b8d3-165e02b45b7c None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:09:04.887950 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dfed59be-cd81-4683-bab9-a938924eae0c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81408) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:09:04.888657 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dfed59be-cd81-4683-bab9-a938924eae0c None admin] REQUEST_METHOD: `GET` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:09:04.889029 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dfed59be-cd81-4683-bab9-a938924eae0c None admin] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:09:04.889398 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dfed59be-cd81-4683-bab9-a938924eae0c None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:09:04.896089 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e5f16de7-da8b-4080-b8d3-165e02b45b7c None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:09:04.897715 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e5f16de7-da8b-4080-b8d3-165e02b45b7c None admin] RBAC: Authorization granted {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:09:04.913443 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application [None req-e5f16de7-da8b-4080-b8d3-165e02b45b7c None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 13 04:09:04.913443 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 13 04:09:04.913443 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 13 04:09:04.913443 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 13 04:09:04.913443 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 13 04:09:04.913443 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 13 04:09:04.913443 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 13 04:09:04.913443 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 13 04:09:04.913443 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 13 04:09:04.913443 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 13 04:09:04.913443 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 13 04:09:04.913443 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 13 04:09:04.913443 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 13 04:09:04.913443 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 13 04:09:04.913443 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 13 04:09:04.913443 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 13 04:09:04.914407 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 13 04:09:04.914407 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 13 04:09:04.914407 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 13 04:09:04.914407 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 13 04:09:04.914407 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 13 04:09:04.914407 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application return self.get_or_create( Oct 13 04:09:04.914407 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 13 04:09:04.914407 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application with Lock( Oct 13 04:09:04.914407 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 13 04:09:04.914407 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application return self._enter() Oct 13 04:09:04.914407 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 13 04:09:04.914407 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 13 04:09:04.914407 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 13 04:09:04.914407 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application return self.creator() Oct 13 04:09:04.914407 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 13 04:09:04.914407 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application created_value = creator( Oct 13 04:09:04.914407 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 13 04:09:04.915873 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 13 04:09:04.915873 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 13 04:09:04.915873 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application Oct 13 04:09:04.922160 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 242/474] 10.222.0.27 () {56 vars in 1108 bytes} [Thu Oct 13 04:09:04 2022] GET /identity/v3/domains/Default => generated 87 bytes in 220 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 13 04:09:04.923254 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e5f16de7-da8b-4080-b8d3-165e02b45b7c None admin] Authenticating user token {{(pid=81409) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:09:04.950104 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG dbcounter [-] [81409] Writing DB stats keystone:SELECT=100 {{(pid=81409) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:09:04.968606 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dfed59be-cd81-4683-bab9-a938924eae0c None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:09:04.970441 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dfed59be-cd81-4683-bab9-a938924eae0c None admin] RBAC: Authorization granted {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:09:04.987505 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG dbcounter [-] [81408] Writing DB stats keystone:SELECT=100 {{(pid=81408) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:09:05.035593 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e5f16de7-da8b-4080-b8d3-165e02b45b7c None admin] Validating token access rules against request {{(pid=81409) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:09:05.072421 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3e0bda37-d5c4-4553-9497-35e9207cf0fa None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81409) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:09:05.073423 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e0bda37-d5c4-4553-9497-35e9207cf0fa None admin] REQUEST_METHOD: `GET` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:09:05.073889 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e0bda37-d5c4-4553-9497-35e9207cf0fa None admin] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:09:05.074391 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e0bda37-d5c4-4553-9497-35e9207cf0fa None admin] PATH_INFO: `/v3/projects/service` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:09:05.083160 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3e0bda37-d5c4-4553-9497-35e9207cf0fa None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:09:05.084877 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3e0bda37-d5c4-4553-9497-35e9207cf0fa None admin] RBAC: Authorization granted {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:09:05.090289 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 233/475] 10.222.0.27 () {58 vars in 1282 bytes} [Thu Oct 13 04:09:04 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 371 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Oct 13 04:09:05.095560 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dfed59be-cd81-4683-bab9-a938924eae0c None admin] Authenticating user token {{(pid=81408) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:09:05.098949 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application [None req-3e0bda37-d5c4-4553-9497-35e9207cf0fa None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Oct 13 04:09:05.098949 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 13 04:09:05.098949 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 13 04:09:05.098949 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 13 04:09:05.098949 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 13 04:09:05.098949 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 13 04:09:05.098949 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 13 04:09:05.098949 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 13 04:09:05.098949 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 13 04:09:05.098949 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 13 04:09:05.098949 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 13 04:09:05.098949 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 13 04:09:05.098949 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Oct 13 04:09:05.098949 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application return self._get_project(project_id) Oct 13 04:09:05.098949 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Oct 13 04:09:05.099761 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Oct 13 04:09:05.099761 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 13 04:09:05.099761 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 13 04:09:05.099761 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 13 04:09:05.099761 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 13 04:09:05.099761 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 13 04:09:05.099761 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application return self.get_or_create( Oct 13 04:09:05.099761 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 13 04:09:05.099761 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application with Lock( Oct 13 04:09:05.099761 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 13 04:09:05.099761 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application return self._enter() Oct 13 04:09:05.099761 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 13 04:09:05.099761 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 13 04:09:05.099761 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 13 04:09:05.099761 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application return self.creator() Oct 13 04:09:05.099761 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 13 04:09:05.099761 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application created_value = creator( Oct 13 04:09:05.100835 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Oct 13 04:09:05.100835 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Oct 13 04:09:05.100835 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Oct 13 04:09:05.100835 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Oct 13 04:09:05.100835 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Oct 13 04:09:05.100835 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Oct 13 04:09:05.100835 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Oct 13 04:09:05.100835 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application Oct 13 04:09:05.103445 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 243/476] 10.222.0.27 () {56 vars in 1111 bytes} [Thu Oct 13 04:09:04 2022] GET /identity/v3/projects/service => generated 88 bytes in 181 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 13 04:09:05.104415 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3e0bda37-d5c4-4553-9497-35e9207cf0fa None admin] Authenticating user token {{(pid=81409) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:09:05.191818 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG dbcounter [-] [81409] Writing DB stats keystone:SELECT=100 {{(pid=81409) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:09:05.200859 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dfed59be-cd81-4683-bab9-a938924eae0c None admin] Validating token access rules against request {{(pid=81408) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:09:05.212711 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3e0bda37-d5c4-4553-9497-35e9207cf0fa None admin] Validating token access rules against request {{(pid=81409) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:09:05.215882 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG dbcounter [-] [81408] Writing DB stats keystone:SELECT=100 {{(pid=81408) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:09:05.237902 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-08985444-3e37-45f6-a3af-abccf0eec323 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81408) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:09:05.238524 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08985444-3e37-45f6-a3af-abccf0eec323 None admin] REQUEST_METHOD: `GET` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:09:05.238910 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08985444-3e37-45f6-a3af-abccf0eec323 None admin] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:09:05.239100 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08985444-3e37-45f6-a3af-abccf0eec323 None admin] PATH_INFO: `/v3/domains` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:09:05.239847 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-08985444-3e37-45f6-a3af-abccf0eec323 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81408) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 13 04:09:05.240301 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-08985444-3e37-45f6-a3af-abccf0eec323 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:09:05.241524 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-08985444-3e37-45f6-a3af-abccf0eec323 None admin] RBAC: Authorization granted {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:09:05.251839 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8d4d9d8d-7cb6-4d4d-a501-eb5206794968 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81409) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:09:05.252461 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d4d9d8d-7cb6-4d4d-a501-eb5206794968 None admin] REQUEST_METHOD: `GET` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:09:05.252732 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d4d9d8d-7cb6-4d4d-a501-eb5206794968 None admin] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:09:05.252996 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d4d9d8d-7cb6-4d4d-a501-eb5206794968 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:09:05.253752 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 234/477] 10.222.0.27 () {56 vars in 1109 bytes} [Thu Oct 13 04:09:05 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 163 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 13 04:09:05.254616 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-08985444-3e37-45f6-a3af-abccf0eec323 None admin] Authenticating user token {{(pid=81408) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:09:05.267748 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8d4d9d8d-7cb6-4d4d-a501-eb5206794968 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:09:05.269847 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8d4d9d8d-7cb6-4d4d-a501-eb5206794968 None admin] RBAC: Authorization granted {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:09:05.287865 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application [None req-8d4d9d8d-7cb6-4d4d-a501-eb5206794968 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 13 04:09:05.287865 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 13 04:09:05.287865 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 13 04:09:05.287865 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 13 04:09:05.287865 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 13 04:09:05.287865 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 13 04:09:05.287865 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 13 04:09:05.287865 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 13 04:09:05.287865 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 13 04:09:05.287865 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 13 04:09:05.287865 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 13 04:09:05.287865 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 13 04:09:05.287865 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 13 04:09:05.287865 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 13 04:09:05.287865 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 13 04:09:05.287865 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 13 04:09:05.288971 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 13 04:09:05.288971 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 13 04:09:05.288971 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 13 04:09:05.288971 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 13 04:09:05.288971 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 13 04:09:05.288971 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application return self.get_or_create( Oct 13 04:09:05.288971 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 13 04:09:05.288971 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application with Lock( Oct 13 04:09:05.288971 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 13 04:09:05.288971 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application return self._enter() Oct 13 04:09:05.288971 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 13 04:09:05.288971 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 13 04:09:05.288971 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 13 04:09:05.288971 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application return self.creator() Oct 13 04:09:05.288971 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 13 04:09:05.288971 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application created_value = creator( Oct 13 04:09:05.288971 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 13 04:09:05.290433 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 13 04:09:05.290433 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 13 04:09:05.290433 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application Oct 13 04:09:05.292140 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 244/478] 10.222.0.27 () {56 vars in 1108 bytes} [Thu Oct 13 04:09:05 2022] GET /identity/v3/domains/Default => generated 87 bytes in 188 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 13 04:09:05.298009 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9239497-c445-47b0-a28f-fbd3e025a153 None None] REQUEST_METHOD: `GET` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:09:05.298316 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9239497-c445-47b0-a28f-fbd3e025a153 None None] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:09:05.298588 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9239497-c445-47b0-a28f-fbd3e025a153 None None] PATH_INFO: `/` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:09:05.302248 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 245/479] 10.222.0.27 () {54 vars in 927 bytes} [Thu Oct 13 04:09:05 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 13 04:09:05.302248 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f9239497-c445-47b0-a28f-fbd3e025a153 None None] Authenticating user token {{(pid=81409) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:09:05.366771 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-08985444-3e37-45f6-a3af-abccf0eec323 None admin] Validating token access rules against request {{(pid=81408) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:09:05.403304 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aee2f9ad-a3db-47f2-b812-ffec82eec4fa None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81408) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:09:05.403921 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aee2f9ad-a3db-47f2-b812-ffec82eec4fa None admin] REQUEST_METHOD: `GET` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:09:05.404183 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aee2f9ad-a3db-47f2-b812-ffec82eec4fa None admin] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:09:05.404453 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aee2f9ad-a3db-47f2-b812-ffec82eec4fa None admin] PATH_INFO: `/v3/projects` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:09:05.405135 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aee2f9ad-a3db-47f2-b812-ffec82eec4fa None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81408) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 13 04:09:05.405442 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aee2f9ad-a3db-47f2-b812-ffec82eec4fa None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:09:05.408896 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aee2f9ad-a3db-47f2-b812-ffec82eec4fa None admin] RBAC: Authorization granted {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:09:05.424544 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f9239497-c445-47b0-a28f-fbd3e025a153 None None] Validating token access rules against request {{(pid=81409) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:09:05.425825 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 235/480] 10.222.0.27 () {56 vars in 1148 bytes} [Thu Oct 13 04:09:05 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 171 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 13 04:09:05.426731 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aee2f9ad-a3db-47f2-b812-ffec82eec4fa None admin] Authenticating user token {{(pid=81408) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:09:05.462983 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG dbcounter [-] [81408] Writing DB stats keystone:SELECT=100 {{(pid=81408) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:09:05.463757 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c1ad065c-281a-4d40-b14f-9d44291ff3f2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81409) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:09:05.464393 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1ad065c-281a-4d40-b14f-9d44291ff3f2 None admin] REQUEST_METHOD: `GET` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:09:05.464689 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1ad065c-281a-4d40-b14f-9d44291ff3f2 None admin] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:09:05.465679 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1ad065c-281a-4d40-b14f-9d44291ff3f2 None admin] PATH_INFO: `/v3/projects/service` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:09:05.467381 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG dbcounter [-] [81409] Writing DB stats keystone:SELECT=100 {{(pid=81409) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:09:05.473631 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c1ad065c-281a-4d40-b14f-9d44291ff3f2 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:09:05.475530 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c1ad065c-281a-4d40-b14f-9d44291ff3f2 None admin] RBAC: Authorization granted {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:09:05.487654 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application [None req-c1ad065c-281a-4d40-b14f-9d44291ff3f2 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Oct 13 04:09:05.487654 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 13 04:09:05.487654 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 13 04:09:05.487654 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 13 04:09:05.487654 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 13 04:09:05.487654 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 13 04:09:05.487654 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 13 04:09:05.487654 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 13 04:09:05.487654 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 13 04:09:05.487654 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 13 04:09:05.487654 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 13 04:09:05.487654 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 13 04:09:05.487654 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Oct 13 04:09:05.487654 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application return self._get_project(project_id) Oct 13 04:09:05.487654 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Oct 13 04:09:05.488605 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Oct 13 04:09:05.488605 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 13 04:09:05.488605 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 13 04:09:05.488605 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 13 04:09:05.488605 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 13 04:09:05.488605 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 13 04:09:05.488605 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application return self.get_or_create( Oct 13 04:09:05.488605 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 13 04:09:05.488605 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application with Lock( Oct 13 04:09:05.488605 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 13 04:09:05.488605 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application return self._enter() Oct 13 04:09:05.488605 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 13 04:09:05.488605 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 13 04:09:05.488605 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 13 04:09:05.488605 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application return self.creator() Oct 13 04:09:05.488605 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 13 04:09:05.488605 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application created_value = creator( Oct 13 04:09:05.489457 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Oct 13 04:09:05.489457 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Oct 13 04:09:05.489457 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Oct 13 04:09:05.489457 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Oct 13 04:09:05.489457 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Oct 13 04:09:05.489457 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Oct 13 04:09:05.489457 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Oct 13 04:09:05.489457 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application Oct 13 04:09:05.489457 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 246/481] 10.222.0.27 () {56 vars in 1111 bytes} [Thu Oct 13 04:09:05 2022] GET /identity/v3/projects/service => generated 88 bytes in 188 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 13 04:09:05.490721 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7108374-f276-4469-8523-f18fc468360f None None] REQUEST_METHOD: `POST` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:09:05.490721 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7108374-f276-4469-8523-f18fc468360f None None] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:09:05.490721 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7108374-f276-4469-8523-f18fc468360f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:09:05.510882 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.auth.core [None req-c7108374-f276-4469-8523-f18fc468360f None None] MFA Rules not processed for user `0e2d979563d6421995451a0e86bf3022`. Rule list: `[]` (Enabled: `True`). {{(pid=81409) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 13 04:09:05.512098 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.fernet_utils [None req-c7108374-f276-4469-8523-f18fc468360f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81409) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:09:05.549787 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aee2f9ad-a3db-47f2-b812-ffec82eec4fa None admin] Validating token access rules against request {{(pid=81408) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:09:05.582630 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 247/482] 10.222.0.27 () {58 vars in 1024 bytes} [Thu Oct 13 04:09:05 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 93 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 13 04:09:05.586331 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-198421af-50a1-4478-9654-2e205a622012 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81408) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:09:05.586971 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-198421af-50a1-4478-9654-2e205a622012 None admin] REQUEST_METHOD: `GET` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:09:05.587256 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-198421af-50a1-4478-9654-2e205a622012 None admin] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:09:05.587508 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-198421af-50a1-4478-9654-2e205a622012 None admin] PATH_INFO: `/v3/domains` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:09:05.588151 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-198421af-50a1-4478-9654-2e205a622012 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81408) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 13 04:09:05.588455 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-198421af-50a1-4478-9654-2e205a622012 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:09:05.589822 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-198421af-50a1-4478-9654-2e205a622012 None admin] RBAC: Authorization granted {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:09:05.595387 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c7108374-f276-4469-8523-f18fc468360f None None] Authenticating user token {{(pid=81409) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:09:05.606233 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 236/483] 10.222.0.27 () {56 vars in 1109 bytes} [Thu Oct 13 04:09:05 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 176 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 13 04:09:05.606233 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-198421af-50a1-4478-9654-2e205a622012 None admin] Authenticating user token {{(pid=81408) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:09:05.717913 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c7108374-f276-4469-8523-f18fc468360f None None] Validating token access rules against request {{(pid=81409) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:09:05.738790 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG dbcounter [-] [81408] Writing DB stats keystone:SELECT=100 {{(pid=81408) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:09:05.749803 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG dbcounter [-] [81409] Writing DB stats keystone:SELECT=100 {{(pid=81409) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:09:05.749980 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-198421af-50a1-4478-9654-2e205a622012 None admin] Validating token access rules against request {{(pid=81408) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:09:05.765645 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a12e0825-b575-4d81-bed6-f9c5a9be6bf9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81409) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:09:05.766585 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a12e0825-b575-4d81-bed6-f9c5a9be6bf9 None admin] REQUEST_METHOD: `GET` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:09:05.767106 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a12e0825-b575-4d81-bed6-f9c5a9be6bf9 None admin] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:09:05.767576 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a12e0825-b575-4d81-bed6-f9c5a9be6bf9 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:09:05.769124 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a12e0825-b575-4d81-bed6-f9c5a9be6bf9 None admin] RBAC: Adding query filter params (role.id=4cfff1d3d68e49c9b04231a19cf76ede, scope.project.id=dcec0431fb6849499d6aec35a234daa6, user.id=6141b8a6c3d54e179d6879bb019f4d12) {{(pid=81409) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 13 04:09:05.769637 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a12e0825-b575-4d81-bed6-f9c5a9be6bf9 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:09:05.771297 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a12e0825-b575-4d81-bed6-f9c5a9be6bf9 None admin] RBAC: Authorization granted {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:09:05.780724 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 248/484] 10.222.0.27 () {56 vars in 1374 bytes} [Thu Oct 13 04:09:05 2022] GET /identity/v3/role_assignments?user.id=6141b8a6c3d54e179d6879bb019f4d12&scope.project.id=dcec0431fb6849499d6aec35a234daa6&role.id=4cfff1d3d68e49c9b04231a19cf76ede => generated 608 bytes in 197 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 13 04:09:05.781868 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a12e0825-b575-4d81-bed6-f9c5a9be6bf9 None admin] Authenticating user token {{(pid=81409) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:09:05.798344 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-edf95654-def4-4b3d-9954-6fcdb60001f0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81408) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:09:05.798998 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-edf95654-def4-4b3d-9954-6fcdb60001f0 None admin] REQUEST_METHOD: `GET` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:09:05.799282 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-edf95654-def4-4b3d-9954-6fcdb60001f0 None admin] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:09:05.799556 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-edf95654-def4-4b3d-9954-6fcdb60001f0 None admin] PATH_INFO: `/v3/projects` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:09:05.800318 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-edf95654-def4-4b3d-9954-6fcdb60001f0 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81408) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 13 04:09:05.800685 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-edf95654-def4-4b3d-9954-6fcdb60001f0 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:09:05.802198 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-edf95654-def4-4b3d-9954-6fcdb60001f0 None admin] RBAC: Authorization granted {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:09:05.826805 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 237/485] 10.222.0.27 () {56 vars in 1148 bytes} [Thu Oct 13 04:09:05 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 224 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 13 04:09:05.832417 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-edf95654-def4-4b3d-9954-6fcdb60001f0 None admin] Authenticating user token {{(pid=81408) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:09:05.908087 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a12e0825-b575-4d81-bed6-f9c5a9be6bf9 None admin] Validating token access rules against request {{(pid=81409) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:09:05.956439 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d9404079-79af-464f-b105-586d74f6431d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81409) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:09:05.957505 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9404079-79af-464f-b105-586d74f6431d None admin] REQUEST_METHOD: `GET` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:09:05.957982 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9404079-79af-464f-b105-586d74f6431d None admin] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:09:05.958497 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9404079-79af-464f-b105-586d74f6431d None admin] PATH_INFO: `/v3/projects/service` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:09:05.959542 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-edf95654-def4-4b3d-9954-6fcdb60001f0 None admin] Validating token access rules against request {{(pid=81408) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:09:05.972626 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d9404079-79af-464f-b105-586d74f6431d None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:09:05.974916 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d9404079-79af-464f-b105-586d74f6431d None admin] RBAC: Authorization granted {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:09:05.993137 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application [None req-d9404079-79af-464f-b105-586d74f6431d None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Oct 13 04:09:05.993137 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 13 04:09:05.993137 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 13 04:09:05.993137 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 13 04:09:05.993137 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 13 04:09:05.993137 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 13 04:09:05.993137 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 13 04:09:05.993137 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 13 04:09:05.993137 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 13 04:09:05.993137 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 13 04:09:05.993137 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 13 04:09:05.993137 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 13 04:09:05.993137 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Oct 13 04:09:05.993137 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application return self._get_project(project_id) Oct 13 04:09:05.993137 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Oct 13 04:09:05.994135 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Oct 13 04:09:05.994135 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 13 04:09:05.994135 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 13 04:09:05.994135 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 13 04:09:05.994135 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 13 04:09:05.994135 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 13 04:09:05.994135 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application return self.get_or_create( Oct 13 04:09:05.994135 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 13 04:09:05.994135 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application with Lock( Oct 13 04:09:05.994135 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 13 04:09:05.994135 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application return self._enter() Oct 13 04:09:05.994135 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 13 04:09:05.994135 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 13 04:09:05.994135 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 13 04:09:05.994135 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application return self.creator() Oct 13 04:09:05.994135 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 13 04:09:05.994135 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application created_value = creator( Oct 13 04:09:05.995013 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Oct 13 04:09:05.995013 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Oct 13 04:09:05.995013 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Oct 13 04:09:05.995013 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Oct 13 04:09:05.995013 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Oct 13 04:09:05.995013 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Oct 13 04:09:05.995013 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Oct 13 04:09:05.995013 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application Oct 13 04:09:06.000816 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 249/486] 10.222.0.27 () {56 vars in 1111 bytes} [Thu Oct 13 04:09:05 2022] GET /identity/v3/projects/service => generated 88 bytes in 215 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 13 04:09:06.000816 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d9404079-79af-464f-b105-586d74f6431d None admin] Authenticating user token {{(pid=81409) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:09:06.011550 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6d08c90b-f04e-4fd2-9484-bcd1ad31af63 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81408) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:09:06.012534 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d08c90b-f04e-4fd2-9484-bcd1ad31af63 None admin] REQUEST_METHOD: `GET` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:09:06.013016 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d08c90b-f04e-4fd2-9484-bcd1ad31af63 None admin] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:09:06.013494 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d08c90b-f04e-4fd2-9484-bcd1ad31af63 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:09:06.014948 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6d08c90b-f04e-4fd2-9484-bcd1ad31af63 None admin] RBAC: Adding query filter params (role.id=4cfff1d3d68e49c9b04231a19cf76ede, scope.project.id=dcec0431fb6849499d6aec35a234daa6, user.id=a6700f49b86543f6a5b92cad8faaba42) {{(pid=81408) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 13 04:09:06.015425 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6d08c90b-f04e-4fd2-9484-bcd1ad31af63 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:09:06.017488 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6d08c90b-f04e-4fd2-9484-bcd1ad31af63 None admin] RBAC: Authorization granted {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:09:06.030374 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 238/487] 10.222.0.27 () {56 vars in 1374 bytes} [Thu Oct 13 04:09:05 2022] GET /identity/v3/role_assignments?user.id=a6700f49b86543f6a5b92cad8faaba42&scope.project.id=dcec0431fb6849499d6aec35a234daa6&role.id=4cfff1d3d68e49c9b04231a19cf76ede => generated 608 bytes in 193 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 13 04:09:06.031384 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6d08c90b-f04e-4fd2-9484-bcd1ad31af63 None admin] Authenticating user token {{(pid=81408) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:09:06.062995 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG dbcounter [-] [81409] Writing DB stats keystone:SELECT=100 {{(pid=81409) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:09:06.069822 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG dbcounter [-] [81408] Writing DB stats keystone:SELECT=100 {{(pid=81408) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:09:06.144615 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d9404079-79af-464f-b105-586d74f6431d None admin] Validating token access rules against request {{(pid=81409) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:09:06.187431 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6d08c90b-f04e-4fd2-9484-bcd1ad31af63 None admin] Validating token access rules against request {{(pid=81408) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:09:06.191521 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-66683374-d4b1-4ebf-a891-1938a75e1ab2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81409) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:09:06.192485 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66683374-d4b1-4ebf-a891-1938a75e1ab2 None admin] REQUEST_METHOD: `GET` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:09:06.192969 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66683374-d4b1-4ebf-a891-1938a75e1ab2 None admin] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:09:06.193457 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66683374-d4b1-4ebf-a891-1938a75e1ab2 None admin] PATH_INFO: `/v3/roles/service` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:09:06.212348 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-66683374-d4b1-4ebf-a891-1938a75e1ab2 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 13 04:09:06.225962 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application [None req-66683374-d4b1-4ebf-a891-1938a75e1ab2 None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Oct 13 04:09:06.225962 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 13 04:09:06.225962 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 13 04:09:06.225962 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 13 04:09:06.225962 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 13 04:09:06.225962 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 13 04:09:06.225962 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 13 04:09:06.225962 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 13 04:09:06.225962 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 13 04:09:06.225962 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 13 04:09:06.225962 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 13 04:09:06.225962 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 13 04:09:06.225962 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 13 04:09:06.225962 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 13 04:09:06.225962 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 13 04:09:06.225962 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 13 04:09:06.226963 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 13 04:09:06.226963 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 13 04:09:06.226963 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 13 04:09:06.226963 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 13 04:09:06.226963 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 13 04:09:06.226963 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 13 04:09:06.226963 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 13 04:09:06.226963 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 13 04:09:06.226963 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 13 04:09:06.226963 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application return self.get_or_create( Oct 13 04:09:06.226963 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 13 04:09:06.226963 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application with Lock( Oct 13 04:09:06.226963 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 13 04:09:06.226963 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application return self._enter() Oct 13 04:09:06.226963 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 13 04:09:06.226963 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 13 04:09:06.227860 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 13 04:09:06.227860 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application return self.creator() Oct 13 04:09:06.227860 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 13 04:09:06.227860 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application created_value = creator( Oct 13 04:09:06.227860 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 13 04:09:06.227860 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 13 04:09:06.227860 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 13 04:09:06.227860 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 13 04:09:06.227860 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 13 04:09:06.227860 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 13 04:09:06.227860 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Oct 13 04:09:06.227860 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application Oct 13 04:09:06.232009 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 250/488] 10.222.0.27 () {56 vars in 1102 bytes} [Thu Oct 13 04:09:05 2022] GET /identity/v3/roles/service => generated 85 bytes in 235 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 13 04:09:06.238185 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-66683374-d4b1-4ebf-a891-1938a75e1ab2 None admin] Authenticating user token {{(pid=81409) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:09:06.269970 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c94e5a7c-db61-4fb1-9a8a-2e037a36ea51 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81408) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:09:06.270668 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c94e5a7c-db61-4fb1-9a8a-2e037a36ea51 None admin] REQUEST_METHOD: `GET` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:09:06.270946 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c94e5a7c-db61-4fb1-9a8a-2e037a36ea51 None admin] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:09:06.271221 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c94e5a7c-db61-4fb1-9a8a-2e037a36ea51 None admin] PATH_INFO: `/v3/projects` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:09:06.271933 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c94e5a7c-db61-4fb1-9a8a-2e037a36ea51 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81408) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 13 04:09:06.272227 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c94e5a7c-db61-4fb1-9a8a-2e037a36ea51 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:09:06.273674 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c94e5a7c-db61-4fb1-9a8a-2e037a36ea51 None admin] RBAC: Authorization granted {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:09:06.295025 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 239/489] 10.222.0.27 () {56 vars in 1148 bytes} [Thu Oct 13 04:09:06 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 263 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 13 04:09:06.295542 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c94e5a7c-db61-4fb1-9a8a-2e037a36ea51 None admin] Authenticating user token {{(pid=81408) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:09:06.379766 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-66683374-d4b1-4ebf-a891-1938a75e1ab2 None admin] Validating token access rules against request {{(pid=81409) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:09:06.391559 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG dbcounter [-] [81409] Writing DB stats keystone:SELECT=100 {{(pid=81409) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:09:06.399606 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG dbcounter [-] [81408] Writing DB stats keystone:SELECT=100 {{(pid=81408) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:09:06.423904 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c94e5a7c-db61-4fb1-9a8a-2e037a36ea51 None admin] Validating token access rules against request {{(pid=81408) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:09:06.432948 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4dfddd45-dbac-4de0-a0b0-b963a87fce47 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81409) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:09:06.433845 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4dfddd45-dbac-4de0-a0b0-b963a87fce47 None admin] REQUEST_METHOD: `GET` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:09:06.434073 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4dfddd45-dbac-4de0-a0b0-b963a87fce47 None admin] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:09:06.434464 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4dfddd45-dbac-4de0-a0b0-b963a87fce47 None admin] PATH_INFO: `/v3/roles` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:09:06.435201 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4dfddd45-dbac-4de0-a0b0-b963a87fce47 None admin] RBAC: Adding query filter params (name=service) {{(pid=81409) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 13 04:09:06.435572 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4dfddd45-dbac-4de0-a0b0-b963a87fce47 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:09:06.437120 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4dfddd45-dbac-4de0-a0b0-b963a87fce47 None admin] RBAC: Authorization granted {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:09:06.447698 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 251/490] 10.222.0.27 () {56 vars in 1103 bytes} [Thu Oct 13 04:09:06 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 210 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 13 04:09:06.455235 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4dfddd45-dbac-4de0-a0b0-b963a87fce47 None admin] Authenticating user token {{(pid=81409) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:09:06.464144 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4b43124c-b767-4168-b05b-7d2ba175e6f6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81408) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:09:06.465000 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b43124c-b767-4168-b05b-7d2ba175e6f6 None admin] REQUEST_METHOD: `GET` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:09:06.465477 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b43124c-b767-4168-b05b-7d2ba175e6f6 None admin] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:09:06.465922 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b43124c-b767-4168-b05b-7d2ba175e6f6 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:09:06.466858 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4b43124c-b767-4168-b05b-7d2ba175e6f6 None admin] RBAC: Adding query filter params (role.id=4cfff1d3d68e49c9b04231a19cf76ede, scope.project.id=dcec0431fb6849499d6aec35a234daa6, user.id=d8812c3974f84871bb2d761cf0016257) {{(pid=81408) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 13 04:09:06.467341 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4b43124c-b767-4168-b05b-7d2ba175e6f6 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:09:06.469841 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4b43124c-b767-4168-b05b-7d2ba175e6f6 None admin] RBAC: Authorization granted {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:09:06.476338 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 240/491] 10.222.0.27 () {56 vars in 1374 bytes} [Thu Oct 13 04:09:06 2022] GET /identity/v3/role_assignments?user.id=d8812c3974f84871bb2d761cf0016257&scope.project.id=dcec0431fb6849499d6aec35a234daa6&role.id=4cfff1d3d68e49c9b04231a19cf76ede => generated 608 bytes in 181 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 13 04:09:06.573710 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4dfddd45-dbac-4de0-a0b0-b963a87fce47 None admin] Validating token access rules against request {{(pid=81409) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:09:06.612488 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-df1c4c3f-3dd1-43ff-8d23-eb062fdf1787 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81409) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:09:06.613130 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df1c4c3f-3dd1-43ff-8d23-eb062fdf1787 None admin] REQUEST_METHOD: `GET` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:09:06.613402 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df1c4c3f-3dd1-43ff-8d23-eb062fdf1787 None admin] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:09:06.613667 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df1c4c3f-3dd1-43ff-8d23-eb062fdf1787 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:09:06.626418 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-df1c4c3f-3dd1-43ff-8d23-eb062fdf1787 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:09:06.627961 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-df1c4c3f-3dd1-43ff-8d23-eb062fdf1787 None admin] RBAC: Authorization granted {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:09:06.645365 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application [None req-df1c4c3f-3dd1-43ff-8d23-eb062fdf1787 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 13 04:09:06.645365 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 13 04:09:06.645365 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 13 04:09:06.645365 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 13 04:09:06.645365 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 13 04:09:06.645365 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 13 04:09:06.645365 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 13 04:09:06.645365 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 13 04:09:06.645365 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 13 04:09:06.645365 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 13 04:09:06.645365 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 13 04:09:06.645365 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 13 04:09:06.645365 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 13 04:09:06.645365 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 13 04:09:06.645365 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 13 04:09:06.645365 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 13 04:09:06.646343 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 13 04:09:06.646343 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 13 04:09:06.646343 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 13 04:09:06.646343 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 13 04:09:06.646343 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 13 04:09:06.646343 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application return self.get_or_create( Oct 13 04:09:06.646343 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 13 04:09:06.646343 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application with Lock( Oct 13 04:09:06.646343 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 13 04:09:06.646343 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application return self._enter() Oct 13 04:09:06.646343 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 13 04:09:06.646343 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 13 04:09:06.646343 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 13 04:09:06.646343 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application return self.creator() Oct 13 04:09:06.646343 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 13 04:09:06.646343 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application created_value = creator( Oct 13 04:09:06.646343 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 13 04:09:06.647734 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 13 04:09:06.647734 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 13 04:09:06.647734 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application Oct 13 04:09:06.647734 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 252/492] 10.222.0.27 () {56 vars in 1108 bytes} [Thu Oct 13 04:09:06 2022] GET /identity/v3/domains/Default => generated 87 bytes in 192 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 13 04:09:06.653570 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4b43124c-b767-4168-b05b-7d2ba175e6f6 None admin] Authenticating user token {{(pid=81408) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:09:06.784285 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4b43124c-b767-4168-b05b-7d2ba175e6f6 None admin] Validating token access rules against request {{(pid=81408) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:09:06.822950 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-97cb5660-e769-42fe-abef-87e92761365c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81408) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:09:06.823558 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97cb5660-e769-42fe-abef-87e92761365c None admin] REQUEST_METHOD: `GET` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:09:06.823884 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97cb5660-e769-42fe-abef-87e92761365c None admin] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:09:06.824134 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97cb5660-e769-42fe-abef-87e92761365c None admin] PATH_INFO: `/v3/domains` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:09:06.824737 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-97cb5660-e769-42fe-abef-87e92761365c None admin] RBAC: Adding query filter params (name=Default) {{(pid=81408) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 13 04:09:06.825034 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-97cb5660-e769-42fe-abef-87e92761365c None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:09:06.826435 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-97cb5660-e769-42fe-abef-87e92761365c None admin] RBAC: Authorization granted {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:09:06.843171 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 241/493] 10.222.0.27 () {56 vars in 1109 bytes} [Thu Oct 13 04:09:06 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 190 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 13 04:09:06.843579 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eff0d00c-3a1b-4cb5-8857-9fb780669ccd None None] REQUEST_METHOD: `GET` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:09:06.843852 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eff0d00c-3a1b-4cb5-8857-9fb780669ccd None None] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:09:06.844122 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eff0d00c-3a1b-4cb5-8857-9fb780669ccd None None] PATH_INFO: `/` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:09:06.850512 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 253/494] 10.222.0.27 () {54 vars in 927 bytes} [Thu Oct 13 04:09:06 2022] GET /identity => generated 270 bytes in 8 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 13 04:09:06.851119 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-97cb5660-e769-42fe-abef-87e92761365c None admin] Authenticating user token {{(pid=81408) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:09:06.856185 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b347d3cc-fd69-4192-bd0c-75a8b4d89daa None None] REQUEST_METHOD: `POST` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:09:06.856468 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b347d3cc-fd69-4192-bd0c-75a8b4d89daa None None] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:09:06.856737 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b347d3cc-fd69-4192-bd0c-75a8b4d89daa None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:09:06.859711 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG dbcounter [-] [81408] Writing DB stats keystone:SELECT=100 {{(pid=81408) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:09:06.905054 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.auth.core [None req-b347d3cc-fd69-4192-bd0c-75a8b4d89daa None None] MFA Rules not processed for user `0e2d979563d6421995451a0e86bf3022`. Rule list: `[]` (Enabled: `True`). {{(pid=81409) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 13 04:09:06.906338 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.fernet_utils [None req-b347d3cc-fd69-4192-bd0c-75a8b4d89daa None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81409) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:09:06.941510 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG dbcounter [-] [81409] Writing DB stats keystone:SELECT=100 {{(pid=81409) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:09:07.011841 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-97cb5660-e769-42fe-abef-87e92761365c None admin] Validating token access rules against request {{(pid=81408) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:09:07.057836 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 254/495] 10.222.0.27 () {58 vars in 1024 bytes} [Thu Oct 13 04:09:06 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 202 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 13 04:09:07.073226 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-433d3e8b-b8fd-40c8-8c96-6ed0ecd9dd94 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81408) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:09:07.073868 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-433d3e8b-b8fd-40c8-8c96-6ed0ecd9dd94 None admin] REQUEST_METHOD: `GET` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:09:07.074220 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-433d3e8b-b8fd-40c8-8c96-6ed0ecd9dd94 None admin] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:09:07.074462 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-433d3e8b-b8fd-40c8-8c96-6ed0ecd9dd94 None admin] PATH_INFO: `/v3/users/swift` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:09:07.083987 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-433d3e8b-b8fd-40c8-8c96-6ed0ecd9dd94 None admin] RBAC: Authorizing `identity:get_user(user_id=swift)` {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:09:07.085583 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-433d3e8b-b8fd-40c8-8c96-6ed0ecd9dd94 None admin] RBAC: Authorization granted {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:09:07.098429 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application [None req-433d3e8b-b8fd-40c8-8c96-6ed0ecd9dd94 None admin] Could not find user: swift.: keystone.exception.UserNotFound: Could not find user: swift. Oct 13 04:09:07.098429 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 13 04:09:07.098429 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 13 04:09:07.098429 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 13 04:09:07.098429 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 13 04:09:07.098429 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 13 04:09:07.098429 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 13 04:09:07.098429 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 13 04:09:07.098429 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 13 04:09:07.098429 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 13 04:09:07.098429 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 13 04:09:07.098429 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 13 04:09:07.098429 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Oct 13 04:09:07.098429 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application return self._get_user(user_id) Oct 13 04:09:07.098429 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Oct 13 04:09:07.098429 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Oct 13 04:09:07.099364 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 13 04:09:07.099364 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 13 04:09:07.099364 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Oct 13 04:09:07.099364 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 13 04:09:07.099364 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Oct 13 04:09:07.099364 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 13 04:09:07.099364 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 13 04:09:07.099364 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 13 04:09:07.099364 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 13 04:09:07.099364 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application return self.get_or_create( Oct 13 04:09:07.099364 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 13 04:09:07.099364 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application with Lock( Oct 13 04:09:07.099364 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 13 04:09:07.099364 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application return self._enter() Oct 13 04:09:07.099364 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 13 04:09:07.099364 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 13 04:09:07.099364 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 13 04:09:07.099364 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application return self.creator() Oct 13 04:09:07.100551 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 13 04:09:07.100551 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application created_value = creator( Oct 13 04:09:07.100551 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Oct 13 04:09:07.100551 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Oct 13 04:09:07.100551 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Oct 13 04:09:07.100551 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Oct 13 04:09:07.100551 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Oct 13 04:09:07.100551 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Oct 13 04:09:07.100551 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: swift. Oct 13 04:09:07.100551 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application Oct 13 04:09:07.100551 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 242/496] 10.222.0.27 () {56 vars in 1096 bytes} [Thu Oct 13 04:09:06 2022] GET /identity/v3/users/swift => generated 83 bytes in 249 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 13 04:09:07.107564 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b347d3cc-fd69-4192-bd0c-75a8b4d89daa None None] Authenticating user token {{(pid=81409) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:09:07.316523 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b347d3cc-fd69-4192-bd0c-75a8b4d89daa None None] Validating token access rules against request {{(pid=81409) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:09:07.370905 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0cd2e35f-3290-4d60-8272-066369de454b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81409) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:09:07.371426 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0cd2e35f-3290-4d60-8272-066369de454b None admin] REQUEST_METHOD: `GET` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:09:07.371695 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0cd2e35f-3290-4d60-8272-066369de454b None admin] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:09:07.371960 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0cd2e35f-3290-4d60-8272-066369de454b None admin] PATH_INFO: `/v3/users` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:09:07.372674 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0cd2e35f-3290-4d60-8272-066369de454b None admin] RBAC: Adding query filter params (domain_id=default, name=swift) {{(pid=81409) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 13 04:09:07.372974 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0cd2e35f-3290-4d60-8272-066369de454b None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:09:07.374544 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0cd2e35f-3290-4d60-8272-066369de454b None admin] RBAC: Authorization granted {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:09:07.392669 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 255/497] 10.222.0.27 () {56 vars in 1135 bytes} [Thu Oct 13 04:09:07 2022] GET /identity/v3/users?domain_id=default&name=swift => generated 370 bytes in 285 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 13 04:09:07.398059 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-433d3e8b-b8fd-40c8-8c96-6ed0ecd9dd94 None admin] Authenticating user token {{(pid=81408) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:09:07.519102 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0cd2e35f-3290-4d60-8272-066369de454b None admin] Authenticating user token {{(pid=81409) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:09:07.544564 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG dbcounter [-] [81409] Writing DB stats keystone:SELECT=100 {{(pid=81409) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:09:07.547206 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG dbcounter [-] [81408] Writing DB stats keystone:SELECT=100 {{(pid=81408) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:09:07.624695 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-433d3e8b-b8fd-40c8-8c96-6ed0ecd9dd94 None admin] Validating token access rules against request {{(pid=81408) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:09:07.679686 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4cecf391-4e13-4a7d-997e-a64176d3e8ab None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81408) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:09:07.680803 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4cecf391-4e13-4a7d-997e-a64176d3e8ab None admin] REQUEST_METHOD: `GET` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:09:07.681095 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4cecf391-4e13-4a7d-997e-a64176d3e8ab None admin] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:09:07.681369 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4cecf391-4e13-4a7d-997e-a64176d3e8ab None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:09:07.811093 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0cd2e35f-3290-4d60-8272-066369de454b None admin] Validating token access rules against request {{(pid=81409) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:09:07.842226 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4cecf391-4e13-4a7d-997e-a64176d3e8ab None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:09:07.846228 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4cecf391-4e13-4a7d-997e-a64176d3e8ab None admin] RBAC: Authorization granted {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:09:07.883745 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-41806667-c253-4bdd-8a8e-05b2ff11d122 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81409) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:09:07.884310 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41806667-c253-4bdd-8a8e-05b2ff11d122 None admin] REQUEST_METHOD: `GET` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:09:07.884586 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41806667-c253-4bdd-8a8e-05b2ff11d122 None admin] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:09:07.884856 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41806667-c253-4bdd-8a8e-05b2ff11d122 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:09:07.905328 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-41806667-c253-4bdd-8a8e-05b2ff11d122 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 13 04:09:07.911030 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application [None req-41806667-c253-4bdd-8a8e-05b2ff11d122 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Oct 13 04:09:07.911030 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 13 04:09:07.911030 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 13 04:09:07.911030 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 13 04:09:07.911030 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 13 04:09:07.911030 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 13 04:09:07.911030 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 13 04:09:07.911030 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 13 04:09:07.911030 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 13 04:09:07.911030 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 13 04:09:07.911030 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 13 04:09:07.911030 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 13 04:09:07.911030 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 13 04:09:07.911030 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 13 04:09:07.911030 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 13 04:09:07.911030 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 13 04:09:07.912010 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 13 04:09:07.912010 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 13 04:09:07.912010 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 13 04:09:07.912010 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 13 04:09:07.912010 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 13 04:09:07.912010 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 13 04:09:07.912010 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 13 04:09:07.912010 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 13 04:09:07.912010 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 13 04:09:07.912010 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application return self.get_or_create( Oct 13 04:09:07.912010 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 13 04:09:07.912010 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application with Lock( Oct 13 04:09:07.912010 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 13 04:09:07.912010 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application return self._enter() Oct 13 04:09:07.912010 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 13 04:09:07.912010 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 13 04:09:07.912812 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 13 04:09:07.912812 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application return self.creator() Oct 13 04:09:07.912812 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 13 04:09:07.912812 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application created_value = creator( Oct 13 04:09:07.912812 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 13 04:09:07.912812 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 13 04:09:07.912812 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 13 04:09:07.912812 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 13 04:09:07.912812 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 13 04:09:07.912812 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 13 04:09:07.912812 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Oct 13 04:09:07.912812 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application Oct 13 04:09:07.920321 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 256/498] 10.222.0.27 () {56 vars in 1096 bytes} [Thu Oct 13 04:09:07 2022] GET /identity/v3/roles/admin => generated 83 bytes in 399 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Oct 13 04:09:07.926332 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-41806667-c253-4bdd-8a8e-05b2ff11d122 None admin] Authenticating user token {{(pid=81409) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:09:07.971940 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG dbcounter [-] [81408] Writing DB stats keystone:SELECT=100 {{(pid=81408) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:09:08.026542 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 243/499] 10.222.0.27 () {58 vars in 1282 bytes} [Thu Oct 13 04:09:07 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 629 msecs (HTTP/1.1 200) 6 headers in 358 bytes (2 switches on core 0) Oct 13 04:09:08.035225 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4cecf391-4e13-4a7d-997e-a64176d3e8ab None admin] Authenticating user token {{(pid=81408) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:09:08.089550 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-41806667-c253-4bdd-8a8e-05b2ff11d122 None admin] Validating token access rules against request {{(pid=81409) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:09:08.094395 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG dbcounter [-] [81409] Writing DB stats keystone:SELECT=100 {{(pid=81409) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:09:08.145055 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3b4d8688-4930-4462-9b66-743711c18674 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81409) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:09:08.145589 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b4d8688-4930-4462-9b66-743711c18674 None admin] REQUEST_METHOD: `GET` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:09:08.145869 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b4d8688-4930-4462-9b66-743711c18674 None admin] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:09:08.146146 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b4d8688-4930-4462-9b66-743711c18674 None admin] PATH_INFO: `/v3/roles` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:09:08.146846 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3b4d8688-4930-4462-9b66-743711c18674 None admin] RBAC: Adding query filter params (name=admin) {{(pid=81409) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 13 04:09:08.147155 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3b4d8688-4930-4462-9b66-743711c18674 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:09:08.148627 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3b4d8688-4930-4462-9b66-743711c18674 None admin] RBAC: Authorization granted {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:09:08.157697 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 257/500] 10.222.0.27 () {56 vars in 1099 bytes} [Thu Oct 13 04:09:07 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 236 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 13 04:09:08.163309 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3b4d8688-4930-4462-9b66-743711c18674 None admin] Authenticating user token {{(pid=81409) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:09:08.235205 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4cecf391-4e13-4a7d-997e-a64176d3e8ab None admin] Validating token access rules against request {{(pid=81408) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:09:08.296026 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ea1f6d02-dff6-4be6-9317-4548d6918b5d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81408) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:09:08.297704 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea1f6d02-dff6-4be6-9317-4548d6918b5d None admin] REQUEST_METHOD: `GET` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:09:08.301824 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea1f6d02-dff6-4be6-9317-4548d6918b5d None admin] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:09:08.302276 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea1f6d02-dff6-4be6-9317-4548d6918b5d None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:09:08.318723 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ea1f6d02-dff6-4be6-9317-4548d6918b5d None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:09:08.321857 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ea1f6d02-dff6-4be6-9317-4548d6918b5d None admin] RBAC: Authorization granted {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:09:08.346860 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application [None req-ea1f6d02-dff6-4be6-9317-4548d6918b5d None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 13 04:09:08.346860 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 13 04:09:08.346860 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 13 04:09:08.346860 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 13 04:09:08.346860 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 13 04:09:08.346860 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 13 04:09:08.346860 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 13 04:09:08.346860 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 13 04:09:08.346860 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 13 04:09:08.346860 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 13 04:09:08.346860 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 13 04:09:08.346860 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 13 04:09:08.346860 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 13 04:09:08.346860 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 13 04:09:08.346860 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 13 04:09:08.346860 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 13 04:09:08.348591 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 13 04:09:08.348591 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 13 04:09:08.348591 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 13 04:09:08.348591 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 13 04:09:08.348591 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 13 04:09:08.348591 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application return self.get_or_create( Oct 13 04:09:08.348591 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 13 04:09:08.348591 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application with Lock( Oct 13 04:09:08.348591 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 13 04:09:08.348591 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application return self._enter() Oct 13 04:09:08.348591 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 13 04:09:08.348591 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 13 04:09:08.348591 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 13 04:09:08.348591 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application return self.creator() Oct 13 04:09:08.348591 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 13 04:09:08.348591 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application created_value = creator( Oct 13 04:09:08.348591 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 13 04:09:08.349670 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 13 04:09:08.349670 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 13 04:09:08.349670 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application Oct 13 04:09:08.355188 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 244/501] 10.222.0.27 () {56 vars in 1108 bytes} [Thu Oct 13 04:09:08 2022] GET /identity/v3/domains/Default => generated 87 bytes in 320 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 13 04:09:08.357611 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3b4d8688-4930-4462-9b66-743711c18674 None admin] Validating token access rules against request {{(pid=81409) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:09:08.361092 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ea1f6d02-dff6-4be6-9317-4548d6918b5d None admin] Authenticating user token {{(pid=81408) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:09:08.391380 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG dbcounter [-] [81408] Writing DB stats keystone:SELECT=100 {{(pid=81408) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:09:08.404166 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG dbcounter [-] [81409] Writing DB stats keystone:SELECT=100 {{(pid=81409) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:09:08.412128 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e7eaef3-e395-4dc6-b834-96279c106646 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81409) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:09:08.413008 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e7eaef3-e395-4dc6-b834-96279c106646 None admin] REQUEST_METHOD: `GET` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:09:08.413435 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e7eaef3-e395-4dc6-b834-96279c106646 None admin] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:09:08.413848 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e7eaef3-e395-4dc6-b834-96279c106646 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:09:08.426487 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1e7eaef3-e395-4dc6-b834-96279c106646 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:09:08.428267 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1e7eaef3-e395-4dc6-b834-96279c106646 None admin] RBAC: Authorization granted {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:09:08.444196 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application [None req-1e7eaef3-e395-4dc6-b834-96279c106646 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 13 04:09:08.444196 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 13 04:09:08.444196 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 13 04:09:08.444196 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 13 04:09:08.444196 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 13 04:09:08.444196 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 13 04:09:08.444196 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 13 04:09:08.444196 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 13 04:09:08.444196 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 13 04:09:08.444196 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 13 04:09:08.444196 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 13 04:09:08.444196 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 13 04:09:08.444196 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 13 04:09:08.444196 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 13 04:09:08.444196 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 13 04:09:08.444196 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 13 04:09:08.445365 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 13 04:09:08.445365 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 13 04:09:08.445365 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 13 04:09:08.445365 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 13 04:09:08.445365 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 13 04:09:08.445365 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application return self.get_or_create( Oct 13 04:09:08.445365 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 13 04:09:08.445365 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application with Lock( Oct 13 04:09:08.445365 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 13 04:09:08.445365 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application return self._enter() Oct 13 04:09:08.445365 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 13 04:09:08.445365 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 13 04:09:08.445365 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 13 04:09:08.445365 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application return self.creator() Oct 13 04:09:08.445365 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 13 04:09:08.445365 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application created_value = creator( Oct 13 04:09:08.445365 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 13 04:09:08.446736 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 13 04:09:08.446736 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 13 04:09:08.446736 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application Oct 13 04:09:08.453218 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 258/502] 10.222.0.27 () {56 vars in 1108 bytes} [Thu Oct 13 04:09:08 2022] GET /identity/v3/domains/Default => generated 87 bytes in 286 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 13 04:09:08.454722 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e7eaef3-e395-4dc6-b834-96279c106646 None admin] Authenticating user token {{(pid=81409) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:09:08.589259 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ea1f6d02-dff6-4be6-9317-4548d6918b5d None admin] Validating token access rules against request {{(pid=81408) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:09:08.606511 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e7eaef3-e395-4dc6-b834-96279c106646 None admin] Validating token access rules against request {{(pid=81409) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:09:08.661201 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-048f0dbc-d0c4-4120-9084-2a12b1e662ca None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81408) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:09:08.661962 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-048f0dbc-d0c4-4120-9084-2a12b1e662ca None admin] REQUEST_METHOD: `GET` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:09:08.662083 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-048f0dbc-d0c4-4120-9084-2a12b1e662ca None admin] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:09:08.662404 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-048f0dbc-d0c4-4120-9084-2a12b1e662ca None admin] PATH_INFO: `/v3/domains` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:09:08.663055 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-048f0dbc-d0c4-4120-9084-2a12b1e662ca None admin] RBAC: Adding query filter params (name=Default) {{(pid=81408) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 13 04:09:08.663361 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-048f0dbc-d0c4-4120-9084-2a12b1e662ca None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:09:08.665024 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-048f0dbc-d0c4-4120-9084-2a12b1e662ca None admin] RBAC: Authorization granted {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:09:08.667866 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bb2dc0b1-fbb2-4fd2-a24b-acfa93a618ba None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81409) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:09:08.668645 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb2dc0b1-fbb2-4fd2-a24b-acfa93a618ba None admin] REQUEST_METHOD: `GET` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:09:08.669066 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb2dc0b1-fbb2-4fd2-a24b-acfa93a618ba None admin] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:09:08.669515 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb2dc0b1-fbb2-4fd2-a24b-acfa93a618ba None admin] PATH_INFO: `/v3/domains` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:09:08.670360 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bb2dc0b1-fbb2-4fd2-a24b-acfa93a618ba None admin] RBAC: Adding query filter params (name=Default) {{(pid=81409) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 13 04:09:08.670809 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bb2dc0b1-fbb2-4fd2-a24b-acfa93a618ba None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:09:08.672323 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bb2dc0b1-fbb2-4fd2-a24b-acfa93a618ba None admin] RBAC: Authorization granted {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:09:08.678184 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 245/503] 10.222.0.27 () {56 vars in 1109 bytes} [Thu Oct 13 04:09:08 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 317 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 13 04:09:08.679446 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-933c7cb2-bad8-4e43-976b-5fcea585c992 None None] REQUEST_METHOD: `GET` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:09:08.679728 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-933c7cb2-bad8-4e43-976b-5fcea585c992 None None] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:09:08.679999 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-933c7cb2-bad8-4e43-976b-5fcea585c992 None None] PATH_INFO: `/` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:09:08.685994 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 246/504] 10.222.0.27 () {54 vars in 927 bytes} [Thu Oct 13 04:09:08 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 13 04:09:08.686865 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-933c7cb2-bad8-4e43-976b-5fcea585c992 None None] Authenticating user token {{(pid=81408) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:09:08.687961 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 259/505] 10.222.0.27 () {56 vars in 1109 bytes} [Thu Oct 13 04:09:08 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 233 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 13 04:09:08.689543 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80281dbe-7bb8-4e5c-8712-f5983f768716 None None] REQUEST_METHOD: `POST` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:09:08.689995 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80281dbe-7bb8-4e5c-8712-f5983f768716 None None] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:09:08.690432 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80281dbe-7bb8-4e5c-8712-f5983f768716 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:09:08.709918 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.auth.core [None req-80281dbe-7bb8-4e5c-8712-f5983f768716 None None] MFA Rules not processed for user `0e2d979563d6421995451a0e86bf3022`. Rule list: `[]` (Enabled: `True`). {{(pid=81409) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 13 04:09:08.711415 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.fernet_utils [None req-80281dbe-7bb8-4e5c-8712-f5983f768716 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81409) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:09:08.757052 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG dbcounter [-] [81409] Writing DB stats keystone:SELECT=100 {{(pid=81409) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:09:08.777543 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG dbcounter [-] [81408] Writing DB stats keystone:SELECT=100 {{(pid=81408) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:09:08.790730 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 260/506] 10.222.0.27 () {58 vars in 1024 bytes} [Thu Oct 13 04:09:08 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 102 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 13 04:09:08.791748 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-80281dbe-7bb8-4e5c-8712-f5983f768716 None None] Authenticating user token {{(pid=81409) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:09:08.806905 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-933c7cb2-bad8-4e43-976b-5fcea585c992 None None] Validating token access rules against request {{(pid=81408) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:09:08.844866 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7c5d58b7-8295-438d-b4b3-f9420c93f029 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81408) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:09:08.845438 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c5d58b7-8295-438d-b4b3-f9420c93f029 None admin] REQUEST_METHOD: `GET` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:09:08.845724 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c5d58b7-8295-438d-b4b3-f9420c93f029 None admin] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:09:08.845950 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c5d58b7-8295-438d-b4b3-f9420c93f029 None admin] PATH_INFO: `/v3/projects/service` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:09:08.854624 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7c5d58b7-8295-438d-b4b3-f9420c93f029 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:09:08.856185 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7c5d58b7-8295-438d-b4b3-f9420c93f029 None admin] RBAC: Authorization granted {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:09:08.869573 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application [None req-7c5d58b7-8295-438d-b4b3-f9420c93f029 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Oct 13 04:09:08.869573 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 13 04:09:08.869573 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 13 04:09:08.869573 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 13 04:09:08.869573 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 13 04:09:08.869573 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 13 04:09:08.869573 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 13 04:09:08.869573 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 13 04:09:08.869573 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 13 04:09:08.869573 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 13 04:09:08.869573 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 13 04:09:08.869573 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 13 04:09:08.869573 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Oct 13 04:09:08.869573 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application return self._get_project(project_id) Oct 13 04:09:08.869573 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Oct 13 04:09:08.870570 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Oct 13 04:09:08.870570 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 13 04:09:08.870570 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 13 04:09:08.870570 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 13 04:09:08.870570 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 13 04:09:08.870570 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 13 04:09:08.870570 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application return self.get_or_create( Oct 13 04:09:08.870570 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 13 04:09:08.870570 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application with Lock( Oct 13 04:09:08.870570 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 13 04:09:08.870570 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application return self._enter() Oct 13 04:09:08.870570 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 13 04:09:08.870570 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 13 04:09:08.870570 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 13 04:09:08.870570 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application return self.creator() Oct 13 04:09:08.870570 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 13 04:09:08.870570 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application created_value = creator( Oct 13 04:09:08.871459 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Oct 13 04:09:08.871459 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Oct 13 04:09:08.871459 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Oct 13 04:09:08.871459 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Oct 13 04:09:08.871459 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Oct 13 04:09:08.871459 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Oct 13 04:09:08.871459 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Oct 13 04:09:08.871459 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application Oct 13 04:09:08.875643 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 247/507] 10.222.0.27 () {56 vars in 1111 bytes} [Thu Oct 13 04:09:08 2022] GET /identity/v3/projects/service => generated 88 bytes in 184 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 13 04:09:08.877106 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7c5d58b7-8295-438d-b4b3-f9420c93f029 None admin] Authenticating user token {{(pid=81408) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:09:08.924602 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-80281dbe-7bb8-4e5c-8712-f5983f768716 None None] Validating token access rules against request {{(pid=81409) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:09:08.985309 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-707fbf7a-c14c-479b-b104-f460b1fea59d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81409) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:09:08.988568 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-707fbf7a-c14c-479b-b104-f460b1fea59d None admin] REQUEST_METHOD: `GET` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:09:08.988850 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-707fbf7a-c14c-479b-b104-f460b1fea59d None admin] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:09:08.989123 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-707fbf7a-c14c-479b-b104-f460b1fea59d None admin] PATH_INFO: `/v3/users/nova` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:09:09.000565 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-707fbf7a-c14c-479b-b104-f460b1fea59d None admin] RBAC: Authorizing `identity:get_user(user_id=nova)` {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:09:09.002017 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-707fbf7a-c14c-479b-b104-f460b1fea59d None admin] RBAC: Authorization granted {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:09:09.012812 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7c5d58b7-8295-438d-b4b3-f9420c93f029 None admin] Validating token access rules against request {{(pid=81408) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:09:09.016377 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application [None req-707fbf7a-c14c-479b-b104-f460b1fea59d None admin] Could not find user: nova.: keystone.exception.UserNotFound: Could not find user: nova. Oct 13 04:09:09.016377 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 13 04:09:09.016377 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 13 04:09:09.016377 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 13 04:09:09.016377 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 13 04:09:09.016377 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 13 04:09:09.016377 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 13 04:09:09.016377 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 13 04:09:09.016377 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 13 04:09:09.016377 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 13 04:09:09.016377 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 13 04:09:09.016377 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 13 04:09:09.016377 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Oct 13 04:09:09.016377 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application return self._get_user(user_id) Oct 13 04:09:09.016377 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Oct 13 04:09:09.016377 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Oct 13 04:09:09.017281 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 13 04:09:09.017281 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 13 04:09:09.017281 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Oct 13 04:09:09.017281 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 13 04:09:09.017281 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Oct 13 04:09:09.017281 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 13 04:09:09.017281 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 13 04:09:09.017281 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 13 04:09:09.017281 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 13 04:09:09.017281 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application return self.get_or_create( Oct 13 04:09:09.017281 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 13 04:09:09.017281 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application with Lock( Oct 13 04:09:09.017281 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 13 04:09:09.017281 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application return self._enter() Oct 13 04:09:09.017281 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 13 04:09:09.017281 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 13 04:09:09.017281 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 13 04:09:09.017281 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application return self.creator() Oct 13 04:09:09.018238 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 13 04:09:09.018238 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application created_value = creator( Oct 13 04:09:09.018238 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Oct 13 04:09:09.018238 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Oct 13 04:09:09.018238 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Oct 13 04:09:09.018238 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Oct 13 04:09:09.018238 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Oct 13 04:09:09.018238 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Oct 13 04:09:09.018238 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: nova. Oct 13 04:09:09.018238 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application Oct 13 04:09:09.018238 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 261/508] 10.222.0.27 () {56 vars in 1093 bytes} [Thu Oct 13 04:09:08 2022] GET /identity/v3/users/nova => generated 82 bytes in 226 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 13 04:09:09.019282 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bfaf0562-2c47-4912-8919-8319afcb7919 None None] REQUEST_METHOD: `GET` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:09:09.019437 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bfaf0562-2c47-4912-8919-8319afcb7919 None None] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:09:09.019674 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bfaf0562-2c47-4912-8919-8319afcb7919 None None] PATH_INFO: `/` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:09:09.021032 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 262/509] 10.222.0.27 () {54 vars in 927 bytes} [Thu Oct 13 04:09:09 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 13 04:09:09.026420 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bfaf0562-2c47-4912-8919-8319afcb7919 None None] Authenticating user token {{(pid=81409) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:09:09.065109 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e79eb27d-5266-4842-85e9-83f1d0bfbe6d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81408) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:09:09.065702 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e79eb27d-5266-4842-85e9-83f1d0bfbe6d None admin] REQUEST_METHOD: `GET` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:09:09.066561 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e79eb27d-5266-4842-85e9-83f1d0bfbe6d None admin] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:09:09.066835 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e79eb27d-5266-4842-85e9-83f1d0bfbe6d None admin] PATH_INFO: `/v3/projects` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:09:09.067516 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e79eb27d-5266-4842-85e9-83f1d0bfbe6d None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81408) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 13 04:09:09.067818 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e79eb27d-5266-4842-85e9-83f1d0bfbe6d None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:09:09.069253 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e79eb27d-5266-4842-85e9-83f1d0bfbe6d None admin] RBAC: Authorization granted {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:09:09.071524 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG dbcounter [-] [81408] Writing DB stats keystone:SELECT=100 {{(pid=81408) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:09:09.074744 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG dbcounter [-] [81409] Writing DB stats keystone:SELECT=100 {{(pid=81409) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:09:09.089335 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 248/510] 10.222.0.27 () {56 vars in 1148 bytes} [Thu Oct 13 04:09:08 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 208 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 13 04:09:09.090780 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ee47c69-6bf2-4723-aa30-57e8a70cc20a None None] REQUEST_METHOD: `POST` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:09:09.091134 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ee47c69-6bf2-4723-aa30-57e8a70cc20a None None] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:09:09.091475 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ee47c69-6bf2-4723-aa30-57e8a70cc20a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:09:09.111517 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.auth.core [None req-2ee47c69-6bf2-4723-aa30-57e8a70cc20a None None] MFA Rules not processed for user `0e2d979563d6421995451a0e86bf3022`. Rule list: `[]` (Enabled: `True`). {{(pid=81408) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 13 04:09:09.112852 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.fernet_utils [None req-2ee47c69-6bf2-4723-aa30-57e8a70cc20a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81408) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:09:09.159016 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bfaf0562-2c47-4912-8919-8319afcb7919 None None] Validating token access rules against request {{(pid=81409) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:09:09.187853 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 249/511] 10.222.0.27 () {58 vars in 1024 bytes} [Thu Oct 13 04:09:09 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 98 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 13 04:09:09.188651 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2ee47c69-6bf2-4723-aa30-57e8a70cc20a None None] Authenticating user token {{(pid=81408) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:09:09.211078 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2956c15d-a75b-498a-8a35-f2a1fd881411 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81409) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:09:09.212123 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2956c15d-a75b-498a-8a35-f2a1fd881411 None admin] REQUEST_METHOD: `GET` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:09:09.212399 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2956c15d-a75b-498a-8a35-f2a1fd881411 None admin] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:09:09.212666 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2956c15d-a75b-498a-8a35-f2a1fd881411 None admin] PATH_INFO: `/v3/users` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:09:09.213467 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2956c15d-a75b-498a-8a35-f2a1fd881411 None admin] RBAC: Adding query filter params (domain_id=default, name=nova) {{(pid=81409) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 13 04:09:09.213768 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2956c15d-a75b-498a-8a35-f2a1fd881411 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:09:09.215560 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2956c15d-a75b-498a-8a35-f2a1fd881411 None admin] RBAC: Authorization granted {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:09:09.238199 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 263/512] 10.222.0.27 () {56 vars in 1133 bytes} [Thu Oct 13 04:09:09 2022] GET /identity/v3/users?domain_id=default&name=nova => generated 368 bytes in 212 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 13 04:09:09.239104 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2956c15d-a75b-498a-8a35-f2a1fd881411 None admin] Authenticating user token {{(pid=81409) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:09:09.316790 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2ee47c69-6bf2-4723-aa30-57e8a70cc20a None None] Validating token access rules against request {{(pid=81408) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:09:09.341563 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG dbcounter [-] [81408] Writing DB stats keystone:SELECT=100 {{(pid=81408) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:09:09.348323 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG dbcounter [-] [81409] Writing DB stats keystone:SELECT=100 {{(pid=81409) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:09:09.353223 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5079e52d-4fb2-42d1-8c3e-a92545244d70 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81408) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:09:09.353919 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5079e52d-4fb2-42d1-8c3e-a92545244d70 None admin] REQUEST_METHOD: `GET` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:09:09.354198 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5079e52d-4fb2-42d1-8c3e-a92545244d70 None admin] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:09:09.354488 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5079e52d-4fb2-42d1-8c3e-a92545244d70 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:09:09.355205 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5079e52d-4fb2-42d1-8c3e-a92545244d70 None admin] RBAC: Adding query filter params (role.id=4cfff1d3d68e49c9b04231a19cf76ede, scope.project.id=dcec0431fb6849499d6aec35a234daa6, user.id=90a0f466b664458898cf9dc4112ae651) {{(pid=81408) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 13 04:09:09.355510 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5079e52d-4fb2-42d1-8c3e-a92545244d70 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:09:09.356930 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5079e52d-4fb2-42d1-8c3e-a92545244d70 None admin] RBAC: Authorization granted {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:09:09.358689 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2956c15d-a75b-498a-8a35-f2a1fd881411 None admin] Validating token access rules against request {{(pid=81409) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:09:09.362604 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 250/513] 10.222.0.27 () {56 vars in 1374 bytes} [Thu Oct 13 04:09:09 2022] GET /identity/v3/role_assignments?user.id=90a0f466b664458898cf9dc4112ae651&scope.project.id=dcec0431fb6849499d6aec35a234daa6&role.id=4cfff1d3d68e49c9b04231a19cf76ede => generated 608 bytes in 174 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 13 04:09:09.362604 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5079e52d-4fb2-42d1-8c3e-a92545244d70 None admin] Authenticating user token {{(pid=81408) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:09:09.424064 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1a3e3890-2880-44a6-b97b-9e1056e116b5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81409) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:09:09.424719 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a3e3890-2880-44a6-b97b-9e1056e116b5 None admin] REQUEST_METHOD: `GET` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:09:09.424999 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a3e3890-2880-44a6-b97b-9e1056e116b5 None admin] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:09:09.425271 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a3e3890-2880-44a6-b97b-9e1056e116b5 None admin] PATH_INFO: `/v3/services/block-storage` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:09:09.425870 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1a3e3890-2880-44a6-b97b-9e1056e116b5 None admin] RBAC: Authorizing `identity:get_service(service_id=block-storage)` {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:09:09.428199 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1a3e3890-2880-44a6-b97b-9e1056e116b5 None admin] RBAC: Authorization granted {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:09:09.442940 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application [None req-1a3e3890-2880-44a6-b97b-9e1056e116b5 None admin] Could not find service: block-storage.: keystone.exception.ServiceNotFound: Could not find service: block-storage. Oct 13 04:09:09.442940 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 13 04:09:09.442940 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Oct 13 04:09:09.442940 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Oct 13 04:09:09.442940 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Oct 13 04:09:09.442940 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Oct 13 04:09:09.442940 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Oct 13 04:09:09.442940 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 13 04:09:09.442940 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: block-storage. Oct 13 04:09:09.442940 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application Oct 13 04:09:09.442940 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Oct 13 04:09:09.442940 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application Oct 13 04:09:09.442940 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 13 04:09:09.442940 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 13 04:09:09.442940 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 13 04:09:09.442940 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 13 04:09:09.442940 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 13 04:09:09.444149 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 13 04:09:09.444149 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 13 04:09:09.444149 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 13 04:09:09.444149 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 13 04:09:09.444149 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 13 04:09:09.444149 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 13 04:09:09.444149 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Oct 13 04:09:09.444149 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application return self._get_service(service_id) Oct 13 04:09:09.444149 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Oct 13 04:09:09.444149 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Oct 13 04:09:09.444149 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 13 04:09:09.444149 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 13 04:09:09.444149 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 13 04:09:09.444149 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 13 04:09:09.444149 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 13 04:09:09.444149 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application return self.get_or_create( Oct 13 04:09:09.445576 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 13 04:09:09.445576 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application with Lock( Oct 13 04:09:09.445576 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 13 04:09:09.445576 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application return self._enter() Oct 13 04:09:09.445576 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 13 04:09:09.445576 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 13 04:09:09.445576 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 13 04:09:09.445576 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application return self.creator() Oct 13 04:09:09.445576 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 13 04:09:09.445576 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application created_value = creator( Oct 13 04:09:09.445576 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Oct 13 04:09:09.445576 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 13 04:09:09.445576 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: block-storage. Oct 13 04:09:09.445576 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application Oct 13 04:09:09.445576 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 264/514] 10.222.0.27 () {56 vars in 1129 bytes} [Thu Oct 13 04:09:09 2022] GET /identity/v3/services/block-storage => generated 94 bytes in 205 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 13 04:09:09.446959 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13e76f0c-333e-4cd2-bb9a-75950a118d8f None None] REQUEST_METHOD: `GET` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:09:09.446959 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13e76f0c-333e-4cd2-bb9a-75950a118d8f None None] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:09:09.446959 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13e76f0c-333e-4cd2-bb9a-75950a118d8f None None] PATH_INFO: `/` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:09:09.449639 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 265/515] 10.222.0.27 () {54 vars in 927 bytes} [Thu Oct 13 04:09:09 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 13 04:09:09.450320 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-13e76f0c-333e-4cd2-bb9a-75950a118d8f None None] Authenticating user token {{(pid=81409) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:09:09.497123 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5079e52d-4fb2-42d1-8c3e-a92545244d70 None admin] Validating token access rules against request {{(pid=81408) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:09:09.538018 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dd346ad5-2358-4ecd-a424-ad3b9ea2eacf None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81408) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:09:09.538683 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd346ad5-2358-4ecd-a424-ad3b9ea2eacf None admin] REQUEST_METHOD: `GET` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:09:09.538962 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd346ad5-2358-4ecd-a424-ad3b9ea2eacf None admin] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:09:09.539225 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd346ad5-2358-4ecd-a424-ad3b9ea2eacf None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:09:09.551029 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dd346ad5-2358-4ecd-a424-ad3b9ea2eacf None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:09:09.552494 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dd346ad5-2358-4ecd-a424-ad3b9ea2eacf None admin] RBAC: Authorization granted {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:09:09.569097 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application [None req-dd346ad5-2358-4ecd-a424-ad3b9ea2eacf None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 13 04:09:09.569097 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 13 04:09:09.569097 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 13 04:09:09.569097 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 13 04:09:09.569097 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 13 04:09:09.569097 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 13 04:09:09.569097 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 13 04:09:09.569097 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 13 04:09:09.569097 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 13 04:09:09.569097 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 13 04:09:09.569097 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 13 04:09:09.569097 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 13 04:09:09.569097 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 13 04:09:09.569097 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 13 04:09:09.569097 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 13 04:09:09.569097 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 13 04:09:09.570103 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 13 04:09:09.570103 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 13 04:09:09.570103 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 13 04:09:09.570103 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 13 04:09:09.570103 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 13 04:09:09.570103 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application return self.get_or_create( Oct 13 04:09:09.570103 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 13 04:09:09.570103 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application with Lock( Oct 13 04:09:09.570103 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 13 04:09:09.570103 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application return self._enter() Oct 13 04:09:09.570103 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 13 04:09:09.570103 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 13 04:09:09.570103 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 13 04:09:09.570103 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application return self.creator() Oct 13 04:09:09.570103 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 13 04:09:09.570103 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application created_value = creator( Oct 13 04:09:09.570103 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 13 04:09:09.571619 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 13 04:09:09.571619 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 13 04:09:09.571619 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application Oct 13 04:09:09.571619 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 251/516] 10.222.0.27 () {56 vars in 1108 bytes} [Thu Oct 13 04:09:09 2022] GET /identity/v3/domains/Default => generated 87 bytes in 207 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 13 04:09:09.572519 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c15f7856-7579-45f7-b1b4-0f1c0e471013 None None] REQUEST_METHOD: `POST` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:09:09.572519 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c15f7856-7579-45f7-b1b4-0f1c0e471013 None None] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:09:09.572759 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c15f7856-7579-45f7-b1b4-0f1c0e471013 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:09:09.582118 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-13e76f0c-333e-4cd2-bb9a-75950a118d8f None None] Validating token access rules against request {{(pid=81409) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:09:09.591559 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.auth.core [None req-c15f7856-7579-45f7-b1b4-0f1c0e471013 None None] MFA Rules not processed for user `0e2d979563d6421995451a0e86bf3022`. Rule list: `[]` (Enabled: `True`). {{(pid=81408) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 13 04:09:09.592749 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.fernet_utils [None req-c15f7856-7579-45f7-b1b4-0f1c0e471013 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81408) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:09:09.619460 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fe539ca0-2722-4c65-ad89-3b07724104fa None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81409) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:09:09.620167 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe539ca0-2722-4c65-ad89-3b07724104fa None admin] REQUEST_METHOD: `GET` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:09:09.620438 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe539ca0-2722-4c65-ad89-3b07724104fa None admin] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:09:09.620723 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe539ca0-2722-4c65-ad89-3b07724104fa None admin] PATH_INFO: `/v3/services` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:09:09.621320 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fe539ca0-2722-4c65-ad89-3b07724104fa None admin] RBAC: Adding query filter params (name=block-storage) {{(pid=81409) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 13 04:09:09.621618 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fe539ca0-2722-4c65-ad89-3b07724104fa None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:09:09.623309 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fe539ca0-2722-4c65-ad89-3b07724104fa None admin] RBAC: Authorization granted {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:09:09.627609 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG dbcounter [-] [81408] Writing DB stats keystone:SELECT=100 {{(pid=81408) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:09:09.628894 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 266/517] 10.222.0.27 () {56 vars in 1124 bytes} [Thu Oct 13 04:09:09 2022] GET /identity/v3/services?name=block-storage => generated 130 bytes in 179 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 13 04:09:09.629976 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fe539ca0-2722-4c65-ad89-3b07724104fa None admin] Authenticating user token {{(pid=81409) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:09:09.655856 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG dbcounter [-] [81409] Writing DB stats keystone:SELECT=100 {{(pid=81409) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:09:09.666851 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 252/518] 10.222.0.27 () {58 vars in 1024 bytes} [Thu Oct 13 04:09:09 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 96 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 13 04:09:09.667529 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c15f7856-7579-45f7-b1b4-0f1c0e471013 None None] Authenticating user token {{(pid=81408) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:09:09.750513 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fe539ca0-2722-4c65-ad89-3b07724104fa None admin] Validating token access rules against request {{(pid=81409) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:09:09.781004 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c15f7856-7579-45f7-b1b4-0f1c0e471013 None None] Validating token access rules against request {{(pid=81408) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:09:09.788852 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4f428484-8bd6-4b28-93d5-03c4ad461033 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81409) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:09:09.789582 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f428484-8bd6-4b28-93d5-03c4ad461033 None admin] REQUEST_METHOD: `GET` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:09:09.790510 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f428484-8bd6-4b28-93d5-03c4ad461033 None admin] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:09:09.790783 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f428484-8bd6-4b28-93d5-03c4ad461033 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:09:09.803029 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4f428484-8bd6-4b28-93d5-03c4ad461033 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:09:09.804575 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4f428484-8bd6-4b28-93d5-03c4ad461033 None admin] RBAC: Authorization granted {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:09:09.822180 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e3808469-633c-47bd-bb36-bd8fbf5357a7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81408) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:09:09.822813 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application [None req-4f428484-8bd6-4b28-93d5-03c4ad461033 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 13 04:09:09.822813 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 13 04:09:09.822813 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 13 04:09:09.822813 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 13 04:09:09.822813 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 13 04:09:09.822813 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 13 04:09:09.822813 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 13 04:09:09.822813 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 13 04:09:09.822813 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 13 04:09:09.822813 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 13 04:09:09.822813 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 13 04:09:09.822813 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 13 04:09:09.822813 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 13 04:09:09.822813 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 13 04:09:09.822813 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 13 04:09:09.822813 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 13 04:09:09.823684 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 13 04:09:09.823684 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 13 04:09:09.823684 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 13 04:09:09.823684 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 13 04:09:09.823684 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 13 04:09:09.823684 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application return self.get_or_create( Oct 13 04:09:09.823684 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 13 04:09:09.823684 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application with Lock( Oct 13 04:09:09.823684 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 13 04:09:09.823684 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application return self._enter() Oct 13 04:09:09.823684 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 13 04:09:09.823684 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 13 04:09:09.823684 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 13 04:09:09.823684 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application return self.creator() Oct 13 04:09:09.823684 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 13 04:09:09.823684 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application created_value = creator( Oct 13 04:09:09.823684 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 13 04:09:09.824559 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 13 04:09:09.824559 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 13 04:09:09.824559 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application Oct 13 04:09:09.824727 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3808469-633c-47bd-bb36-bd8fbf5357a7 None admin] REQUEST_METHOD: `GET` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:09:09.824727 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3808469-633c-47bd-bb36-bd8fbf5357a7 None admin] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:09:09.824727 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3808469-633c-47bd-bb36-bd8fbf5357a7 None admin] PATH_INFO: `/v3/domains` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:09:09.824727 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e3808469-633c-47bd-bb36-bd8fbf5357a7 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81408) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 13 04:09:09.824964 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 267/519] 10.222.0.27 () {56 vars in 1108 bytes} [Thu Oct 13 04:09:09 2022] GET /identity/v3/domains/Default => generated 87 bytes in 195 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 13 04:09:09.825030 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e3808469-633c-47bd-bb36-bd8fbf5357a7 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:09:09.825097 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4f428484-8bd6-4b28-93d5-03c4ad461033 None admin] Authenticating user token {{(pid=81409) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:09:09.827205 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e3808469-633c-47bd-bb36-bd8fbf5357a7 None admin] RBAC: Authorization granted {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:09:09.839270 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 253/520] 10.222.0.27 () {56 vars in 1109 bytes} [Thu Oct 13 04:09:09 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 172 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 13 04:09:09.840121 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e3808469-633c-47bd-bb36-bd8fbf5357a7 None admin] Authenticating user token {{(pid=81408) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:09:09.876137 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG dbcounter [-] [81408] Writing DB stats keystone:SELECT=100 {{(pid=81408) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:09:09.908986 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG dbcounter [-] [81409] Writing DB stats keystone:SELECT=100 {{(pid=81409) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:09:09.947408 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4f428484-8bd6-4b28-93d5-03c4ad461033 None admin] Validating token access rules against request {{(pid=81409) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:09:09.964513 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e3808469-633c-47bd-bb36-bd8fbf5357a7 None admin] Validating token access rules against request {{(pid=81408) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:09:09.983379 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c29f6c39-c937-4a26-bdad-92d0f8e80b42 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81409) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:09:09.983989 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c29f6c39-c937-4a26-bdad-92d0f8e80b42 None admin] REQUEST_METHOD: `GET` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:09:09.984328 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c29f6c39-c937-4a26-bdad-92d0f8e80b42 None admin] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:09:09.984623 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c29f6c39-c937-4a26-bdad-92d0f8e80b42 None admin] PATH_INFO: `/v3/services` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:09:09.985256 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c29f6c39-c937-4a26-bdad-92d0f8e80b42 None admin] RBAC: Adding query filter params (type=block-storage) {{(pid=81409) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 13 04:09:09.985558 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c29f6c39-c937-4a26-bdad-92d0f8e80b42 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:09:09.986963 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c29f6c39-c937-4a26-bdad-92d0f8e80b42 None admin] RBAC: Authorization granted {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:09:09.994037 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 268/521] 10.222.0.27 () {56 vars in 1124 bytes} [Thu Oct 13 04:09:09 2022] GET /identity/v3/services?type=block-storage => generated 130 bytes in 169 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 13 04:09:09.995072 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c29f6c39-c937-4a26-bdad-92d0f8e80b42 None admin] Authenticating user token {{(pid=81409) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:09:10.000133 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c9f14e99-a916-4d77-b061-d35350b3c831 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81408) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:09:10.000911 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9f14e99-a916-4d77-b061-d35350b3c831 None admin] REQUEST_METHOD: `GET` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:09:10.001188 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9f14e99-a916-4d77-b061-d35350b3c831 None admin] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:09:10.001522 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9f14e99-a916-4d77-b061-d35350b3c831 None admin] PATH_INFO: `/v3/domains` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:09:10.002362 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c9f14e99-a916-4d77-b061-d35350b3c831 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81408) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 13 04:09:10.002743 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c9f14e99-a916-4d77-b061-d35350b3c831 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:09:10.004296 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c9f14e99-a916-4d77-b061-d35350b3c831 None admin] RBAC: Authorization granted {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:09:10.018850 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 254/522] 10.222.0.27 () {56 vars in 1109 bytes} [Thu Oct 13 04:09:09 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 179 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 13 04:09:10.019940 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c9f14e99-a916-4d77-b061-d35350b3c831 None admin] Authenticating user token {{(pid=81408) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:09:10.126960 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c29f6c39-c937-4a26-bdad-92d0f8e80b42 None admin] Validating token access rules against request {{(pid=81409) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:09:10.130381 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG dbcounter [-] [81408] Writing DB stats keystone:SELECT=100 {{(pid=81408) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:09:10.144216 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c9f14e99-a916-4d77-b061-d35350b3c831 None admin] Validating token access rules against request {{(pid=81408) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:09:10.167750 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4231e265-7e05-487d-9a6e-826743018fd4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81409) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:09:10.168498 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4231e265-7e05-487d-9a6e-826743018fd4 None admin] REQUEST_METHOD: `GET` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:09:10.168838 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4231e265-7e05-487d-9a6e-826743018fd4 None admin] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:09:10.169254 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4231e265-7e05-487d-9a6e-826743018fd4 None admin] PATH_INFO: `/v3/projects/service` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:09:10.172786 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG dbcounter [-] [81409] Writing DB stats keystone:SELECT=100 {{(pid=81409) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:09:10.177673 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4231e265-7e05-487d-9a6e-826743018fd4 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:09:10.182275 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4231e265-7e05-487d-9a6e-826743018fd4 None admin] RBAC: Authorization granted {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:09:10.203789 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application [None req-4231e265-7e05-487d-9a6e-826743018fd4 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Oct 13 04:09:10.203789 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 13 04:09:10.203789 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 13 04:09:10.203789 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 13 04:09:10.203789 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 13 04:09:10.203789 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 13 04:09:10.203789 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 13 04:09:10.203789 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 13 04:09:10.203789 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 13 04:09:10.203789 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 13 04:09:10.203789 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 13 04:09:10.203789 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 13 04:09:10.203789 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Oct 13 04:09:10.203789 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application return self._get_project(project_id) Oct 13 04:09:10.203789 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Oct 13 04:09:10.204775 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Oct 13 04:09:10.204775 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 13 04:09:10.204775 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 13 04:09:10.204775 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 13 04:09:10.204775 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 13 04:09:10.204775 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 13 04:09:10.204775 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application return self.get_or_create( Oct 13 04:09:10.204775 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 13 04:09:10.204775 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application with Lock( Oct 13 04:09:10.204775 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 13 04:09:10.204775 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application return self._enter() Oct 13 04:09:10.204775 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 13 04:09:10.204775 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 13 04:09:10.204775 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 13 04:09:10.204775 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application return self.creator() Oct 13 04:09:10.204775 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 13 04:09:10.204775 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application created_value = creator( Oct 13 04:09:10.205654 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Oct 13 04:09:10.205654 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Oct 13 04:09:10.205654 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Oct 13 04:09:10.205654 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Oct 13 04:09:10.205654 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Oct 13 04:09:10.205654 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Oct 13 04:09:10.205654 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Oct 13 04:09:10.205654 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application Oct 13 04:09:10.205654 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 269/523] 10.222.0.27 () {56 vars in 1111 bytes} [Thu Oct 13 04:09:09 2022] GET /identity/v3/projects/service => generated 88 bytes in 210 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 13 04:09:10.206635 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4231e265-7e05-487d-9a6e-826743018fd4 None admin] Authenticating user token {{(pid=81409) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:09:10.208068 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1407ee92-7085-4d81-9609-5ed35c2e0ce5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81408) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:09:10.208937 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1407ee92-7085-4d81-9609-5ed35c2e0ce5 None admin] REQUEST_METHOD: `GET` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:09:10.209413 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1407ee92-7085-4d81-9609-5ed35c2e0ce5 None admin] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:09:10.209875 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1407ee92-7085-4d81-9609-5ed35c2e0ce5 None admin] PATH_INFO: `/v3/services/network` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:09:10.210712 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1407ee92-7085-4d81-9609-5ed35c2e0ce5 None admin] RBAC: Authorizing `identity:get_service(service_id=network)` {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:09:10.212947 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1407ee92-7085-4d81-9609-5ed35c2e0ce5 None admin] RBAC: Authorization granted {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:09:10.231939 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application [None req-1407ee92-7085-4d81-9609-5ed35c2e0ce5 None admin] Could not find service: network.: keystone.exception.ServiceNotFound: Could not find service: network. Oct 13 04:09:10.231939 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 13 04:09:10.231939 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Oct 13 04:09:10.231939 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Oct 13 04:09:10.231939 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Oct 13 04:09:10.231939 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Oct 13 04:09:10.231939 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Oct 13 04:09:10.231939 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 13 04:09:10.231939 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: network. Oct 13 04:09:10.231939 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application Oct 13 04:09:10.231939 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Oct 13 04:09:10.231939 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application Oct 13 04:09:10.231939 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 13 04:09:10.231939 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 13 04:09:10.231939 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 13 04:09:10.231939 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 13 04:09:10.231939 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 13 04:09:10.233013 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 13 04:09:10.233013 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 13 04:09:10.233013 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 13 04:09:10.233013 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 13 04:09:10.233013 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 13 04:09:10.233013 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 13 04:09:10.233013 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Oct 13 04:09:10.233013 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application return self._get_service(service_id) Oct 13 04:09:10.233013 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Oct 13 04:09:10.233013 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Oct 13 04:09:10.233013 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 13 04:09:10.233013 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 13 04:09:10.233013 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 13 04:09:10.233013 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 13 04:09:10.233013 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 13 04:09:10.233013 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application return self.get_or_create( Oct 13 04:09:10.233865 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 13 04:09:10.233865 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application with Lock( Oct 13 04:09:10.233865 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 13 04:09:10.233865 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application return self._enter() Oct 13 04:09:10.233865 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 13 04:09:10.233865 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 13 04:09:10.233865 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 13 04:09:10.233865 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application return self.creator() Oct 13 04:09:10.233865 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 13 04:09:10.233865 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application created_value = creator( Oct 13 04:09:10.233865 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Oct 13 04:09:10.233865 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 13 04:09:10.233865 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: network. Oct 13 04:09:10.233865 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application Oct 13 04:09:10.233865 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 255/524] 10.222.0.27 () {56 vars in 1111 bytes} [Thu Oct 13 04:09:10 2022] GET /identity/v3/services/network => generated 88 bytes in 214 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 13 04:09:10.234693 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1407ee92-7085-4d81-9609-5ed35c2e0ce5 None admin] Authenticating user token {{(pid=81408) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:09:10.341977 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4231e265-7e05-487d-9a6e-826743018fd4 None admin] Validating token access rules against request {{(pid=81409) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:09:10.349076 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1407ee92-7085-4d81-9609-5ed35c2e0ce5 None admin] Validating token access rules against request {{(pid=81408) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:09:10.379922 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6154d61c-5c9a-49b0-9769-6642e8c0483d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81409) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:09:10.380865 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6154d61c-5c9a-49b0-9769-6642e8c0483d None admin] REQUEST_METHOD: `POST` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:09:10.381343 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6154d61c-5c9a-49b0-9769-6642e8c0483d None admin] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:09:10.381948 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6154d61c-5c9a-49b0-9769-6642e8c0483d None admin] PATH_INFO: `/v3/users` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:09:10.382904 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6154d61c-5c9a-49b0-9769-6642e8c0483d None admin] RBAC: Authorizing `identity:create_user()` {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:09:10.385577 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6154d61c-5c9a-49b0-9769-6642e8c0483d None admin] RBAC: Authorization granted {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:09:10.385692 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5eeff5b1-1774-4f67-8529-6c8e2ec954dc None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81408) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:09:10.386310 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5eeff5b1-1774-4f67-8529-6c8e2ec954dc None admin] REQUEST_METHOD: `GET` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:09:10.386580 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5eeff5b1-1774-4f67-8529-6c8e2ec954dc None admin] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:09:10.386853 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5eeff5b1-1774-4f67-8529-6c8e2ec954dc None admin] PATH_INFO: `/v3/projects` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:09:10.387545 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5eeff5b1-1774-4f67-8529-6c8e2ec954dc None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81408) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 13 04:09:10.387855 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5eeff5b1-1774-4f67-8529-6c8e2ec954dc None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:09:10.389278 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5eeff5b1-1774-4f67-8529-6c8e2ec954dc None admin] RBAC: Authorization granted {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:09:10.402470 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 256/525] 10.222.0.27 () {56 vars in 1148 bytes} [Thu Oct 13 04:09:10 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 168 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 13 04:09:10.403195 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5eeff5b1-1774-4f67-8529-6c8e2ec954dc None admin] Authenticating user token {{(pid=81408) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:09:10.418584 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 270/526] 10.222.0.27 () {60 vars in 1132 bytes} [Thu Oct 13 04:09:10 2022] POST /identity/v3/users => generated 254 bytes in 212 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 13 04:09:10.419564 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6154d61c-5c9a-49b0-9769-6642e8c0483d None admin] Authenticating user token {{(pid=81409) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:09:10.443480 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG dbcounter [-] [81408] Writing DB stats keystone:SELECT=100 {{(pid=81408) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:09:10.514708 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG dbcounter [-] [81409] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81409) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:09:10.589571 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6154d61c-5c9a-49b0-9769-6642e8c0483d None admin] Validating token access rules against request {{(pid=81409) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:09:10.646793 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5eeff5b1-1774-4f67-8529-6c8e2ec954dc None admin] Validating token access rules against request {{(pid=81408) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:09:10.659715 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-de719ae5-5ee5-4b70-941c-80d9dd90b788 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81409) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:09:10.660446 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de719ae5-5ee5-4b70-941c-80d9dd90b788 None admin] REQUEST_METHOD: `PUT` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:09:10.660828 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de719ae5-5ee5-4b70-941c-80d9dd90b788 None admin] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:09:10.661107 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de719ae5-5ee5-4b70-941c-80d9dd90b788 None admin] PATH_INFO: `/v3/projects/dcec0431fb6849499d6aec35a234daa6/users/cb740ed7f7684b079e45f887be83f795/roles/ef55fbd94e1040be83695754075a7ed5` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:09:10.663047 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-de719ae5-5ee5-4b70-941c-80d9dd90b788 None admin] RBAC: Authorizing `identity:create_grant(project_id=dcec0431fb6849499d6aec35a234daa6, user_id=cb740ed7f7684b079e45f887be83f795, role_id=ef55fbd94e1040be83695754075a7ed5)` {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:09:10.664545 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-de719ae5-5ee5-4b70-941c-80d9dd90b788 None admin] RBAC: Authorization granted {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:09:10.683813 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 271/527] 10.222.0.27 () {58 vars in 1439 bytes} [Thu Oct 13 04:09:10 2022] PUT /identity/v3/projects/dcec0431fb6849499d6aec35a234daa6/users/cb740ed7f7684b079e45f887be83f795/roles/ef55fbd94e1040be83695754075a7ed5 => generated 0 bytes in 265 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 13 04:09:10.704514 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7d5a1da7-b42b-43c5-8da6-f14f849f6198 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81408) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:09:10.705144 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d5a1da7-b42b-43c5-8da6-f14f849f6198 None admin] REQUEST_METHOD: `GET` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:09:10.705422 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d5a1da7-b42b-43c5-8da6-f14f849f6198 None admin] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:09:10.705692 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d5a1da7-b42b-43c5-8da6-f14f849f6198 None admin] PATH_INFO: `/v3/services` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:09:10.706457 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7d5a1da7-b42b-43c5-8da6-f14f849f6198 None admin] RBAC: Adding query filter params (name=network) {{(pid=81408) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 13 04:09:10.706762 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7d5a1da7-b42b-43c5-8da6-f14f849f6198 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:09:10.708206 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7d5a1da7-b42b-43c5-8da6-f14f849f6198 None admin] RBAC: Authorization granted {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:09:10.716491 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 257/528] 10.222.0.27 () {56 vars in 1112 bytes} [Thu Oct 13 04:09:10 2022] GET /identity/v3/services?name=network => generated 124 bytes in 313 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 13 04:09:10.722433 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-de719ae5-5ee5-4b70-941c-80d9dd90b788 None admin] Authenticating user token {{(pid=81409) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:09:10.867041 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-de719ae5-5ee5-4b70-941c-80d9dd90b788 None admin] Validating token access rules against request {{(pid=81409) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:09:10.881442 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG dbcounter [-] [81409] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81409) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:09:10.909836 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-924bc3be-e34c-4c9f-bd87-aec914148f3d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81409) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:09:10.910395 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-924bc3be-e34c-4c9f-bd87-aec914148f3d None admin] REQUEST_METHOD: `GET` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:09:10.910672 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-924bc3be-e34c-4c9f-bd87-aec914148f3d None admin] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:09:10.910941 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-924bc3be-e34c-4c9f-bd87-aec914148f3d None admin] PATH_INFO: `/v3/services` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:09:10.911574 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-924bc3be-e34c-4c9f-bd87-aec914148f3d None admin] RBAC: Adding query filter params (type=network) {{(pid=81409) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 13 04:09:10.911872 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-924bc3be-e34c-4c9f-bd87-aec914148f3d None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:09:10.913276 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-924bc3be-e34c-4c9f-bd87-aec914148f3d None admin] RBAC: Authorization granted {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:09:10.918759 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 272/529] 10.222.0.27 () {56 vars in 1112 bytes} [Thu Oct 13 04:09:10 2022] GET /identity/v3/services?type=network => generated 124 bytes in 197 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 13 04:09:11.723216 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21ef68ea-4d3c-429e-926e-f24e18d21193 None None] REQUEST_METHOD: `GET` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:09:11.723758 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21ef68ea-4d3c-429e-926e-f24e18d21193 None None] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:09:11.723758 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21ef68ea-4d3c-429e-926e-f24e18d21193 None None] PATH_INFO: `/` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:09:11.725583 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 258/530] 10.222.0.27 () {54 vars in 927 bytes} [Thu Oct 13 04:09:11 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 13 04:09:11.730837 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ea15234-609d-4c6f-b1fe-41127d6b3878 None None] REQUEST_METHOD: `POST` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:09:11.731238 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ea15234-609d-4c6f-b1fe-41127d6b3878 None None] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:09:11.731420 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ea15234-609d-4c6f-b1fe-41127d6b3878 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:09:11.755195 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.auth.core [None req-7ea15234-609d-4c6f-b1fe-41127d6b3878 None None] MFA Rules not processed for user `0e2d979563d6421995451a0e86bf3022`. Rule list: `[]` (Enabled: `True`). {{(pid=81409) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 13 04:09:11.756382 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.fernet_utils [None req-7ea15234-609d-4c6f-b1fe-41127d6b3878 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81409) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:09:11.851044 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 273/531] 10.222.0.27 () {58 vars in 1024 bytes} [Thu Oct 13 04:09:11 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 121 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 13 04:09:12.248172 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-21ef68ea-4d3c-429e-926e-f24e18d21193 None None] Authenticating user token {{(pid=81408) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:09:12.350464 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG dbcounter [-] [81408] Writing DB stats keystone:SELECT=100 {{(pid=81408) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:09:12.374983 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-21ef68ea-4d3c-429e-926e-f24e18d21193 None None] Validating token access rules against request {{(pid=81408) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:09:12.423344 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3b452f13-245a-4817-84c2-d07eb08ea20a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81408) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:09:12.423983 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b452f13-245a-4817-84c2-d07eb08ea20a None admin] REQUEST_METHOD: `GET` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:09:12.424260 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b452f13-245a-4817-84c2-d07eb08ea20a None admin] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:09:12.424528 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b452f13-245a-4817-84c2-d07eb08ea20a None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:09:12.440580 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3b452f13-245a-4817-84c2-d07eb08ea20a None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 13 04:09:12.445421 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application [None req-3b452f13-245a-4817-84c2-d07eb08ea20a None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Oct 13 04:09:12.445421 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 13 04:09:12.445421 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 13 04:09:12.445421 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 13 04:09:12.445421 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 13 04:09:12.445421 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 13 04:09:12.445421 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 13 04:09:12.445421 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 13 04:09:12.445421 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 13 04:09:12.445421 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 13 04:09:12.445421 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 13 04:09:12.445421 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 13 04:09:12.445421 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 13 04:09:12.445421 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 13 04:09:12.445421 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 13 04:09:12.445421 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 13 04:09:12.446444 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 13 04:09:12.446444 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 13 04:09:12.446444 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 13 04:09:12.446444 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 13 04:09:12.446444 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 13 04:09:12.446444 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 13 04:09:12.446444 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 13 04:09:12.446444 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 13 04:09:12.446444 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 13 04:09:12.446444 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application return self.get_or_create( Oct 13 04:09:12.446444 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 13 04:09:12.446444 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application with Lock( Oct 13 04:09:12.446444 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 13 04:09:12.446444 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application return self._enter() Oct 13 04:09:12.446444 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 13 04:09:12.446444 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 13 04:09:12.447469 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 13 04:09:12.447469 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application return self.creator() Oct 13 04:09:12.447469 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 13 04:09:12.447469 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application created_value = creator( Oct 13 04:09:12.447469 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 13 04:09:12.447469 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 13 04:09:12.447469 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 13 04:09:12.447469 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 13 04:09:12.447469 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 13 04:09:12.447469 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 13 04:09:12.447469 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Oct 13 04:09:12.447469 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application Oct 13 04:09:12.447469 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 259/532] 10.222.0.27 () {56 vars in 1096 bytes} [Thu Oct 13 04:09:12 2022] GET /identity/v3/roles/admin => generated 83 bytes in 200 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 13 04:09:12.455399 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7ea15234-609d-4c6f-b1fe-41127d6b3878 None None] Authenticating user token {{(pid=81409) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:09:12.576435 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7ea15234-609d-4c6f-b1fe-41127d6b3878 None None] Validating token access rules against request {{(pid=81409) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:09:12.584571 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG dbcounter [-] [81409] Writing DB stats keystone:SELECT=100 {{(pid=81409) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:09:12.614264 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-19da4391-21af-401a-826a-256125f09670 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81409) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:09:12.615219 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19da4391-21af-401a-826a-256125f09670 None admin] REQUEST_METHOD: `GET` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:09:12.615699 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19da4391-21af-401a-826a-256125f09670 None admin] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:09:12.616169 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19da4391-21af-401a-826a-256125f09670 None admin] PATH_INFO: `/v3/roles` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:09:12.617046 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-19da4391-21af-401a-826a-256125f09670 None admin] RBAC: Adding query filter params (name=admin) {{(pid=81409) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 13 04:09:12.617365 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25fe6d9b-b8bc-4a14-852c-335a798a14c2 None None] REQUEST_METHOD: `GET` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:09:12.617652 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25fe6d9b-b8bc-4a14-852c-335a798a14c2 None None] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:09:12.618070 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25fe6d9b-b8bc-4a14-852c-335a798a14c2 None None] PATH_INFO: `/` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:09:12.618175 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-19da4391-21af-401a-826a-256125f09670 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:09:12.619987 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-19da4391-21af-401a-826a-256125f09670 None admin] RBAC: Authorization granted {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:09:12.622313 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 260/533] 10.222.0.27 () {54 vars in 927 bytes} [Thu Oct 13 04:09:12 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 13 04:09:12.629419 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 274/534] 10.222.0.27 () {56 vars in 1099 bytes} [Thu Oct 13 04:09:12 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 174 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 13 04:09:12.634041 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa5a7ed3-0e33-4db7-b2a8-6c644d30168c None None] REQUEST_METHOD: `POST` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:09:12.634041 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa5a7ed3-0e33-4db7-b2a8-6c644d30168c None None] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:09:12.634041 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa5a7ed3-0e33-4db7-b2a8-6c644d30168c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:09:12.635625 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-19da4391-21af-401a-826a-256125f09670 None admin] Authenticating user token {{(pid=81409) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:09:12.667193 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.auth.core [None req-aa5a7ed3-0e33-4db7-b2a8-6c644d30168c None None] MFA Rules not processed for user `0e2d979563d6421995451a0e86bf3022`. Rule list: `[]` (Enabled: `True`). {{(pid=81408) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 13 04:09:12.668390 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.fernet_utils [None req-aa5a7ed3-0e33-4db7-b2a8-6c644d30168c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81408) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:09:12.784280 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 261/535] 10.222.0.27 () {58 vars in 1024 bytes} [Thu Oct 13 04:09:12 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 154 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 13 04:09:12.812920 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-19da4391-21af-401a-826a-256125f09670 None admin] Validating token access rules against request {{(pid=81409) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:09:12.869673 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-189be062-cd88-4e0a-abf9-f7a4ca2b8213 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81409) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:09:12.870263 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-189be062-cd88-4e0a-abf9-f7a4ca2b8213 None admin] REQUEST_METHOD: `GET` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:09:12.870543 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-189be062-cd88-4e0a-abf9-f7a4ca2b8213 None admin] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:09:12.870811 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-189be062-cd88-4e0a-abf9-f7a4ca2b8213 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:09:12.882261 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-189be062-cd88-4e0a-abf9-f7a4ca2b8213 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:09:12.883740 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-189be062-cd88-4e0a-abf9-f7a4ca2b8213 None admin] RBAC: Authorization granted {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:09:12.906884 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application [None req-189be062-cd88-4e0a-abf9-f7a4ca2b8213 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 13 04:09:12.906884 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 13 04:09:12.906884 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 13 04:09:12.906884 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 13 04:09:12.906884 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 13 04:09:12.906884 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 13 04:09:12.906884 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 13 04:09:12.906884 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 13 04:09:12.906884 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 13 04:09:12.906884 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 13 04:09:12.906884 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 13 04:09:12.906884 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 13 04:09:12.906884 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 13 04:09:12.906884 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 13 04:09:12.906884 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 13 04:09:12.906884 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 13 04:09:12.907835 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 13 04:09:12.907835 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 13 04:09:12.907835 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 13 04:09:12.907835 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 13 04:09:12.907835 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 13 04:09:12.907835 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application return self.get_or_create( Oct 13 04:09:12.907835 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 13 04:09:12.907835 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application with Lock( Oct 13 04:09:12.907835 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 13 04:09:12.907835 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application return self._enter() Oct 13 04:09:12.907835 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 13 04:09:12.907835 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 13 04:09:12.907835 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 13 04:09:12.907835 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application return self.creator() Oct 13 04:09:12.907835 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 13 04:09:12.907835 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application created_value = creator( Oct 13 04:09:12.907835 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 13 04:09:12.908723 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 13 04:09:12.908723 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 13 04:09:12.908723 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application Oct 13 04:09:12.908723 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 275/536] 10.222.0.27 () {56 vars in 1108 bytes} [Thu Oct 13 04:09:12 2022] GET /identity/v3/domains/Default => generated 87 bytes in 273 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 13 04:09:12.913732 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aa5a7ed3-0e33-4db7-b2a8-6c644d30168c None None] Authenticating user token {{(pid=81408) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:09:12.971901 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e18e3c2-d1e1-458c-a29a-8b6b1a3947f7 None None] REQUEST_METHOD: `GET` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:09:12.972530 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e18e3c2-d1e1-458c-a29a-8b6b1a3947f7 None None] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:09:12.973008 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e18e3c2-d1e1-458c-a29a-8b6b1a3947f7 None None] PATH_INFO: `/` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:09:12.974984 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 276/537] 10.222.0.27 () {54 vars in 927 bytes} [Thu Oct 13 04:09:12 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 13 04:09:12.980580 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-569bea36-675e-4298-8125-18150a4c5c54 None None] REQUEST_METHOD: `POST` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:09:12.981114 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-569bea36-675e-4298-8125-18150a4c5c54 None None] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:09:12.982000 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-569bea36-675e-4298-8125-18150a4c5c54 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:09:13.015586 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG dbcounter [-] [81408] Writing DB stats keystone:SELECT=100 {{(pid=81408) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:09:13.019799 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.auth.core [None req-569bea36-675e-4298-8125-18150a4c5c54 None None] MFA Rules not processed for user `0e2d979563d6421995451a0e86bf3022`. Rule list: `[]` (Enabled: `True`). {{(pid=81409) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 13 04:09:13.021012 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.fernet_utils [None req-569bea36-675e-4298-8125-18150a4c5c54 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81409) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:09:13.052409 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG dbcounter [-] [81409] Writing DB stats keystone:SELECT=100 {{(pid=81409) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:09:13.054069 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aa5a7ed3-0e33-4db7-b2a8-6c644d30168c None None] Validating token access rules against request {{(pid=81408) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:09:13.093013 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1b28a644-457d-460c-921f-3ec2d41a5b0b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81408) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:09:13.093632 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b28a644-457d-460c-921f-3ec2d41a5b0b None admin] REQUEST_METHOD: `GET` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:09:13.093909 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b28a644-457d-460c-921f-3ec2d41a5b0b None admin] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:09:13.094179 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b28a644-457d-460c-921f-3ec2d41a5b0b None admin] PATH_INFO: `/v3/domains` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:09:13.094939 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1b28a644-457d-460c-921f-3ec2d41a5b0b None admin] RBAC: Adding query filter params (name=Default) {{(pid=81408) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 13 04:09:13.095347 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1b28a644-457d-460c-921f-3ec2d41a5b0b None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:09:13.096718 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1b28a644-457d-460c-921f-3ec2d41a5b0b None admin] RBAC: Authorization granted {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:09:13.111222 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 262/538] 10.222.0.27 () {56 vars in 1109 bytes} [Thu Oct 13 04:09:12 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 198 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 13 04:09:13.116940 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1b28a644-457d-460c-921f-3ec2d41a5b0b None admin] Authenticating user token {{(pid=81408) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:09:13.145540 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 277/539] 10.222.0.27 () {58 vars in 1024 bytes} [Thu Oct 13 04:09:12 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 165 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 13 04:09:13.146507 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-569bea36-675e-4298-8125-18150a4c5c54 None None] Authenticating user token {{(pid=81409) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:09:13.238113 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1b28a644-457d-460c-921f-3ec2d41a5b0b None admin] Validating token access rules against request {{(pid=81408) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:09:13.266702 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG dbcounter [-] [81408] Writing DB stats keystone:SELECT=100 {{(pid=81408) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:09:13.277637 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9fd25407-d70e-4386-8534-fe56b7ab684e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81408) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:09:13.278401 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9fd25407-d70e-4386-8534-fe56b7ab684e None admin] REQUEST_METHOD: `GET` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:09:13.278599 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9fd25407-d70e-4386-8534-fe56b7ab684e None admin] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:09:13.278835 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9fd25407-d70e-4386-8534-fe56b7ab684e None admin] PATH_INFO: `/v3/users/swift` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:09:13.282733 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-569bea36-675e-4298-8125-18150a4c5c54 None None] Validating token access rules against request {{(pid=81409) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:09:13.286442 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9fd25407-d70e-4386-8534-fe56b7ab684e None admin] RBAC: Authorizing `identity:get_user(user_id=swift)` {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:09:13.287879 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9fd25407-d70e-4386-8534-fe56b7ab684e None admin] RBAC: Authorization granted {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:09:13.300781 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application [None req-9fd25407-d70e-4386-8534-fe56b7ab684e None admin] Could not find user: swift.: keystone.exception.UserNotFound: Could not find user: swift. Oct 13 04:09:13.300781 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 13 04:09:13.300781 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 13 04:09:13.300781 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 13 04:09:13.300781 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 13 04:09:13.300781 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 13 04:09:13.300781 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 13 04:09:13.300781 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 13 04:09:13.300781 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 13 04:09:13.300781 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 13 04:09:13.300781 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 13 04:09:13.300781 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 13 04:09:13.300781 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Oct 13 04:09:13.300781 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application return self._get_user(user_id) Oct 13 04:09:13.300781 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Oct 13 04:09:13.300781 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Oct 13 04:09:13.301763 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 13 04:09:13.301763 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 13 04:09:13.301763 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Oct 13 04:09:13.301763 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 13 04:09:13.301763 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Oct 13 04:09:13.301763 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 13 04:09:13.301763 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 13 04:09:13.301763 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 13 04:09:13.301763 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 13 04:09:13.301763 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application return self.get_or_create( Oct 13 04:09:13.301763 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 13 04:09:13.301763 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application with Lock( Oct 13 04:09:13.301763 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 13 04:09:13.301763 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application return self._enter() Oct 13 04:09:13.301763 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 13 04:09:13.301763 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 13 04:09:13.301763 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 13 04:09:13.301763 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application return self.creator() Oct 13 04:09:13.307311 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 13 04:09:13.307311 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application created_value = creator( Oct 13 04:09:13.307311 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Oct 13 04:09:13.307311 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Oct 13 04:09:13.307311 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Oct 13 04:09:13.307311 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Oct 13 04:09:13.307311 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Oct 13 04:09:13.307311 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Oct 13 04:09:13.307311 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: swift. Oct 13 04:09:13.307311 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application Oct 13 04:09:13.307311 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 263/540] 10.222.0.27 () {56 vars in 1096 bytes} [Thu Oct 13 04:09:13 2022] GET /identity/v3/users/swift => generated 83 bytes in 185 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Oct 13 04:09:13.310005 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9fd25407-d70e-4386-8534-fe56b7ab684e None admin] Authenticating user token {{(pid=81408) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:09:13.354957 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f74b1e1b-d069-46dc-8714-c8515329a6d9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81409) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:09:13.355582 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f74b1e1b-d069-46dc-8714-c8515329a6d9 None admin] REQUEST_METHOD: `POST` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:09:13.355856 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f74b1e1b-d069-46dc-8714-c8515329a6d9 None admin] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:09:13.356122 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f74b1e1b-d069-46dc-8714-c8515329a6d9 None admin] PATH_INFO: `/v3/services` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:09:13.357227 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f74b1e1b-d069-46dc-8714-c8515329a6d9 None admin] RBAC: Authorizing `identity:create_service()` {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:09:13.358216 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f74b1e1b-d069-46dc-8714-c8515329a6d9 None admin] RBAC: Authorization granted {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:09:13.391128 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 278/541] 10.222.0.27 () {60 vars in 1141 bytes} [Thu Oct 13 04:09:13 2022] POST /identity/v3/services => generated 251 bytes in 245 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 13 04:09:13.428787 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9fd25407-d70e-4386-8534-fe56b7ab684e None admin] Validating token access rules against request {{(pid=81408) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:09:13.472961 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e8014799-6c37-4f29-af6c-6104cf7d2fa4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81408) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:09:13.473571 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8014799-6c37-4f29-af6c-6104cf7d2fa4 None admin] REQUEST_METHOD: `GET` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:09:13.473848 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8014799-6c37-4f29-af6c-6104cf7d2fa4 None admin] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:09:13.474116 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8014799-6c37-4f29-af6c-6104cf7d2fa4 None admin] PATH_INFO: `/v3/users` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:09:13.474870 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e8014799-6c37-4f29-af6c-6104cf7d2fa4 None admin] RBAC: Adding query filter params (domain_id=default, name=swift) {{(pid=81408) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 13 04:09:13.475178 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e8014799-6c37-4f29-af6c-6104cf7d2fa4 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:09:13.476907 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e8014799-6c37-4f29-af6c-6104cf7d2fa4 None admin] RBAC: Authorization granted {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:09:13.496875 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 264/542] 10.222.0.27 () {56 vars in 1135 bytes} [Thu Oct 13 04:09:13 2022] GET /identity/v3/users?domain_id=default&name=swift => generated 370 bytes in 187 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 13 04:09:13.499674 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f74b1e1b-d069-46dc-8714-c8515329a6d9 None admin] Authenticating user token {{(pid=81409) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:09:13.502716 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e8014799-6c37-4f29-af6c-6104cf7d2fa4 None admin] Authenticating user token {{(pid=81408) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:09:13.515294 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG dbcounter [-] [81409] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81409) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:09:13.551549 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG dbcounter [-] [81408] Writing DB stats keystone:SELECT=100 {{(pid=81408) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:09:13.619149 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e8014799-6c37-4f29-af6c-6104cf7d2fa4 None admin] Validating token access rules against request {{(pid=81408) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:09:13.621101 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f74b1e1b-d069-46dc-8714-c8515329a6d9 None admin] Validating token access rules against request {{(pid=81409) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:09:13.675476 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8273c050-f2ed-4f23-9082-56c697e79b45 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81408) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:09:13.676048 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8273c050-f2ed-4f23-9082-56c697e79b45 None admin] REQUEST_METHOD: `GET` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:09:13.676322 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8273c050-f2ed-4f23-9082-56c697e79b45 None admin] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:09:13.676592 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8273c050-f2ed-4f23-9082-56c697e79b45 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:09:13.697865 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-34808c9c-56c2-4d81-8557-c78925f96e42 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81409) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:09:13.698548 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34808c9c-56c2-4d81-8557-c78925f96e42 None admin] REQUEST_METHOD: `GET` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:09:13.698825 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34808c9c-56c2-4d81-8557-c78925f96e42 None admin] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:09:13.699094 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34808c9c-56c2-4d81-8557-c78925f96e42 None admin] PATH_INFO: `/v3/roles/service` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:09:13.703054 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8273c050-f2ed-4f23-9082-56c697e79b45 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:09:13.704752 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8273c050-f2ed-4f23-9082-56c697e79b45 None admin] RBAC: Authorization granted {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:09:13.711242 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-34808c9c-56c2-4d81-8557-c78925f96e42 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 13 04:09:13.717792 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application [None req-34808c9c-56c2-4d81-8557-c78925f96e42 None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Oct 13 04:09:13.717792 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 13 04:09:13.717792 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 13 04:09:13.717792 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 13 04:09:13.717792 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 13 04:09:13.717792 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 13 04:09:13.717792 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 13 04:09:13.717792 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 13 04:09:13.717792 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 13 04:09:13.717792 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 13 04:09:13.717792 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 13 04:09:13.717792 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 13 04:09:13.717792 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 13 04:09:13.717792 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 13 04:09:13.717792 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 13 04:09:13.717792 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 13 04:09:13.718758 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 13 04:09:13.718758 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 13 04:09:13.718758 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 13 04:09:13.718758 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 13 04:09:13.718758 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 13 04:09:13.718758 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 13 04:09:13.718758 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 13 04:09:13.718758 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 13 04:09:13.718758 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 13 04:09:13.718758 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application return self.get_or_create( Oct 13 04:09:13.718758 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 13 04:09:13.718758 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application with Lock( Oct 13 04:09:13.718758 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 13 04:09:13.718758 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application return self._enter() Oct 13 04:09:13.718758 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 13 04:09:13.718758 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 13 04:09:13.719558 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 13 04:09:13.719558 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application return self.creator() Oct 13 04:09:13.719558 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 13 04:09:13.719558 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application created_value = creator( Oct 13 04:09:13.719558 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 13 04:09:13.719558 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 13 04:09:13.719558 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 13 04:09:13.719558 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 13 04:09:13.719558 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 13 04:09:13.719558 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 13 04:09:13.719558 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Oct 13 04:09:13.719558 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application Oct 13 04:09:13.720639 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 279/543] 10.222.0.27 () {56 vars in 1102 bytes} [Thu Oct 13 04:09:13 2022] GET /identity/v3/roles/service => generated 85 bytes in 221 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 13 04:09:13.727065 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-34808c9c-56c2-4d81-8557-c78925f96e42 None admin] Authenticating user token {{(pid=81409) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:09:13.741855 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application [None req-8273c050-f2ed-4f23-9082-56c697e79b45 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 13 04:09:13.741855 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 13 04:09:13.741855 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 13 04:09:13.741855 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 13 04:09:13.741855 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 13 04:09:13.741855 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 13 04:09:13.741855 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 13 04:09:13.741855 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 13 04:09:13.741855 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 13 04:09:13.741855 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 13 04:09:13.741855 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 13 04:09:13.741855 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 13 04:09:13.741855 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 13 04:09:13.741855 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 13 04:09:13.741855 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 13 04:09:13.741855 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 13 04:09:13.743432 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 13 04:09:13.743432 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 13 04:09:13.743432 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 13 04:09:13.743432 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 13 04:09:13.743432 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 13 04:09:13.743432 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application return self.get_or_create( Oct 13 04:09:13.743432 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 13 04:09:13.743432 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application with Lock( Oct 13 04:09:13.743432 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 13 04:09:13.743432 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application return self._enter() Oct 13 04:09:13.743432 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 13 04:09:13.743432 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 13 04:09:13.743432 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 13 04:09:13.743432 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application return self.creator() Oct 13 04:09:13.743432 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 13 04:09:13.743432 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application created_value = creator( Oct 13 04:09:13.743432 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 13 04:09:13.744705 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 13 04:09:13.744705 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 13 04:09:13.744705 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application Oct 13 04:09:13.747466 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 265/544] 10.222.0.27 () {56 vars in 1108 bytes} [Thu Oct 13 04:09:13 2022] GET /identity/v3/domains/Default => generated 87 bytes in 245 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Oct 13 04:09:13.755638 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8273c050-f2ed-4f23-9082-56c697e79b45 None admin] Authenticating user token {{(pid=81408) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:09:13.829008 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG dbcounter [-] [81409] Writing DB stats keystone:SELECT=100 {{(pid=81409) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:09:13.872978 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-34808c9c-56c2-4d81-8557-c78925f96e42 None admin] Validating token access rules against request {{(pid=81409) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:09:13.873606 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG dbcounter [-] [81408] Writing DB stats keystone:SELECT=100 {{(pid=81408) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:09:13.880547 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8273c050-f2ed-4f23-9082-56c697e79b45 None admin] Validating token access rules against request {{(pid=81408) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:09:13.917700 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-432a7912-7906-4f17-bdcb-d26fc01e9c13 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81408) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:09:13.918422 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4470e534-bb74-4a85-b988-a6d6f22c2063 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81409) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:09:13.918591 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-432a7912-7906-4f17-bdcb-d26fc01e9c13 None admin] REQUEST_METHOD: `GET` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:09:13.918967 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-432a7912-7906-4f17-bdcb-d26fc01e9c13 None admin] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:09:13.919328 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-432a7912-7906-4f17-bdcb-d26fc01e9c13 None admin] PATH_INFO: `/v3/domains` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:09:13.919639 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4470e534-bb74-4a85-b988-a6d6f22c2063 None admin] REQUEST_METHOD: `GET` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:09:13.920047 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4470e534-bb74-4a85-b988-a6d6f22c2063 None admin] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:09:13.920247 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-432a7912-7906-4f17-bdcb-d26fc01e9c13 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81408) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 13 04:09:13.920590 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4470e534-bb74-4a85-b988-a6d6f22c2063 None admin] PATH_INFO: `/v3/roles` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:09:13.920697 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-432a7912-7906-4f17-bdcb-d26fc01e9c13 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:09:13.921481 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4470e534-bb74-4a85-b988-a6d6f22c2063 None admin] RBAC: Adding query filter params (name=service) {{(pid=81409) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 13 04:09:13.921915 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4470e534-bb74-4a85-b988-a6d6f22c2063 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:09:13.922408 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-432a7912-7906-4f17-bdcb-d26fc01e9c13 None admin] RBAC: Authorization granted {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:09:13.923607 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4470e534-bb74-4a85-b988-a6d6f22c2063 None admin] RBAC: Authorization granted {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:09:13.935438 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 280/545] 10.222.0.27 () {56 vars in 1103 bytes} [Thu Oct 13 04:09:13 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 209 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 13 04:09:13.936350 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 266/546] 10.222.0.27 () {56 vars in 1109 bytes} [Thu Oct 13 04:09:13 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 182 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 13 04:09:13.937321 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d32ce359-1baa-4d7d-ae1c-47d4cf2d55e5 None None] REQUEST_METHOD: `GET` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:09:13.937754 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d32ce359-1baa-4d7d-ae1c-47d4cf2d55e5 None None] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:09:13.937932 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5947b87b-8952-4faa-a6bb-29cbd2244639 None None] REQUEST_METHOD: `GET` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:09:13.938335 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5947b87b-8952-4faa-a6bb-29cbd2244639 None None] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:09:13.938508 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d32ce359-1baa-4d7d-ae1c-47d4cf2d55e5 None None] PATH_INFO: `/` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:09:13.938787 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5947b87b-8952-4faa-a6bb-29cbd2244639 None None] PATH_INFO: `/` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:09:13.940271 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 281/547] 10.222.0.27 () {54 vars in 927 bytes} [Thu Oct 13 04:09:13 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 13 04:09:13.940388 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 267/548] 10.222.0.27 () {54 vars in 927 bytes} [Thu Oct 13 04:09:13 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 13 04:09:13.941240 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d32ce359-1baa-4d7d-ae1c-47d4cf2d55e5 None None] Authenticating user token {{(pid=81409) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:09:13.945206 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5947b87b-8952-4faa-a6bb-29cbd2244639 None None] Authenticating user token {{(pid=81408) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:09:14.051702 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d32ce359-1baa-4d7d-ae1c-47d4cf2d55e5 None None] Validating token access rules against request {{(pid=81409) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:09:14.057365 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5947b87b-8952-4faa-a6bb-29cbd2244639 None None] Validating token access rules against request {{(pid=81408) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:09:14.085405 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG dbcounter [-] [81409] Writing DB stats keystone:SELECT=100 {{(pid=81409) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:09:14.091925 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-16ee8eb4-813b-4284-b730-7e425f5b7376 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81409) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:09:14.092626 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16ee8eb4-813b-4284-b730-7e425f5b7376 None admin] REQUEST_METHOD: `GET` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:09:14.092903 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16ee8eb4-813b-4284-b730-7e425f5b7376 None admin] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:09:14.093174 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16ee8eb4-813b-4284-b730-7e425f5b7376 None admin] PATH_INFO: `/v3/projects/service` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:09:14.093727 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0624b4ca-ec6f-425d-83be-eb5caa661196 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81408) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:09:14.094386 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0624b4ca-ec6f-425d-83be-eb5caa661196 None admin] REQUEST_METHOD: `GET` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:09:14.094662 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0624b4ca-ec6f-425d-83be-eb5caa661196 None admin] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:09:14.094927 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0624b4ca-ec6f-425d-83be-eb5caa661196 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:09:14.100519 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-16ee8eb4-813b-4284-b730-7e425f5b7376 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:09:14.101957 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-16ee8eb4-813b-4284-b730-7e425f5b7376 None admin] RBAC: Authorization granted {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:09:14.105850 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0624b4ca-ec6f-425d-83be-eb5caa661196 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:09:14.107592 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0624b4ca-ec6f-425d-83be-eb5caa661196 None admin] RBAC: Authorization granted {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:09:14.113724 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application [None req-16ee8eb4-813b-4284-b730-7e425f5b7376 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Oct 13 04:09:14.113724 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 13 04:09:14.113724 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 13 04:09:14.113724 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 13 04:09:14.113724 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 13 04:09:14.113724 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 13 04:09:14.113724 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 13 04:09:14.113724 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 13 04:09:14.113724 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 13 04:09:14.113724 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 13 04:09:14.113724 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 13 04:09:14.113724 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 13 04:09:14.113724 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Oct 13 04:09:14.113724 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application return self._get_project(project_id) Oct 13 04:09:14.113724 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Oct 13 04:09:14.114689 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Oct 13 04:09:14.114689 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 13 04:09:14.114689 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 13 04:09:14.114689 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 13 04:09:14.114689 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 13 04:09:14.114689 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 13 04:09:14.114689 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application return self.get_or_create( Oct 13 04:09:14.114689 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 13 04:09:14.114689 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application with Lock( Oct 13 04:09:14.114689 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 13 04:09:14.114689 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application return self._enter() Oct 13 04:09:14.114689 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 13 04:09:14.114689 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 13 04:09:14.114689 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 13 04:09:14.114689 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application return self.creator() Oct 13 04:09:14.114689 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 13 04:09:14.114689 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application created_value = creator( Oct 13 04:09:14.115659 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Oct 13 04:09:14.115659 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Oct 13 04:09:14.115659 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Oct 13 04:09:14.115659 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Oct 13 04:09:14.115659 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Oct 13 04:09:14.115659 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Oct 13 04:09:14.115659 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Oct 13 04:09:14.115659 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application Oct 13 04:09:14.120272 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 282/549] 10.222.0.27 () {56 vars in 1111 bytes} [Thu Oct 13 04:09:13 2022] GET /identity/v3/projects/service => generated 88 bytes in 179 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Oct 13 04:09:14.121239 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85d4a5b4-000c-4835-93e0-b48ac144ec79 None None] REQUEST_METHOD: `POST` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:09:14.121522 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85d4a5b4-000c-4835-93e0-b48ac144ec79 None None] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:09:14.121794 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85d4a5b4-000c-4835-93e0-b48ac144ec79 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:09:14.124866 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application [None req-0624b4ca-ec6f-425d-83be-eb5caa661196 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 13 04:09:14.124866 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 13 04:09:14.124866 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 13 04:09:14.124866 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 13 04:09:14.124866 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 13 04:09:14.124866 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 13 04:09:14.124866 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 13 04:09:14.124866 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 13 04:09:14.124866 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 13 04:09:14.124866 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 13 04:09:14.124866 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 13 04:09:14.124866 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 13 04:09:14.124866 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 13 04:09:14.124866 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 13 04:09:14.124866 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 13 04:09:14.124866 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 13 04:09:14.125774 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 13 04:09:14.125774 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 13 04:09:14.125774 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 13 04:09:14.125774 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 13 04:09:14.125774 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 13 04:09:14.125774 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application return self.get_or_create( Oct 13 04:09:14.125774 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 13 04:09:14.125774 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application with Lock( Oct 13 04:09:14.125774 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 13 04:09:14.125774 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application return self._enter() Oct 13 04:09:14.125774 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 13 04:09:14.125774 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 13 04:09:14.125774 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 13 04:09:14.125774 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application return self.creator() Oct 13 04:09:14.125774 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 13 04:09:14.125774 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application created_value = creator( Oct 13 04:09:14.125774 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 13 04:09:14.129230 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 13 04:09:14.129230 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 13 04:09:14.129230 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application Oct 13 04:09:14.129230 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 268/550] 10.222.0.27 () {56 vars in 1108 bytes} [Thu Oct 13 04:09:13 2022] GET /identity/v3/domains/Default => generated 87 bytes in 185 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 13 04:09:14.129230 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d39cceac-0222-4df1-8835-e7b891993f3a None None] REQUEST_METHOD: `POST` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:09:14.129230 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d39cceac-0222-4df1-8835-e7b891993f3a None None] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:09:14.129230 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d39cceac-0222-4df1-8835-e7b891993f3a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:09:14.142141 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG dbcounter [-] [81408] Writing DB stats keystone:SELECT=100 {{(pid=81408) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:09:14.145722 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.auth.core [None req-85d4a5b4-000c-4835-93e0-b48ac144ec79 None None] MFA Rules not processed for user `0e2d979563d6421995451a0e86bf3022`. Rule list: `[]` (Enabled: `True`). {{(pid=81409) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 13 04:09:14.146920 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.fernet_utils [None req-85d4a5b4-000c-4835-93e0-b48ac144ec79 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81409) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:09:14.148034 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.auth.core [None req-d39cceac-0222-4df1-8835-e7b891993f3a None None] MFA Rules not processed for user `0e2d979563d6421995451a0e86bf3022`. Rule list: `[]` (Enabled: `True`). {{(pid=81408) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 13 04:09:14.149309 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.fernet_utils [None req-d39cceac-0222-4df1-8835-e7b891993f3a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81408) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:09:14.217264 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 283/551] 10.222.0.27 () {58 vars in 1024 bytes} [Thu Oct 13 04:09:14 2022] POST /identity/v3/auth/tokens => generated 903 bytes in 96 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 13 04:09:14.230302 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 269/552] 10.222.0.27 () {58 vars in 1024 bytes} [Thu Oct 13 04:09:14 2022] POST /identity/v3/auth/tokens => generated 903 bytes in 98 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 13 04:09:14.230302 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d39cceac-0222-4df1-8835-e7b891993f3a None None] Authenticating user token {{(pid=81408) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:09:14.231112 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-85d4a5b4-000c-4835-93e0-b48ac144ec79 None None] Authenticating user token {{(pid=81409) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:09:14.379233 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d39cceac-0222-4df1-8835-e7b891993f3a None None] Validating token access rules against request {{(pid=81408) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:09:14.419676 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-85d4a5b4-000c-4835-93e0-b48ac144ec79 None None] Validating token access rules against request {{(pid=81409) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:09:14.448819 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9c86de39-064c-484b-85f9-3ab8da87e210 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81408) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:09:14.449757 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c86de39-064c-484b-85f9-3ab8da87e210 None admin] REQUEST_METHOD: `GET` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:09:14.450240 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c86de39-064c-484b-85f9-3ab8da87e210 None admin] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:09:14.450687 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c86de39-064c-484b-85f9-3ab8da87e210 None admin] PATH_INFO: `/v3/domains` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:09:14.451511 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9c86de39-064c-484b-85f9-3ab8da87e210 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81408) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 13 04:09:14.452063 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9c86de39-064c-484b-85f9-3ab8da87e210 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:09:14.454325 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9c86de39-064c-484b-85f9-3ab8da87e210 None admin] RBAC: Authorization granted {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:09:14.461229 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG dbcounter [-] [81408] Writing DB stats keystone:SELECT=100 {{(pid=81408) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:09:14.468846 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG dbcounter [-] [81409] Writing DB stats keystone:SELECT=100 {{(pid=81409) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:09:14.479255 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 270/553] 10.222.0.27 () {56 vars in 1109 bytes} [Thu Oct 13 04:09:14 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 254 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 13 04:09:14.484516 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9c86de39-064c-484b-85f9-3ab8da87e210 None admin] Authenticating user token {{(pid=81408) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:09:14.492083 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0a2c98f7-2675-42c7-a301-8a1d47b00754 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81409) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:09:14.494609 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a2c98f7-2675-42c7-a301-8a1d47b00754 None admin] REQUEST_METHOD: `GET` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:09:14.495072 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a2c98f7-2675-42c7-a301-8a1d47b00754 None admin] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:09:14.499856 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a2c98f7-2675-42c7-a301-8a1d47b00754 None admin] PATH_INFO: `/v3/projects` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:09:14.500782 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0a2c98f7-2675-42c7-a301-8a1d47b00754 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81409) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 13 04:09:14.501282 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0a2c98f7-2675-42c7-a301-8a1d47b00754 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:09:14.502937 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0a2c98f7-2675-42c7-a301-8a1d47b00754 None admin] RBAC: Authorization granted {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:09:14.516960 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 284/554] 10.222.0.27 () {56 vars in 1148 bytes} [Thu Oct 13 04:09:14 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 299 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 13 04:09:14.522859 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0a2c98f7-2675-42c7-a301-8a1d47b00754 None admin] Authenticating user token {{(pid=81409) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:09:14.633061 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9c86de39-064c-484b-85f9-3ab8da87e210 None admin] Validating token access rules against request {{(pid=81408) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:09:14.637465 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0a2c98f7-2675-42c7-a301-8a1d47b00754 None admin] Validating token access rules against request {{(pid=81409) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:09:14.668395 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2f32ab79-98f0-4d3e-a088-2055e8f8f4ae None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81408) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:09:14.669019 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f32ab79-98f0-4d3e-a088-2055e8f8f4ae None admin] REQUEST_METHOD: `GET` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:09:14.669322 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f32ab79-98f0-4d3e-a088-2055e8f8f4ae None admin] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:09:14.669570 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f32ab79-98f0-4d3e-a088-2055e8f8f4ae None admin] PATH_INFO: `/v3/users/glance-swift` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:09:14.674348 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-81218c1e-7b53-4d82-aba3-875924e629fa None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81409) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:09:14.675274 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81218c1e-7b53-4d82-aba3-875924e629fa None admin] REQUEST_METHOD: `PUT` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:09:14.675748 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81218c1e-7b53-4d82-aba3-875924e629fa None admin] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:09:14.676484 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81218c1e-7b53-4d82-aba3-875924e629fa None admin] PATH_INFO: `/v3/projects/dcec0431fb6849499d6aec35a234daa6/users/90a0f466b664458898cf9dc4112ae651/roles/ef55fbd94e1040be83695754075a7ed5` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:09:14.677083 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2f32ab79-98f0-4d3e-a088-2055e8f8f4ae None admin] RBAC: Authorizing `identity:get_user(user_id=glance-swift)` {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:09:14.678567 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2f32ab79-98f0-4d3e-a088-2055e8f8f4ae None admin] RBAC: Authorization granted {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:09:14.679171 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-81218c1e-7b53-4d82-aba3-875924e629fa None admin] RBAC: Authorizing `identity:create_grant(project_id=dcec0431fb6849499d6aec35a234daa6, user_id=90a0f466b664458898cf9dc4112ae651, role_id=ef55fbd94e1040be83695754075a7ed5)` {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:09:14.682180 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-81218c1e-7b53-4d82-aba3-875924e629fa None admin] RBAC: Authorization granted {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:09:14.691846 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application [None req-2f32ab79-98f0-4d3e-a088-2055e8f8f4ae None admin] Could not find user: glance-swift.: keystone.exception.UserNotFound: Could not find user: glance-swift. Oct 13 04:09:14.691846 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 13 04:09:14.691846 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 13 04:09:14.691846 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 13 04:09:14.691846 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 13 04:09:14.691846 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 13 04:09:14.691846 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 13 04:09:14.691846 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 13 04:09:14.691846 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 13 04:09:14.691846 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 13 04:09:14.691846 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 13 04:09:14.691846 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 13 04:09:14.691846 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Oct 13 04:09:14.691846 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application return self._get_user(user_id) Oct 13 04:09:14.691846 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Oct 13 04:09:14.691846 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Oct 13 04:09:14.692798 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 13 04:09:14.692798 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 13 04:09:14.692798 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Oct 13 04:09:14.692798 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 13 04:09:14.692798 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Oct 13 04:09:14.692798 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 13 04:09:14.692798 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 13 04:09:14.692798 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 13 04:09:14.692798 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 13 04:09:14.692798 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application return self.get_or_create( Oct 13 04:09:14.692798 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 13 04:09:14.692798 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application with Lock( Oct 13 04:09:14.692798 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 13 04:09:14.692798 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application return self._enter() Oct 13 04:09:14.692798 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 13 04:09:14.692798 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 13 04:09:14.692798 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 13 04:09:14.692798 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application return self.creator() Oct 13 04:09:14.693723 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 13 04:09:14.693723 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application created_value = creator( Oct 13 04:09:14.693723 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Oct 13 04:09:14.693723 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Oct 13 04:09:14.693723 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Oct 13 04:09:14.693723 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Oct 13 04:09:14.693723 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Oct 13 04:09:14.693723 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Oct 13 04:09:14.693723 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: glance-swift. Oct 13 04:09:14.693723 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application Oct 13 04:09:14.693723 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 271/555] 10.222.0.27 () {56 vars in 1117 bytes} [Thu Oct 13 04:09:14 2022] GET /identity/v3/users/glance-swift => generated 90 bytes in 209 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 13 04:09:14.693723 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2f32ab79-98f0-4d3e-a088-2055e8f8f4ae None admin] Authenticating user token {{(pid=81408) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:09:14.703561 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 285/556] 10.222.0.27 () {58 vars in 1439 bytes} [Thu Oct 13 04:09:14 2022] PUT /identity/v3/projects/dcec0431fb6849499d6aec35a234daa6/users/90a0f466b664458898cf9dc4112ae651/roles/ef55fbd94e1040be83695754075a7ed5 => generated 0 bytes in 181 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 13 04:09:14.704358 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-81218c1e-7b53-4d82-aba3-875924e629fa None admin] Authenticating user token {{(pid=81409) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:09:14.765732 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG dbcounter [-] [81408] Writing DB stats keystone:SELECT=100 {{(pid=81408) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:09:14.771373 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG dbcounter [-] [81409] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81409) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:09:14.820769 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2f32ab79-98f0-4d3e-a088-2055e8f8f4ae None admin] Validating token access rules against request {{(pid=81408) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:09:14.846120 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-81218c1e-7b53-4d82-aba3-875924e629fa None admin] Validating token access rules against request {{(pid=81409) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:09:14.862617 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1735074a-753f-494e-a7d5-fcdc05abf0ab None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81408) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:09:14.863564 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1735074a-753f-494e-a7d5-fcdc05abf0ab None admin] REQUEST_METHOD: `GET` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:09:14.864138 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1735074a-753f-494e-a7d5-fcdc05abf0ab None admin] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:09:14.864564 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1735074a-753f-494e-a7d5-fcdc05abf0ab None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:09:14.878496 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1735074a-753f-494e-a7d5-fcdc05abf0ab None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 13 04:09:14.885033 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application [None req-1735074a-753f-494e-a7d5-fcdc05abf0ab None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Oct 13 04:09:14.885033 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 13 04:09:14.885033 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 13 04:09:14.885033 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 13 04:09:14.885033 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 13 04:09:14.885033 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 13 04:09:14.885033 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 13 04:09:14.885033 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 13 04:09:14.885033 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 13 04:09:14.885033 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 13 04:09:14.885033 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 13 04:09:14.885033 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 13 04:09:14.885033 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 13 04:09:14.885033 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 13 04:09:14.885033 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 13 04:09:14.885033 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 13 04:09:14.886240 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 13 04:09:14.886240 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 13 04:09:14.886240 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 13 04:09:14.886240 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 13 04:09:14.886240 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 13 04:09:14.886240 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 13 04:09:14.886240 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 13 04:09:14.886240 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 13 04:09:14.886240 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 13 04:09:14.886240 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application return self.get_or_create( Oct 13 04:09:14.886240 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 13 04:09:14.886240 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application with Lock( Oct 13 04:09:14.886240 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 13 04:09:14.886240 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application return self._enter() Oct 13 04:09:14.886240 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 13 04:09:14.886240 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 13 04:09:14.887288 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 13 04:09:14.887288 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application return self.creator() Oct 13 04:09:14.887288 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 13 04:09:14.887288 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application created_value = creator( Oct 13 04:09:14.887288 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 13 04:09:14.887288 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 13 04:09:14.887288 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 13 04:09:14.887288 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 13 04:09:14.887288 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 13 04:09:14.887288 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 13 04:09:14.887288 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Oct 13 04:09:14.887288 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application Oct 13 04:09:14.887959 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2fc184ad-edd5-404d-86e0-393079d3c814 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81409) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:09:14.887959 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2fc184ad-edd5-404d-86e0-393079d3c814 None admin] REQUEST_METHOD: `POST` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:09:14.887959 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2fc184ad-edd5-404d-86e0-393079d3c814 None admin] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:09:14.887959 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2fc184ad-edd5-404d-86e0-393079d3c814 None admin] PATH_INFO: `/v3/services` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:09:14.888222 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2fc184ad-edd5-404d-86e0-393079d3c814 None admin] RBAC: Authorizing `identity:create_service()` {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:09:14.889294 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2fc184ad-edd5-404d-86e0-393079d3c814 None admin] RBAC: Authorization granted {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:09:14.890144 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 272/557] 10.222.0.27 () {56 vars in 1096 bytes} [Thu Oct 13 04:09:14 2022] GET /identity/v3/roles/admin => generated 83 bytes in 196 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 13 04:09:14.891209 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1735074a-753f-494e-a7d5-fcdc05abf0ab None admin] Authenticating user token {{(pid=81408) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:09:14.909321 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 286/558] 10.222.0.27 () {60 vars in 1141 bytes} [Thu Oct 13 04:09:14 2022] POST /identity/v3/services => generated 240 bytes in 205 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 13 04:09:14.910252 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2fc184ad-edd5-404d-86e0-393079d3c814 None admin] Authenticating user token {{(pid=81409) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:09:15.014449 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1735074a-753f-494e-a7d5-fcdc05abf0ab None admin] Validating token access rules against request {{(pid=81408) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:09:15.025870 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG dbcounter [-] [81409] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81409) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:09:15.038029 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2fc184ad-edd5-404d-86e0-393079d3c814 None admin] Validating token access rules against request {{(pid=81409) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:09:15.055965 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG dbcounter [-] [81408] Writing DB stats keystone:SELECT=100 {{(pid=81408) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:09:15.062766 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cd815da9-ce73-4105-a252-eca5fe9231fe None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81408) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:09:15.063606 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd815da9-ce73-4105-a252-eca5fe9231fe None admin] REQUEST_METHOD: `GET` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:09:15.064095 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd815da9-ce73-4105-a252-eca5fe9231fe None admin] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:09:15.064609 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd815da9-ce73-4105-a252-eca5fe9231fe None admin] PATH_INFO: `/v3/users` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:09:15.065598 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cd815da9-ce73-4105-a252-eca5fe9231fe None admin] RBAC: Adding query filter params (domain_id=default, name=glance-swift) {{(pid=81408) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 13 04:09:15.066050 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cd815da9-ce73-4105-a252-eca5fe9231fe None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:09:15.067870 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cd815da9-ce73-4105-a252-eca5fe9231fe None admin] RBAC: Authorization granted {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:09:15.077500 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-43a92cdb-8ccc-43d3-b3e3-5791bfdb16f5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81409) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:09:15.082469 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43a92cdb-8ccc-43d3-b3e3-5791bfdb16f5 None admin] REQUEST_METHOD: `GET` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:09:15.082469 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43a92cdb-8ccc-43d3-b3e3-5791bfdb16f5 None admin] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:09:15.082469 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43a92cdb-8ccc-43d3-b3e3-5791bfdb16f5 None admin] PATH_INFO: `/v3/roles` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:09:15.082469 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-43a92cdb-8ccc-43d3-b3e3-5791bfdb16f5 None admin] RBAC: Adding query filter params (name=admin) {{(pid=81409) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 13 04:09:15.082469 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-43a92cdb-8ccc-43d3-b3e3-5791bfdb16f5 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:09:15.082469 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-43a92cdb-8ccc-43d3-b3e3-5791bfdb16f5 None admin] RBAC: Authorization granted {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:09:15.094237 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 287/559] 10.222.0.27 () {56 vars in 1099 bytes} [Thu Oct 13 04:09:14 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 181 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 13 04:09:15.096502 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-43a92cdb-8ccc-43d3-b3e3-5791bfdb16f5 None admin] Authenticating user token {{(pid=81409) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:09:15.109291 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 273/560] 10.222.0.27 () {56 vars in 1149 bytes} [Thu Oct 13 04:09:14 2022] GET /identity/v3/users?domain_id=default&name=glance-swift => generated 384 bytes in 214 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 13 04:09:15.116104 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cd815da9-ce73-4105-a252-eca5fe9231fe None admin] Authenticating user token {{(pid=81408) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:09:15.251882 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-43a92cdb-8ccc-43d3-b3e3-5791bfdb16f5 None admin] Validating token access rules against request {{(pid=81409) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:09:15.265537 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cd815da9-ce73-4105-a252-eca5fe9231fe None admin] Validating token access rules against request {{(pid=81408) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:09:15.289853 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a7177d3f-7b60-4d8b-bed8-a4202bd66df9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81409) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:09:15.290413 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7177d3f-7b60-4d8b-bed8-a4202bd66df9 None admin] REQUEST_METHOD: `GET` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:09:15.290692 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7177d3f-7b60-4d8b-bed8-a4202bd66df9 None admin] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:09:15.290960 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7177d3f-7b60-4d8b-bed8-a4202bd66df9 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:09:15.301818 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a7177d3f-7b60-4d8b-bed8-a4202bd66df9 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:09:15.303308 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a7177d3f-7b60-4d8b-bed8-a4202bd66df9 None admin] RBAC: Authorization granted {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:09:15.312552 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-289dbd06-6c17-4770-b3b1-83d64f08a09d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81408) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:09:15.313194 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-289dbd06-6c17-4770-b3b1-83d64f08a09d None admin] REQUEST_METHOD: `GET` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:09:15.313515 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-289dbd06-6c17-4770-b3b1-83d64f08a09d None admin] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:09:15.314306 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-289dbd06-6c17-4770-b3b1-83d64f08a09d None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:09:15.320396 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application [None req-a7177d3f-7b60-4d8b-bed8-a4202bd66df9 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 13 04:09:15.320396 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 13 04:09:15.320396 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 13 04:09:15.320396 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 13 04:09:15.320396 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 13 04:09:15.320396 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 13 04:09:15.320396 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 13 04:09:15.320396 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 13 04:09:15.320396 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 13 04:09:15.320396 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 13 04:09:15.320396 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 13 04:09:15.320396 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 13 04:09:15.320396 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 13 04:09:15.320396 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 13 04:09:15.320396 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 13 04:09:15.320396 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 13 04:09:15.321599 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 13 04:09:15.321599 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 13 04:09:15.321599 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 13 04:09:15.321599 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 13 04:09:15.321599 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 13 04:09:15.321599 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application return self.get_or_create( Oct 13 04:09:15.321599 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 13 04:09:15.321599 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application with Lock( Oct 13 04:09:15.321599 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 13 04:09:15.321599 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application return self._enter() Oct 13 04:09:15.321599 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 13 04:09:15.321599 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 13 04:09:15.321599 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 13 04:09:15.321599 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application return self.creator() Oct 13 04:09:15.321599 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 13 04:09:15.321599 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application created_value = creator( Oct 13 04:09:15.321599 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 13 04:09:15.322515 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 13 04:09:15.322515 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 13 04:09:15.322515 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application Oct 13 04:09:15.325743 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-289dbd06-6c17-4770-b3b1-83d64f08a09d None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:09:15.326278 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 288/561] 10.222.0.27 () {56 vars in 1108 bytes} [Thu Oct 13 04:09:15 2022] GET /identity/v3/domains/Default => generated 87 bytes in 226 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 13 04:09:15.327314 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-289dbd06-6c17-4770-b3b1-83d64f08a09d None admin] RBAC: Authorization granted {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:09:15.327806 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a7177d3f-7b60-4d8b-bed8-a4202bd66df9 None admin] Authenticating user token {{(pid=81409) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:09:15.346892 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG dbcounter [-] [81409] Writing DB stats keystone:SELECT=100 {{(pid=81409) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:09:15.351361 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application [None req-289dbd06-6c17-4770-b3b1-83d64f08a09d None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 13 04:09:15.351361 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 13 04:09:15.351361 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 13 04:09:15.351361 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 13 04:09:15.351361 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 13 04:09:15.351361 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 13 04:09:15.351361 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 13 04:09:15.351361 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 13 04:09:15.351361 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 13 04:09:15.351361 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 13 04:09:15.351361 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 13 04:09:15.351361 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 13 04:09:15.351361 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 13 04:09:15.351361 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 13 04:09:15.351361 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 13 04:09:15.351361 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 13 04:09:15.352508 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 13 04:09:15.352508 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 13 04:09:15.352508 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 13 04:09:15.352508 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 13 04:09:15.352508 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 13 04:09:15.352508 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application return self.get_or_create( Oct 13 04:09:15.352508 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 13 04:09:15.352508 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application with Lock( Oct 13 04:09:15.352508 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 13 04:09:15.352508 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application return self._enter() Oct 13 04:09:15.352508 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 13 04:09:15.352508 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 13 04:09:15.352508 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 13 04:09:15.352508 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application return self.creator() Oct 13 04:09:15.352508 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 13 04:09:15.352508 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application created_value = creator( Oct 13 04:09:15.352508 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 13 04:09:15.353827 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 13 04:09:15.353827 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 13 04:09:15.353827 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application Oct 13 04:09:15.355660 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 274/562] 10.222.0.27 () {56 vars in 1108 bytes} [Thu Oct 13 04:09:15 2022] GET /identity/v3/domains/Default => generated 87 bytes in 240 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Oct 13 04:09:15.361178 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-289dbd06-6c17-4770-b3b1-83d64f08a09d None admin] Authenticating user token {{(pid=81408) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:09:15.405811 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG dbcounter [-] [81408] Writing DB stats keystone:SELECT=100 {{(pid=81408) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:09:15.471155 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a7177d3f-7b60-4d8b-bed8-a4202bd66df9 None admin] Validating token access rules against request {{(pid=81409) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:09:15.533893 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3e4993cb-b1e4-4758-be73-f2fca20859f1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81409) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:09:15.534534 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e4993cb-b1e4-4758-be73-f2fca20859f1 None admin] REQUEST_METHOD: `GET` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:09:15.534805 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e4993cb-b1e4-4758-be73-f2fca20859f1 None admin] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:09:15.535067 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e4993cb-b1e4-4758-be73-f2fca20859f1 None admin] PATH_INFO: `/v3/domains` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:09:15.535901 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3e4993cb-b1e4-4758-be73-f2fca20859f1 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81409) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 13 04:09:15.536199 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3e4993cb-b1e4-4758-be73-f2fca20859f1 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:09:15.538493 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3e4993cb-b1e4-4758-be73-f2fca20859f1 None admin] RBAC: Authorization granted {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:09:15.562170 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-289dbd06-6c17-4770-b3b1-83d64f08a09d None admin] Validating token access rules against request {{(pid=81408) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:09:15.567005 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 289/563] 10.222.0.27 () {56 vars in 1109 bytes} [Thu Oct 13 04:09:15 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 240 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 13 04:09:15.573635 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3e4993cb-b1e4-4758-be73-f2fca20859f1 None admin] Authenticating user token {{(pid=81409) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:09:15.611208 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-02775f03-e6af-4d01-b765-fac88a34b1bf None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81408) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:09:15.612236 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-02775f03-e6af-4d01-b765-fac88a34b1bf None admin] REQUEST_METHOD: `GET` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:09:15.612829 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-02775f03-e6af-4d01-b765-fac88a34b1bf None admin] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:09:15.613388 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-02775f03-e6af-4d01-b765-fac88a34b1bf None admin] PATH_INFO: `/v3/domains` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:09:15.614254 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-02775f03-e6af-4d01-b765-fac88a34b1bf None admin] RBAC: Adding query filter params (name=Default) {{(pid=81408) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 13 04:09:15.614789 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-02775f03-e6af-4d01-b765-fac88a34b1bf None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:09:15.616359 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-02775f03-e6af-4d01-b765-fac88a34b1bf None admin] RBAC: Authorization granted {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:09:15.629365 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 275/564] 10.222.0.27 () {56 vars in 1109 bytes} [Thu Oct 13 04:09:15 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 268 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 13 04:09:15.635912 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-02775f03-e6af-4d01-b765-fac88a34b1bf None admin] Authenticating user token {{(pid=81408) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:09:15.738577 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG dbcounter [-] [81409] Writing DB stats keystone:SELECT=100 {{(pid=81409) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:09:15.753589 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG dbcounter [-] [81408] Writing DB stats keystone:SELECT=100 {{(pid=81408) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:09:15.781217 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-02775f03-e6af-4d01-b765-fac88a34b1bf None admin] Validating token access rules against request {{(pid=81408) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:09:15.787731 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3e4993cb-b1e4-4758-be73-f2fca20859f1 None admin] Validating token access rules against request {{(pid=81409) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:09:15.839100 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-03514120-fe15-49b7-98b6-c5447c9e9197 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81408) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:09:15.840046 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03514120-fe15-49b7-98b6-c5447c9e9197 None admin] REQUEST_METHOD: `GET` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:09:15.840326 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03514120-fe15-49b7-98b6-c5447c9e9197 None admin] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:09:15.840596 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03514120-fe15-49b7-98b6-c5447c9e9197 None admin] PATH_INFO: `/v3/projects/service` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:09:15.848381 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-03514120-fe15-49b7-98b6-c5447c9e9197 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:09:15.849976 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-03514120-fe15-49b7-98b6-c5447c9e9197 None admin] RBAC: Authorization granted {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:09:15.862748 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application [None req-03514120-fe15-49b7-98b6-c5447c9e9197 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Oct 13 04:09:15.862748 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 13 04:09:15.862748 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 13 04:09:15.862748 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 13 04:09:15.862748 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 13 04:09:15.862748 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 13 04:09:15.862748 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 13 04:09:15.862748 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 13 04:09:15.862748 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 13 04:09:15.862748 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 13 04:09:15.862748 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 13 04:09:15.862748 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 13 04:09:15.862748 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Oct 13 04:09:15.862748 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application return self._get_project(project_id) Oct 13 04:09:15.862748 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Oct 13 04:09:15.863614 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Oct 13 04:09:15.863614 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 13 04:09:15.863614 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 13 04:09:15.863614 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 13 04:09:15.863614 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 13 04:09:15.863614 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 13 04:09:15.863614 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application return self.get_or_create( Oct 13 04:09:15.863614 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 13 04:09:15.863614 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application with Lock( Oct 13 04:09:15.863614 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 13 04:09:15.863614 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application return self._enter() Oct 13 04:09:15.863614 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 13 04:09:15.863614 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 13 04:09:15.863614 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 13 04:09:15.863614 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application return self.creator() Oct 13 04:09:15.863614 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 13 04:09:15.863614 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application created_value = creator( Oct 13 04:09:15.864488 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Oct 13 04:09:15.864488 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Oct 13 04:09:15.864488 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Oct 13 04:09:15.864488 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Oct 13 04:09:15.864488 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Oct 13 04:09:15.864488 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Oct 13 04:09:15.864488 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Oct 13 04:09:15.864488 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application Oct 13 04:09:15.864488 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 276/565] 10.222.0.27 () {56 vars in 1111 bytes} [Thu Oct 13 04:09:15 2022] GET /identity/v3/projects/service => generated 88 bytes in 229 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 13 04:09:15.869785 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-03514120-fe15-49b7-98b6-c5447c9e9197 None admin] Authenticating user token {{(pid=81408) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:09:15.879320 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-384e2ec8-9a6e-4c9d-b419-621c31180188 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81409) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:09:15.879962 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-384e2ec8-9a6e-4c9d-b419-621c31180188 None admin] REQUEST_METHOD: `GET` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:09:15.880238 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-384e2ec8-9a6e-4c9d-b419-621c31180188 None admin] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:09:15.880506 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-384e2ec8-9a6e-4c9d-b419-621c31180188 None admin] PATH_INFO: `/v3/users/nova` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:09:15.901664 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-384e2ec8-9a6e-4c9d-b419-621c31180188 None admin] RBAC: Authorizing `identity:get_user(user_id=nova)` {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:09:15.903195 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-384e2ec8-9a6e-4c9d-b419-621c31180188 None admin] RBAC: Authorization granted {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:09:15.924707 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application [None req-384e2ec8-9a6e-4c9d-b419-621c31180188 None admin] Could not find user: nova.: keystone.exception.UserNotFound: Could not find user: nova. Oct 13 04:09:15.924707 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 13 04:09:15.924707 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 13 04:09:15.924707 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 13 04:09:15.924707 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 13 04:09:15.924707 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 13 04:09:15.924707 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 13 04:09:15.924707 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 13 04:09:15.924707 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 13 04:09:15.924707 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 13 04:09:15.924707 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 13 04:09:15.924707 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 13 04:09:15.924707 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Oct 13 04:09:15.924707 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application return self._get_user(user_id) Oct 13 04:09:15.924707 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Oct 13 04:09:15.924707 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Oct 13 04:09:15.925711 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 13 04:09:15.925711 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 13 04:09:15.925711 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Oct 13 04:09:15.925711 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 13 04:09:15.925711 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Oct 13 04:09:15.925711 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 13 04:09:15.925711 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 13 04:09:15.925711 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 13 04:09:15.925711 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 13 04:09:15.925711 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application return self.get_or_create( Oct 13 04:09:15.925711 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 13 04:09:15.925711 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application with Lock( Oct 13 04:09:15.925711 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 13 04:09:15.925711 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application return self._enter() Oct 13 04:09:15.925711 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 13 04:09:15.925711 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 13 04:09:15.925711 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 13 04:09:15.925711 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application return self.creator() Oct 13 04:09:15.927283 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 13 04:09:15.927283 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application created_value = creator( Oct 13 04:09:15.927283 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Oct 13 04:09:15.927283 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Oct 13 04:09:15.927283 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Oct 13 04:09:15.927283 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Oct 13 04:09:15.927283 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Oct 13 04:09:15.927283 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Oct 13 04:09:15.927283 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: nova. Oct 13 04:09:15.927283 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application Oct 13 04:09:15.927283 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 290/566] 10.222.0.27 () {56 vars in 1093 bytes} [Thu Oct 13 04:09:15 2022] GET /identity/v3/users/nova => generated 82 bytes in 354 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 13 04:09:15.933969 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-384e2ec8-9a6e-4c9d-b419-621c31180188 None admin] Authenticating user token {{(pid=81409) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:09:16.046017 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-384e2ec8-9a6e-4c9d-b419-621c31180188 None admin] Validating token access rules against request {{(pid=81409) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:09:16.083670 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0274e738-ede2-4c15-84e8-8bd7fc453a3d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81409) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:09:16.084287 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0274e738-ede2-4c15-84e8-8bd7fc453a3d None admin] REQUEST_METHOD: `GET` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:09:16.084567 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0274e738-ede2-4c15-84e8-8bd7fc453a3d None admin] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:09:16.084845 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0274e738-ede2-4c15-84e8-8bd7fc453a3d None admin] PATH_INFO: `/v3/users` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:09:16.085569 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0274e738-ede2-4c15-84e8-8bd7fc453a3d None admin] RBAC: Adding query filter params (domain_id=default, name=nova) {{(pid=81409) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 13 04:09:16.085677 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-03514120-fe15-49b7-98b6-c5447c9e9197 None admin] Validating token access rules against request {{(pid=81408) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:09:16.086086 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0274e738-ede2-4c15-84e8-8bd7fc453a3d None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:09:16.087154 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG dbcounter [-] [81409] Writing DB stats keystone:SELECT=100 {{(pid=81409) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:09:16.089659 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0274e738-ede2-4c15-84e8-8bd7fc453a3d None admin] RBAC: Authorization granted {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:09:16.109626 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 291/567] 10.222.0.27 () {56 vars in 1133 bytes} [Thu Oct 13 04:09:15 2022] GET /identity/v3/users?domain_id=default&name=nova => generated 368 bytes in 176 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 13 04:09:16.111069 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99b6761e-b986-48db-8183-dd7f85769098 None None] REQUEST_METHOD: `GET` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:09:16.111365 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99b6761e-b986-48db-8183-dd7f85769098 None None] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:09:16.111689 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99b6761e-b986-48db-8183-dd7f85769098 None None] PATH_INFO: `/` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:09:16.113160 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 292/568] 10.222.0.27 () {54 vars in 927 bytes} [Thu Oct 13 04:09:16 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 13 04:09:16.116968 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-99b6761e-b986-48db-8183-dd7f85769098 None None] Authenticating user token {{(pid=81409) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:09:16.136527 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7400fb99-fe32-41d5-ae7f-255bcfa3572d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81408) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:09:16.137413 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7400fb99-fe32-41d5-ae7f-255bcfa3572d None admin] REQUEST_METHOD: `GET` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:09:16.137687 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7400fb99-fe32-41d5-ae7f-255bcfa3572d None admin] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:09:16.137958 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7400fb99-fe32-41d5-ae7f-255bcfa3572d None admin] PATH_INFO: `/v3/projects` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:09:16.138829 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7400fb99-fe32-41d5-ae7f-255bcfa3572d None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81408) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 13 04:09:16.139135 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7400fb99-fe32-41d5-ae7f-255bcfa3572d None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:09:16.140914 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7400fb99-fe32-41d5-ae7f-255bcfa3572d None admin] RBAC: Authorization granted {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:09:16.159175 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 277/569] 10.222.0.27 () {56 vars in 1148 bytes} [Thu Oct 13 04:09:15 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 290 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 13 04:09:16.160524 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-02d9076d-8cc5-432d-a05e-3f221fefca66 None None] REQUEST_METHOD: `POST` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:09:16.160803 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-02d9076d-8cc5-432d-a05e-3f221fefca66 None None] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:09:16.161068 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-02d9076d-8cc5-432d-a05e-3f221fefca66 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:09:16.181183 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG dbcounter [-] [81408] Writing DB stats keystone:SELECT=100 {{(pid=81408) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:09:16.190075 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.auth.core [None req-02d9076d-8cc5-432d-a05e-3f221fefca66 None None] MFA Rules not processed for user `0e2d979563d6421995451a0e86bf3022`. Rule list: `[]` (Enabled: `True`). {{(pid=81408) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 13 04:09:16.192285 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.fernet_utils [None req-02d9076d-8cc5-432d-a05e-3f221fefca66 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81408) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:09:16.241564 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-99b6761e-b986-48db-8183-dd7f85769098 None None] Validating token access rules against request {{(pid=81409) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:09:16.279195 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-325b6594-10cb-4e91-993f-2e09a9c822cf None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81409) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:09:16.279806 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-325b6594-10cb-4e91-993f-2e09a9c822cf None admin] REQUEST_METHOD: `GET` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:09:16.280134 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-325b6594-10cb-4e91-993f-2e09a9c822cf None admin] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:09:16.280402 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-325b6594-10cb-4e91-993f-2e09a9c822cf None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:09:16.286463 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 278/570] 10.222.0.27 () {58 vars in 1024 bytes} [Thu Oct 13 04:09:16 2022] POST /identity/v3/auth/tokens => generated 1002 bytes in 127 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Oct 13 04:09:16.287768 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-02d9076d-8cc5-432d-a05e-3f221fefca66 None None] Authenticating user token {{(pid=81408) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:09:16.394009 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-325b6594-10cb-4e91-993f-2e09a9c822cf None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:09:16.396210 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-325b6594-10cb-4e91-993f-2e09a9c822cf None admin] RBAC: Authorization granted {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:09:16.398404 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG dbcounter [-] [81409] Writing DB stats keystone:SELECT=100 {{(pid=81409) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:09:16.518348 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 293/571] 10.222.0.27 () {58 vars in 1282 bytes} [Thu Oct 13 04:09:16 2022] GET /identity/v3/auth/tokens => generated 1002 bytes in 401 msecs (HTTP/1.1 200) 6 headers in 359 bytes (1 switches on core 0) Oct 13 04:09:16.523657 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-325b6594-10cb-4e91-993f-2e09a9c822cf None admin] Authenticating user token {{(pid=81409) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:09:16.572052 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-02d9076d-8cc5-432d-a05e-3f221fefca66 None None] Validating token access rules against request {{(pid=81408) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:09:16.619795 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-48625bfd-4107-408a-ba91-64dc160b9ed2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81408) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:09:16.620458 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48625bfd-4107-408a-ba91-64dc160b9ed2 None admin] REQUEST_METHOD: `PUT` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:09:16.621252 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48625bfd-4107-408a-ba91-64dc160b9ed2 None admin] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:09:16.621542 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48625bfd-4107-408a-ba91-64dc160b9ed2 None admin] PATH_INFO: `/v3/projects/dcec0431fb6849499d6aec35a234daa6/users/487b1605fd4143cfaa422a3a7a0344c7/roles/4cfff1d3d68e49c9b04231a19cf76ede` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:09:16.623789 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG dbcounter [-] [81409] Writing DB stats keystone:SELECT=100 {{(pid=81409) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:09:16.627566 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG dbcounter [-] [81408] Writing DB stats keystone:SELECT=100 {{(pid=81408) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:09:16.637234 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-325b6594-10cb-4e91-993f-2e09a9c822cf None admin] Validating token access rules against request {{(pid=81409) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:09:16.645167 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-48625bfd-4107-408a-ba91-64dc160b9ed2 None admin] RBAC: Authorizing `identity:create_grant(project_id=dcec0431fb6849499d6aec35a234daa6, user_id=487b1605fd4143cfaa422a3a7a0344c7, role_id=4cfff1d3d68e49c9b04231a19cf76ede)` {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:09:16.647214 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-48625bfd-4107-408a-ba91-64dc160b9ed2 None admin] RBAC: Authorization granted {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:09:16.668870 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 279/572] 10.222.0.27 () {58 vars in 1439 bytes} [Thu Oct 13 04:09:16 2022] PUT /identity/v3/projects/dcec0431fb6849499d6aec35a234daa6/users/487b1605fd4143cfaa422a3a7a0344c7/roles/4cfff1d3d68e49c9b04231a19cf76ede => generated 0 bytes in 382 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 13 04:09:16.670570 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-48625bfd-4107-408a-ba91-64dc160b9ed2 None admin] Authenticating user token {{(pid=81408) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:09:16.677707 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c90327f5-a7d9-4bfa-bc71-07741cde48a2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81409) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:09:16.678784 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c90327f5-a7d9-4bfa-bc71-07741cde48a2 None admin] REQUEST_METHOD: `GET` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:09:16.679282 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c90327f5-a7d9-4bfa-bc71-07741cde48a2 None admin] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:09:16.679753 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c90327f5-a7d9-4bfa-bc71-07741cde48a2 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:09:16.692559 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c90327f5-a7d9-4bfa-bc71-07741cde48a2 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:09:16.694429 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c90327f5-a7d9-4bfa-bc71-07741cde48a2 None admin] RBAC: Authorization granted {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:09:16.710881 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application [None req-c90327f5-a7d9-4bfa-bc71-07741cde48a2 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 13 04:09:16.710881 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 13 04:09:16.710881 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 13 04:09:16.710881 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 13 04:09:16.710881 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 13 04:09:16.710881 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 13 04:09:16.710881 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 13 04:09:16.710881 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 13 04:09:16.710881 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 13 04:09:16.710881 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 13 04:09:16.710881 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 13 04:09:16.710881 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 13 04:09:16.710881 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 13 04:09:16.710881 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 13 04:09:16.710881 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 13 04:09:16.710881 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 13 04:09:16.712500 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 13 04:09:16.712500 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 13 04:09:16.712500 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 13 04:09:16.712500 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 13 04:09:16.712500 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 13 04:09:16.712500 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application return self.get_or_create( Oct 13 04:09:16.712500 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 13 04:09:16.712500 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application with Lock( Oct 13 04:09:16.712500 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 13 04:09:16.712500 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application return self._enter() Oct 13 04:09:16.712500 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 13 04:09:16.712500 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 13 04:09:16.712500 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 13 04:09:16.712500 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application return self.creator() Oct 13 04:09:16.712500 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 13 04:09:16.712500 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application created_value = creator( Oct 13 04:09:16.712500 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 13 04:09:16.713374 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 13 04:09:16.713374 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 13 04:09:16.713374 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application Oct 13 04:09:16.715673 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 294/573] 10.222.0.27 () {56 vars in 1108 bytes} [Thu Oct 13 04:09:16 2022] GET /identity/v3/domains/Default => generated 87 bytes in 192 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Oct 13 04:09:16.721034 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c90327f5-a7d9-4bfa-bc71-07741cde48a2 None admin] Authenticating user token {{(pid=81409) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:09:16.802314 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-48625bfd-4107-408a-ba91-64dc160b9ed2 None admin] Validating token access rules against request {{(pid=81408) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:09:16.847105 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-df6a8a6b-8dbd-4f24-ab6d-2cb786ef590e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81408) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:09:16.847801 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df6a8a6b-8dbd-4f24-ab6d-2cb786ef590e None admin] REQUEST_METHOD: `GET` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:09:16.848083 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df6a8a6b-8dbd-4f24-ab6d-2cb786ef590e None admin] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:09:16.848354 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df6a8a6b-8dbd-4f24-ab6d-2cb786ef590e None admin] PATH_INFO: `/v3/services/block-storage` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:09:16.848945 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-df6a8a6b-8dbd-4f24-ab6d-2cb786ef590e None admin] RBAC: Authorizing `identity:get_service(service_id=block-storage)` {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:09:16.850914 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-df6a8a6b-8dbd-4f24-ab6d-2cb786ef590e None admin] RBAC: Authorization granted {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:09:16.866230 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application [None req-df6a8a6b-8dbd-4f24-ab6d-2cb786ef590e None admin] Could not find service: block-storage.: keystone.exception.ServiceNotFound: Could not find service: block-storage. Oct 13 04:09:16.866230 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 13 04:09:16.866230 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Oct 13 04:09:16.866230 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Oct 13 04:09:16.866230 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Oct 13 04:09:16.866230 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Oct 13 04:09:16.866230 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Oct 13 04:09:16.866230 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 13 04:09:16.866230 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: block-storage. Oct 13 04:09:16.866230 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application Oct 13 04:09:16.866230 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Oct 13 04:09:16.866230 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application Oct 13 04:09:16.866230 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 13 04:09:16.866230 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 13 04:09:16.866230 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 13 04:09:16.866230 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 13 04:09:16.866230 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 13 04:09:16.867259 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 13 04:09:16.867259 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 13 04:09:16.867259 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 13 04:09:16.867259 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 13 04:09:16.867259 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 13 04:09:16.867259 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 13 04:09:16.867259 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Oct 13 04:09:16.867259 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application return self._get_service(service_id) Oct 13 04:09:16.867259 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Oct 13 04:09:16.867259 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Oct 13 04:09:16.867259 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 13 04:09:16.867259 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 13 04:09:16.867259 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 13 04:09:16.867259 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 13 04:09:16.867259 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 13 04:09:16.867259 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application return self.get_or_create( Oct 13 04:09:16.868628 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 13 04:09:16.868628 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application with Lock( Oct 13 04:09:16.868628 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 13 04:09:16.868628 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application return self._enter() Oct 13 04:09:16.868628 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 13 04:09:16.868628 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 13 04:09:16.868628 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 13 04:09:16.868628 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application return self.creator() Oct 13 04:09:16.868628 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 13 04:09:16.868628 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application created_value = creator( Oct 13 04:09:16.868628 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Oct 13 04:09:16.868628 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 13 04:09:16.868628 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: block-storage. Oct 13 04:09:16.868628 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application Oct 13 04:09:16.868628 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 280/574] 10.222.0.27 () {56 vars in 1129 bytes} [Thu Oct 13 04:09:16 2022] GET /identity/v3/services/block-storage => generated 94 bytes in 195 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 13 04:09:16.870811 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-df6a8a6b-8dbd-4f24-ab6d-2cb786ef590e None admin] Authenticating user token {{(pid=81408) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:09:16.957611 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c90327f5-a7d9-4bfa-bc71-07741cde48a2 None admin] Validating token access rules against request {{(pid=81409) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:09:16.973558 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG dbcounter [-] [81408] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81408) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:09:17.016014 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-318dc886-a259-47da-9bad-ba028cd6fd70 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81409) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:09:17.016630 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-318dc886-a259-47da-9bad-ba028cd6fd70 None admin] REQUEST_METHOD: `GET` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:09:17.016902 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-318dc886-a259-47da-9bad-ba028cd6fd70 None admin] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:09:17.017167 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-318dc886-a259-47da-9bad-ba028cd6fd70 None admin] PATH_INFO: `/v3/domains` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:09:17.017807 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-318dc886-a259-47da-9bad-ba028cd6fd70 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81409) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 13 04:09:17.018123 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-318dc886-a259-47da-9bad-ba028cd6fd70 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:09:17.019618 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-318dc886-a259-47da-9bad-ba028cd6fd70 None admin] RBAC: Authorization granted {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:09:17.038349 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-df6a8a6b-8dbd-4f24-ab6d-2cb786ef590e None admin] Validating token access rules against request {{(pid=81408) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:09:17.044318 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 295/575] 10.222.0.27 () {56 vars in 1109 bytes} [Thu Oct 13 04:09:16 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 324 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 13 04:09:17.051663 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-318dc886-a259-47da-9bad-ba028cd6fd70 None admin] Authenticating user token {{(pid=81409) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:09:17.057774 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG dbcounter [-] [81409] Writing DB stats keystone:SELECT=100 {{(pid=81409) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:09:17.084667 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4e07be16-aa7e-4480-8861-4e44e7c672dd None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81408) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:09:17.085782 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e07be16-aa7e-4480-8861-4e44e7c672dd None admin] REQUEST_METHOD: `GET` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:09:17.087210 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e07be16-aa7e-4480-8861-4e44e7c672dd None admin] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:09:17.087691 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e07be16-aa7e-4480-8861-4e44e7c672dd None admin] PATH_INFO: `/v3/services` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:09:17.088596 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4e07be16-aa7e-4480-8861-4e44e7c672dd None admin] RBAC: Adding query filter params (name=block-storage) {{(pid=81408) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 13 04:09:17.089200 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4e07be16-aa7e-4480-8861-4e44e7c672dd None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:09:17.090845 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4e07be16-aa7e-4480-8861-4e44e7c672dd None admin] RBAC: Authorization granted {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:09:17.096756 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 281/576] 10.222.0.27 () {56 vars in 1124 bytes} [Thu Oct 13 04:09:16 2022] GET /identity/v3/services?name=block-storage => generated 130 bytes in 226 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 13 04:09:17.103782 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4e07be16-aa7e-4480-8861-4e44e7c672dd None admin] Authenticating user token {{(pid=81408) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:09:17.206829 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-318dc886-a259-47da-9bad-ba028cd6fd70 None admin] Validating token access rules against request {{(pid=81409) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:09:17.246400 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e7897fa0-0a37-45db-a6a3-ab34cbe1fde1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81409) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:09:17.247085 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7897fa0-0a37-45db-a6a3-ab34cbe1fde1 None admin] REQUEST_METHOD: `GET` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:09:17.247361 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7897fa0-0a37-45db-a6a3-ab34cbe1fde1 None admin] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:09:17.247633 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7897fa0-0a37-45db-a6a3-ab34cbe1fde1 None admin] PATH_INFO: `/v3/projects/service` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:09:17.255342 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e7897fa0-0a37-45db-a6a3-ab34cbe1fde1 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:09:17.256796 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e7897fa0-0a37-45db-a6a3-ab34cbe1fde1 None admin] RBAC: Authorization granted {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:09:17.271044 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application [None req-e7897fa0-0a37-45db-a6a3-ab34cbe1fde1 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Oct 13 04:09:17.271044 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 13 04:09:17.271044 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 13 04:09:17.271044 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 13 04:09:17.271044 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 13 04:09:17.271044 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 13 04:09:17.271044 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 13 04:09:17.271044 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 13 04:09:17.271044 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 13 04:09:17.271044 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 13 04:09:17.271044 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 13 04:09:17.271044 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 13 04:09:17.271044 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Oct 13 04:09:17.271044 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application return self._get_project(project_id) Oct 13 04:09:17.271044 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Oct 13 04:09:17.271958 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Oct 13 04:09:17.271958 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 13 04:09:17.271958 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 13 04:09:17.271958 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 13 04:09:17.271958 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 13 04:09:17.271958 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 13 04:09:17.271958 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application return self.get_or_create( Oct 13 04:09:17.271958 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 13 04:09:17.271958 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application with Lock( Oct 13 04:09:17.271958 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 13 04:09:17.271958 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application return self._enter() Oct 13 04:09:17.271958 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 13 04:09:17.271958 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 13 04:09:17.271958 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 13 04:09:17.271958 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application return self.creator() Oct 13 04:09:17.271958 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 13 04:09:17.271958 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application created_value = creator( Oct 13 04:09:17.272923 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Oct 13 04:09:17.272923 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Oct 13 04:09:17.272923 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Oct 13 04:09:17.272923 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Oct 13 04:09:17.272923 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Oct 13 04:09:17.272923 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Oct 13 04:09:17.272923 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Oct 13 04:09:17.272923 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application Oct 13 04:09:17.272923 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 296/577] 10.222.0.27 () {56 vars in 1111 bytes} [Thu Oct 13 04:09:17 2022] GET /identity/v3/projects/service => generated 88 bytes in 221 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 13 04:09:17.281369 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e7897fa0-0a37-45db-a6a3-ab34cbe1fde1 None admin] Authenticating user token {{(pid=81409) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:09:17.361026 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4e07be16-aa7e-4480-8861-4e44e7c672dd None admin] Validating token access rules against request {{(pid=81408) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:09:17.371090 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG dbcounter [-] [81409] Writing DB stats keystone:SELECT=100 {{(pid=81409) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:09:17.375846 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG dbcounter [-] [81408] Writing DB stats keystone:SELECT=100 {{(pid=81408) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:09:17.399950 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9240d274-70e1-4f51-ba88-755cba7a45cc None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81408) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:09:17.400567 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9240d274-70e1-4f51-ba88-755cba7a45cc None admin] REQUEST_METHOD: `GET` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:09:17.400838 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9240d274-70e1-4f51-ba88-755cba7a45cc None admin] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:09:17.401104 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9240d274-70e1-4f51-ba88-755cba7a45cc None admin] PATH_INFO: `/v3/services` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:09:17.402126 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9240d274-70e1-4f51-ba88-755cba7a45cc None admin] RBAC: Adding query filter params (type=block-storage) {{(pid=81408) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 13 04:09:17.402467 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9240d274-70e1-4f51-ba88-755cba7a45cc None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:09:17.404168 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9240d274-70e1-4f51-ba88-755cba7a45cc None admin] RBAC: Authorization granted {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:09:17.415953 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 282/578] 10.222.0.27 () {56 vars in 1124 bytes} [Thu Oct 13 04:09:17 2022] GET /identity/v3/services?type=block-storage => generated 367 bytes in 309 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 13 04:09:17.419404 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ab8e182-d5ee-4455-a85b-37fb71b9c42b None None] REQUEST_METHOD: `GET` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:09:17.419876 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ab8e182-d5ee-4455-a85b-37fb71b9c42b None None] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:09:17.420185 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ab8e182-d5ee-4455-a85b-37fb71b9c42b None None] PATH_INFO: `/` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:09:17.421682 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 283/579] 10.222.0.27 () {54 vars in 927 bytes} [Thu Oct 13 04:09:17 2022] GET /identity => generated 270 bytes in 5 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 13 04:09:17.422425 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5ab8e182-d5ee-4455-a85b-37fb71b9c42b None None] Authenticating user token {{(pid=81408) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:09:17.439254 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e7897fa0-0a37-45db-a6a3-ab34cbe1fde1 None admin] Validating token access rules against request {{(pid=81409) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:09:17.479529 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-93a17873-3b4d-4976-a58c-caf326dde8a1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81409) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:09:17.480490 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93a17873-3b4d-4976-a58c-caf326dde8a1 None admin] REQUEST_METHOD: `GET` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:09:17.480909 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93a17873-3b4d-4976-a58c-caf326dde8a1 None admin] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:09:17.481319 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93a17873-3b4d-4976-a58c-caf326dde8a1 None admin] PATH_INFO: `/v3/projects` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:09:17.482158 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-93a17873-3b4d-4976-a58c-caf326dde8a1 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81409) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 13 04:09:17.482745 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-93a17873-3b4d-4976-a58c-caf326dde8a1 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:09:17.484407 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-93a17873-3b4d-4976-a58c-caf326dde8a1 None admin] RBAC: Authorization granted {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:09:17.503392 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 297/580] 10.222.0.27 () {56 vars in 1148 bytes} [Thu Oct 13 04:09:17 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 218 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 13 04:09:17.505030 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-890fec3c-9a6f-4bc0-b5ae-a7c1e988e923 None None] REQUEST_METHOD: `POST` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:09:17.505478 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-890fec3c-9a6f-4bc0-b5ae-a7c1e988e923 None None] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:09:17.505951 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-890fec3c-9a6f-4bc0-b5ae-a7c1e988e923 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:09:17.526349 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.auth.core [None req-890fec3c-9a6f-4bc0-b5ae-a7c1e988e923 None None] MFA Rules not processed for user `0e2d979563d6421995451a0e86bf3022`. Rule list: `[]` (Enabled: `True`). {{(pid=81409) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 13 04:09:17.527771 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.fernet_utils [None req-890fec3c-9a6f-4bc0-b5ae-a7c1e988e923 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81409) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:09:17.538911 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5ab8e182-d5ee-4455-a85b-37fb71b9c42b None None] Validating token access rules against request {{(pid=81408) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:09:17.575752 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-06fc83bc-4600-4533-992e-455686e95aec None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81408) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:09:17.576727 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06fc83bc-4600-4533-992e-455686e95aec None admin] REQUEST_METHOD: `GET` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:09:17.577006 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06fc83bc-4600-4533-992e-455686e95aec None admin] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:09:17.577290 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06fc83bc-4600-4533-992e-455686e95aec None admin] PATH_INFO: `/v3/endpoints` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:09:17.578350 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-06fc83bc-4600-4533-992e-455686e95aec None admin] RBAC: Adding query filter params (interface=public, service_id=5c052208fb5643a7aabebfcc3a2c4115, region_id=RegionOne) {{(pid=81408) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 13 04:09:17.578876 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-06fc83bc-4600-4533-992e-455686e95aec None admin] RBAC: Authorizing `identity:list_endpoints()` {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:09:17.580287 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-06fc83bc-4600-4533-992e-455686e95aec None admin] RBAC: Authorization granted {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:09:17.588341 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 284/581] 10.222.0.27 () {56 vars in 1251 bytes} [Thu Oct 13 04:09:17 2022] GET /identity/v3/endpoints?service_id=5c052208fb5643a7aabebfcc3a2c4115&interface=public®ion_id=RegionOne => generated 194 bytes in 166 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 13 04:09:17.589221 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-06fc83bc-4600-4533-992e-455686e95aec None admin] Authenticating user token {{(pid=81408) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:09:17.607519 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 298/582] 10.222.0.27 () {58 vars in 1024 bytes} [Thu Oct 13 04:09:17 2022] POST /identity/v3/auth/tokens => generated 1002 bytes in 103 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Oct 13 04:09:17.608767 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9f6a940-3482-44c1-b3a9-a147347323ba None None] REQUEST_METHOD: `GET` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:09:17.609052 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9f6a940-3482-44c1-b3a9-a147347323ba None None] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:09:17.609319 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9f6a940-3482-44c1-b3a9-a147347323ba None None] PATH_INFO: `/` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:09:17.615035 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 299/583] 10.222.0.27 () {54 vars in 927 bytes} [Thu Oct 13 04:09:17 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 13 04:09:17.615035 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d9f6a940-3482-44c1-b3a9-a147347323ba None None] Authenticating user token {{(pid=81409) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:09:17.658093 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG dbcounter [-] [81408] Writing DB stats keystone:SELECT=100 {{(pid=81408) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:09:17.672715 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG dbcounter [-] [81409] Writing DB stats keystone:SELECT=100 {{(pid=81409) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:09:17.726125 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d9f6a940-3482-44c1-b3a9-a147347323ba None None] Validating token access rules against request {{(pid=81409) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:09:17.736495 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-06fc83bc-4600-4533-992e-455686e95aec None admin] Validating token access rules against request {{(pid=81408) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:09:17.765262 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-35ad7e38-b5f0-47d5-acb7-6d8220fd703e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81409) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:09:17.766175 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35ad7e38-b5f0-47d5-acb7-6d8220fd703e None admin] REQUEST_METHOD: `GET` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:09:17.766810 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35ad7e38-b5f0-47d5-acb7-6d8220fd703e None admin] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:09:17.767266 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35ad7e38-b5f0-47d5-acb7-6d8220fd703e None admin] PATH_INFO: `/v3/services` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:09:17.768383 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-35ad7e38-b5f0-47d5-acb7-6d8220fd703e None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:09:17.769959 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-35ad7e38-b5f0-47d5-acb7-6d8220fd703e None admin] RBAC: Authorization granted {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:09:17.775972 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 300/584] 10.222.0.27 () {56 vars in 1087 bytes} [Thu Oct 13 04:09:17 2022] GET /identity/v3/services => generated 772 bytes in 165 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 13 04:09:17.780292 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-deb95424-a308-4a64-87e8-f5954b1bd546 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81408) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:09:17.781330 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-deb95424-a308-4a64-87e8-f5954b1bd546 None admin] REQUEST_METHOD: `GET` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:09:17.781810 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-deb95424-a308-4a64-87e8-f5954b1bd546 None admin] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:09:17.782305 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-deb95424-a308-4a64-87e8-f5954b1bd546 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:09:17.783244 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-deb95424-a308-4a64-87e8-f5954b1bd546 None admin] RBAC: Adding query filter params (role.id=ef55fbd94e1040be83695754075a7ed5, scope.project.id=dcec0431fb6849499d6aec35a234daa6, user.id=cb740ed7f7684b079e45f887be83f795) {{(pid=81408) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 13 04:09:17.783763 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2fe6061-9258-4819-9a84-ac11d80d5bff None None] REQUEST_METHOD: `POST` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:09:17.783896 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-deb95424-a308-4a64-87e8-f5954b1bd546 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:09:17.784075 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2fe6061-9258-4819-9a84-ac11d80d5bff None None] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:09:17.784347 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2fe6061-9258-4819-9a84-ac11d80d5bff None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:09:17.785926 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-deb95424-a308-4a64-87e8-f5954b1bd546 None admin] RBAC: Authorization granted {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:09:17.794308 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 285/585] 10.222.0.27 () {56 vars in 1374 bytes} [Thu Oct 13 04:09:17 2022] GET /identity/v3/role_assignments?user.id=cb740ed7f7684b079e45f887be83f795&scope.project.id=dcec0431fb6849499d6aec35a234daa6&role.id=ef55fbd94e1040be83695754075a7ed5 => generated 608 bytes in 205 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 13 04:09:17.827541 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.auth.core [None req-c2fe6061-9258-4819-9a84-ac11d80d5bff None None] MFA Rules not processed for user `0e2d979563d6421995451a0e86bf3022`. Rule list: `[]` (Enabled: `True`). {{(pid=81409) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 13 04:09:17.828837 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.fernet_utils [None req-c2fe6061-9258-4819-9a84-ac11d80d5bff None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81409) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:09:17.890017 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-deb95424-a308-4a64-87e8-f5954b1bd546 None admin] Authenticating user token {{(pid=81408) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:09:17.933355 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 301/586] 10.222.0.27 () {58 vars in 1024 bytes} [Thu Oct 13 04:09:17 2022] POST /identity/v3/auth/tokens => generated 1002 bytes in 155 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Oct 13 04:09:18.060096 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-deb95424-a308-4a64-87e8-f5954b1bd546 None admin] Validating token access rules against request {{(pid=81408) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:09:18.074175 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG dbcounter [-] [81408] Writing DB stats keystone:SELECT=100 {{(pid=81408) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:09:18.104714 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7b67979e-813f-455a-856f-9522ed508004 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81408) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:09:18.105325 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b67979e-813f-455a-856f-9522ed508004 None admin] REQUEST_METHOD: `GET` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:09:18.105596 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b67979e-813f-455a-856f-9522ed508004 None admin] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:09:18.105862 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b67979e-813f-455a-856f-9522ed508004 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:09:18.116431 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7b67979e-813f-455a-856f-9522ed508004 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 13 04:09:18.122285 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application [None req-7b67979e-813f-455a-856f-9522ed508004 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Oct 13 04:09:18.122285 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 13 04:09:18.122285 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 13 04:09:18.122285 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 13 04:09:18.122285 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 13 04:09:18.122285 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 13 04:09:18.122285 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 13 04:09:18.122285 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 13 04:09:18.122285 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 13 04:09:18.122285 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 13 04:09:18.122285 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 13 04:09:18.122285 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 13 04:09:18.122285 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 13 04:09:18.122285 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 13 04:09:18.122285 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 13 04:09:18.122285 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 13 04:09:18.123244 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 13 04:09:18.123244 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 13 04:09:18.123244 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 13 04:09:18.123244 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 13 04:09:18.123244 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 13 04:09:18.123244 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 13 04:09:18.123244 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 13 04:09:18.123244 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 13 04:09:18.123244 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 13 04:09:18.123244 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application return self.get_or_create( Oct 13 04:09:18.123244 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 13 04:09:18.123244 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application with Lock( Oct 13 04:09:18.123244 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 13 04:09:18.123244 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application return self._enter() Oct 13 04:09:18.123244 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 13 04:09:18.123244 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 13 04:09:18.124200 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 13 04:09:18.124200 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application return self.creator() Oct 13 04:09:18.124200 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 13 04:09:18.124200 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application created_value = creator( Oct 13 04:09:18.124200 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 13 04:09:18.124200 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 13 04:09:18.124200 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 13 04:09:18.124200 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 13 04:09:18.124200 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 13 04:09:18.124200 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 13 04:09:18.124200 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Oct 13 04:09:18.124200 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application Oct 13 04:09:18.124200 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 286/587] 10.222.0.27 () {56 vars in 1096 bytes} [Thu Oct 13 04:09:17 2022] GET /identity/v3/roles/admin => generated 83 bytes in 234 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Oct 13 04:09:18.130185 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c2fe6061-9258-4819-9a84-ac11d80d5bff None None] Authenticating user token {{(pid=81409) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:09:18.187443 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG dbcounter [-] [81409] Writing DB stats keystone:SELECT=100 {{(pid=81409) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:09:18.246559 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7b67979e-813f-455a-856f-9522ed508004 None admin] Authenticating user token {{(pid=81408) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:09:18.249977 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c2fe6061-9258-4819-9a84-ac11d80d5bff None None] Validating token access rules against request {{(pid=81409) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:09:18.292684 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d4476856-4213-4b72-86f8-40e70592fa28 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81409) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:09:18.293716 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4476856-4213-4b72-86f8-40e70592fa28 None admin] REQUEST_METHOD: `GET` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:09:18.293991 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4476856-4213-4b72-86f8-40e70592fa28 None admin] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:09:18.294290 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4476856-4213-4b72-86f8-40e70592fa28 None admin] PATH_INFO: `/v3/roles` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:09:18.294958 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d4476856-4213-4b72-86f8-40e70592fa28 None admin] RBAC: Adding query filter params (name=admin) {{(pid=81409) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 13 04:09:18.295256 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d4476856-4213-4b72-86f8-40e70592fa28 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:09:18.296747 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d4476856-4213-4b72-86f8-40e70592fa28 None admin] RBAC: Authorization granted {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:09:18.317781 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 302/588] 10.222.0.27 () {56 vars in 1099 bytes} [Thu Oct 13 04:09:18 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 188 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 13 04:09:18.324479 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d4476856-4213-4b72-86f8-40e70592fa28 None admin] Authenticating user token {{(pid=81409) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:09:18.410506 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7b67979e-813f-455a-856f-9522ed508004 None admin] Validating token access rules against request {{(pid=81408) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:09:18.484675 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d4476856-4213-4b72-86f8-40e70592fa28 None admin] Validating token access rules against request {{(pid=81409) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:09:18.497518 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG dbcounter [-] [81409] Writing DB stats keystone:SELECT=100 {{(pid=81409) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:09:18.503500 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3e91df7d-88ef-4f37-a754-720c7bfe3a0d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81408) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:09:18.504206 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e91df7d-88ef-4f37-a754-720c7bfe3a0d None admin] REQUEST_METHOD: `GET` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:09:18.504486 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e91df7d-88ef-4f37-a754-720c7bfe3a0d None admin] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:09:18.504759 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e91df7d-88ef-4f37-a754-720c7bfe3a0d None admin] PATH_INFO: `/v3/services/network` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:09:18.505350 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3e91df7d-88ef-4f37-a754-720c7bfe3a0d None admin] RBAC: Authorizing `identity:get_service(service_id=network)` {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:09:18.506775 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3e91df7d-88ef-4f37-a754-720c7bfe3a0d None admin] RBAC: Authorization granted {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:09:18.524790 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application [None req-3e91df7d-88ef-4f37-a754-720c7bfe3a0d None admin] Could not find service: network.: keystone.exception.ServiceNotFound: Could not find service: network. Oct 13 04:09:18.524790 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 13 04:09:18.524790 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Oct 13 04:09:18.524790 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Oct 13 04:09:18.524790 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Oct 13 04:09:18.524790 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Oct 13 04:09:18.524790 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Oct 13 04:09:18.524790 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 13 04:09:18.524790 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: network. Oct 13 04:09:18.524790 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application Oct 13 04:09:18.524790 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Oct 13 04:09:18.524790 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application Oct 13 04:09:18.524790 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 13 04:09:18.524790 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 13 04:09:18.524790 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 13 04:09:18.524790 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 13 04:09:18.524790 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 13 04:09:18.525868 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 13 04:09:18.525868 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 13 04:09:18.525868 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 13 04:09:18.525868 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 13 04:09:18.525868 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 13 04:09:18.525868 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 13 04:09:18.525868 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Oct 13 04:09:18.525868 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application return self._get_service(service_id) Oct 13 04:09:18.525868 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Oct 13 04:09:18.525868 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Oct 13 04:09:18.525868 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 13 04:09:18.525868 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 13 04:09:18.525868 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 13 04:09:18.525868 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 13 04:09:18.525868 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 13 04:09:18.525868 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application return self.get_or_create( Oct 13 04:09:18.526758 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 13 04:09:18.526758 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application with Lock( Oct 13 04:09:18.526758 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 13 04:09:18.526758 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application return self._enter() Oct 13 04:09:18.526758 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 13 04:09:18.526758 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 13 04:09:18.526758 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 13 04:09:18.526758 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application return self.creator() Oct 13 04:09:18.526758 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 13 04:09:18.526758 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application created_value = creator( Oct 13 04:09:18.526758 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Oct 13 04:09:18.526758 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 13 04:09:18.526758 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: network. Oct 13 04:09:18.526758 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application Oct 13 04:09:18.531641 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ac80605f-76cf-4721-91d1-76060d9662a9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81409) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:09:18.532527 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac80605f-76cf-4721-91d1-76060d9662a9 None admin] REQUEST_METHOD: `GET` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:09:18.533023 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac80605f-76cf-4721-91d1-76060d9662a9 None admin] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:09:18.533496 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac80605f-76cf-4721-91d1-76060d9662a9 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:09:18.534864 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 287/589] 10.222.0.27 () {56 vars in 1111 bytes} [Thu Oct 13 04:09:18 2022] GET /identity/v3/services/network => generated 88 bytes in 289 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Oct 13 04:09:18.541236 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3e91df7d-88ef-4f37-a754-720c7bfe3a0d None admin] Authenticating user token {{(pid=81408) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:09:18.558172 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ac80605f-76cf-4721-91d1-76060d9662a9 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:09:18.560074 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ac80605f-76cf-4721-91d1-76060d9662a9 None admin] RBAC: Authorization granted {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:09:18.587857 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application [None req-ac80605f-76cf-4721-91d1-76060d9662a9 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 13 04:09:18.587857 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 13 04:09:18.587857 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 13 04:09:18.587857 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 13 04:09:18.587857 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 13 04:09:18.587857 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 13 04:09:18.587857 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 13 04:09:18.587857 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 13 04:09:18.587857 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 13 04:09:18.587857 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 13 04:09:18.587857 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 13 04:09:18.587857 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 13 04:09:18.587857 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 13 04:09:18.587857 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 13 04:09:18.587857 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 13 04:09:18.587857 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 13 04:09:18.588858 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 13 04:09:18.588858 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 13 04:09:18.588858 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 13 04:09:18.588858 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 13 04:09:18.588858 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 13 04:09:18.588858 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application return self.get_or_create( Oct 13 04:09:18.588858 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 13 04:09:18.588858 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application with Lock( Oct 13 04:09:18.588858 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 13 04:09:18.588858 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application return self._enter() Oct 13 04:09:18.588858 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 13 04:09:18.588858 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 13 04:09:18.588858 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 13 04:09:18.588858 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application return self.creator() Oct 13 04:09:18.588858 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 13 04:09:18.588858 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application created_value = creator( Oct 13 04:09:18.588858 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 13 04:09:18.589765 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 13 04:09:18.589765 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 13 04:09:18.589765 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application Oct 13 04:09:18.592014 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 303/590] 10.222.0.27 () {56 vars in 1108 bytes} [Thu Oct 13 04:09:18 2022] GET /identity/v3/domains/Default => generated 87 bytes in 268 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 13 04:09:18.597193 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ac80605f-76cf-4721-91d1-76060d9662a9 None admin] Authenticating user token {{(pid=81409) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:09:18.624157 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG dbcounter [-] [81408] Writing DB stats keystone:SELECT=100 {{(pid=81408) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:09:18.720896 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ac80605f-76cf-4721-91d1-76060d9662a9 None admin] Validating token access rules against request {{(pid=81409) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:09:18.762505 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-40475c77-7f7f-4706-b24d-c27bf545b0c2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81409) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:09:18.763288 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40475c77-7f7f-4706-b24d-c27bf545b0c2 None admin] REQUEST_METHOD: `GET` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:09:18.763566 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40475c77-7f7f-4706-b24d-c27bf545b0c2 None admin] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:09:18.763831 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40475c77-7f7f-4706-b24d-c27bf545b0c2 None admin] PATH_INFO: `/v3/domains` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:09:18.764636 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-40475c77-7f7f-4706-b24d-c27bf545b0c2 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81409) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 13 04:09:18.764984 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-40475c77-7f7f-4706-b24d-c27bf545b0c2 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:09:18.766460 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-40475c77-7f7f-4706-b24d-c27bf545b0c2 None admin] RBAC: Authorization granted {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:09:18.780663 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 304/591] 10.222.0.27 () {56 vars in 1109 bytes} [Thu Oct 13 04:09:18 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 184 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 13 04:09:18.786556 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-40475c77-7f7f-4706-b24d-c27bf545b0c2 None admin] Authenticating user token {{(pid=81409) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:09:18.805097 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3e91df7d-88ef-4f37-a754-720c7bfe3a0d None admin] Validating token access rules against request {{(pid=81408) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:09:18.812165 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG dbcounter [-] [81409] Writing DB stats keystone:SELECT=100 {{(pid=81409) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:09:18.896855 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-40475c77-7f7f-4706-b24d-c27bf545b0c2 None admin] Validating token access rules against request {{(pid=81409) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:09:18.919556 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cf4e9065-f6a3-4068-adc2-16222f8e9b7d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81408) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:09:18.920174 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf4e9065-f6a3-4068-adc2-16222f8e9b7d None admin] REQUEST_METHOD: `GET` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:09:18.920447 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf4e9065-f6a3-4068-adc2-16222f8e9b7d None admin] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:09:18.920724 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf4e9065-f6a3-4068-adc2-16222f8e9b7d None admin] PATH_INFO: `/v3/services` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:09:18.921357 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cf4e9065-f6a3-4068-adc2-16222f8e9b7d None admin] RBAC: Adding query filter params (name=network) {{(pid=81408) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 13 04:09:18.921660 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cf4e9065-f6a3-4068-adc2-16222f8e9b7d None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:09:18.923108 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cf4e9065-f6a3-4068-adc2-16222f8e9b7d None admin] RBAC: Authorization granted {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:09:18.935909 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c2da0f50-55e9-4939-84f8-d17dcb65b0f9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81409) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:09:18.936569 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2da0f50-55e9-4939-84f8-d17dcb65b0f9 None admin] REQUEST_METHOD: `GET` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:09:18.936843 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2da0f50-55e9-4939-84f8-d17dcb65b0f9 None admin] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:09:18.937109 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2da0f50-55e9-4939-84f8-d17dcb65b0f9 None admin] PATH_INFO: `/v3/users/swift` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:09:18.940644 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 288/592] 10.222.0.27 () {56 vars in 1112 bytes} [Thu Oct 13 04:09:18 2022] GET /identity/v3/services?name=network => generated 124 bytes in 401 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 13 04:09:18.946115 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c2da0f50-55e9-4939-84f8-d17dcb65b0f9 None admin] RBAC: Authorizing `identity:get_user(user_id=swift)` {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:09:18.947713 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c2da0f50-55e9-4939-84f8-d17dcb65b0f9 None admin] RBAC: Authorization granted {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:09:18.951323 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cf4e9065-f6a3-4068-adc2-16222f8e9b7d None admin] Authenticating user token {{(pid=81408) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:09:18.962295 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application [None req-c2da0f50-55e9-4939-84f8-d17dcb65b0f9 None admin] Could not find user: swift.: keystone.exception.UserNotFound: Could not find user: swift. Oct 13 04:09:18.962295 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 13 04:09:18.962295 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 13 04:09:18.962295 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 13 04:09:18.962295 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 13 04:09:18.962295 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 13 04:09:18.962295 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 13 04:09:18.962295 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 13 04:09:18.962295 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 13 04:09:18.962295 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 13 04:09:18.962295 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 13 04:09:18.962295 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 13 04:09:18.962295 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Oct 13 04:09:18.962295 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application return self._get_user(user_id) Oct 13 04:09:18.962295 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Oct 13 04:09:18.962295 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Oct 13 04:09:18.963304 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 13 04:09:18.963304 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 13 04:09:18.963304 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Oct 13 04:09:18.963304 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 13 04:09:18.963304 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Oct 13 04:09:18.963304 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 13 04:09:18.963304 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 13 04:09:18.963304 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 13 04:09:18.963304 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 13 04:09:18.963304 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application return self.get_or_create( Oct 13 04:09:18.963304 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 13 04:09:18.963304 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application with Lock( Oct 13 04:09:18.963304 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 13 04:09:18.963304 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application return self._enter() Oct 13 04:09:18.963304 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 13 04:09:18.963304 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 13 04:09:18.963304 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 13 04:09:18.963304 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application return self.creator() Oct 13 04:09:18.964716 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 13 04:09:18.964716 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application created_value = creator( Oct 13 04:09:18.964716 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Oct 13 04:09:18.964716 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Oct 13 04:09:18.964716 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Oct 13 04:09:18.964716 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Oct 13 04:09:18.964716 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Oct 13 04:09:18.964716 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Oct 13 04:09:18.964716 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: swift. Oct 13 04:09:18.964716 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application Oct 13 04:09:18.964716 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 305/593] 10.222.0.27 () {56 vars in 1096 bytes} [Thu Oct 13 04:09:18 2022] GET /identity/v3/users/swift => generated 83 bytes in 178 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 13 04:09:18.969741 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c2da0f50-55e9-4939-84f8-d17dcb65b0f9 None admin] Authenticating user token {{(pid=81409) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:09:19.061600 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG dbcounter [-] [81409] Writing DB stats keystone:SELECT=100 {{(pid=81409) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:09:19.079205 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c2da0f50-55e9-4939-84f8-d17dcb65b0f9 None admin] Validating token access rules against request {{(pid=81409) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:09:19.120091 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3ebd2955-5617-421c-97ae-7a773c359e36 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81409) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:09:19.120810 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ebd2955-5617-421c-97ae-7a773c359e36 None admin] REQUEST_METHOD: `GET` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:09:19.121085 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ebd2955-5617-421c-97ae-7a773c359e36 None admin] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:09:19.121354 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ebd2955-5617-421c-97ae-7a773c359e36 None admin] PATH_INFO: `/v3/users` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:09:19.122078 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3ebd2955-5617-421c-97ae-7a773c359e36 None admin] RBAC: Adding query filter params (domain_id=default, name=swift) {{(pid=81409) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 13 04:09:19.122425 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3ebd2955-5617-421c-97ae-7a773c359e36 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:09:19.123934 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3ebd2955-5617-421c-97ae-7a773c359e36 None admin] RBAC: Authorization granted {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:09:19.142923 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 306/594] 10.222.0.27 () {56 vars in 1135 bytes} [Thu Oct 13 04:09:18 2022] GET /identity/v3/users?domain_id=default&name=swift => generated 370 bytes in 174 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 13 04:09:19.147710 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG dbcounter [-] [81408] Writing DB stats keystone:SELECT=100 {{(pid=81408) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:09:19.148859 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7426966-78cd-4328-9524-e7401622709e None None] REQUEST_METHOD: `GET` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:09:19.149846 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7426966-78cd-4328-9524-e7401622709e None None] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:09:19.150173 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7426966-78cd-4328-9524-e7401622709e None None] PATH_INFO: `/` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:09:19.151452 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 307/595] 10.222.0.27 () {54 vars in 927 bytes} [Thu Oct 13 04:09:19 2022] GET /identity => generated 270 bytes in 8 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 13 04:09:19.152084 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f7426966-78cd-4328-9524-e7401622709e None None] Authenticating user token {{(pid=81409) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:09:19.156402 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cf4e9065-f6a3-4068-adc2-16222f8e9b7d None admin] Validating token access rules against request {{(pid=81408) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:09:19.190394 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ac4e0de8-2223-46ab-9825-03dade46c484 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81408) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:09:19.190996 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac4e0de8-2223-46ab-9825-03dade46c484 None admin] REQUEST_METHOD: `GET` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:09:19.191269 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac4e0de8-2223-46ab-9825-03dade46c484 None admin] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:09:19.191542 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac4e0de8-2223-46ab-9825-03dade46c484 None admin] PATH_INFO: `/v3/services` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:09:19.192252 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ac4e0de8-2223-46ab-9825-03dade46c484 None admin] RBAC: Adding query filter params (type=network) {{(pid=81408) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 13 04:09:19.192456 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ac4e0de8-2223-46ab-9825-03dade46c484 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:09:19.193821 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ac4e0de8-2223-46ab-9825-03dade46c484 None admin] RBAC: Authorization granted {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:09:19.199175 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 289/596] 10.222.0.27 () {56 vars in 1112 bytes} [Thu Oct 13 04:09:18 2022] GET /identity/v3/services?type=network => generated 350 bytes in 248 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 13 04:09:19.200338 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4add7d44-acc6-4257-8abd-b1cda9ca1553 None None] REQUEST_METHOD: `POST` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:09:19.200619 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4add7d44-acc6-4257-8abd-b1cda9ca1553 None None] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:09:19.200906 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4add7d44-acc6-4257-8abd-b1cda9ca1553 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:09:19.218922 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.auth.core [None req-4add7d44-acc6-4257-8abd-b1cda9ca1553 None None] MFA Rules not processed for user `0e2d979563d6421995451a0e86bf3022`. Rule list: `[]` (Enabled: `True`). {{(pid=81408) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 13 04:09:19.220137 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.fernet_utils [None req-4add7d44-acc6-4257-8abd-b1cda9ca1553 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81408) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:09:19.271395 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f7426966-78cd-4328-9524-e7401622709e None None] Validating token access rules against request {{(pid=81409) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:09:19.298232 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 290/597] 10.222.0.27 () {58 vars in 1024 bytes} [Thu Oct 13 04:09:19 2022] POST /identity/v3/auth/tokens => generated 1002 bytes in 92 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Oct 13 04:09:19.298232 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4add7d44-acc6-4257-8abd-b1cda9ca1553 None None] Authenticating user token {{(pid=81408) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:09:19.332688 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-182afb7a-6265-4725-889c-9decdbdc0921 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81409) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:09:19.333733 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-182afb7a-6265-4725-889c-9decdbdc0921 None admin] REQUEST_METHOD: `GET` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:09:19.334289 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-182afb7a-6265-4725-889c-9decdbdc0921 None admin] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:09:19.334772 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-182afb7a-6265-4725-889c-9decdbdc0921 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:09:19.347170 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG dbcounter [-] [81409] Writing DB stats keystone:SELECT=100 {{(pid=81409) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:09:19.386626 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG dbcounter [-] [81408] Writing DB stats keystone:SELECT=100 {{(pid=81408) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:09:19.398630 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4add7d44-acc6-4257-8abd-b1cda9ca1553 None None] Validating token access rules against request {{(pid=81408) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:09:19.437401 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-68c62ee9-299d-4f83-962a-73576c07f7e0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81408) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:09:19.438378 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68c62ee9-299d-4f83-962a-73576c07f7e0 None admin] REQUEST_METHOD: `GET` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:09:19.438815 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68c62ee9-299d-4f83-962a-73576c07f7e0 None admin] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:09:19.439277 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68c62ee9-299d-4f83-962a-73576c07f7e0 None admin] PATH_INFO: `/v3/endpoints` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:09:19.440158 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-68c62ee9-299d-4f83-962a-73576c07f7e0 None admin] RBAC: Adding query filter params (interface=public, service_id=fbd8a3e93e73499788c96339cf261edf, region_id=RegionOne) {{(pid=81408) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 13 04:09:19.440649 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-68c62ee9-299d-4f83-962a-73576c07f7e0 None admin] RBAC: Authorizing `identity:list_endpoints()` {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:09:19.442452 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-68c62ee9-299d-4f83-962a-73576c07f7e0 None admin] RBAC: Authorization granted {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:09:19.444660 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-182afb7a-6265-4725-889c-9decdbdc0921 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:09:19.447045 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-182afb7a-6265-4725-889c-9decdbdc0921 None admin] RBAC: Authorization granted {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:09:19.449256 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 291/598] 10.222.0.27 () {56 vars in 1251 bytes} [Thu Oct 13 04:09:19 2022] GET /identity/v3/endpoints?service_id=fbd8a3e93e73499788c96339cf261edf&interface=public®ion_id=RegionOne => generated 194 bytes in 157 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 13 04:09:19.455164 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-68c62ee9-299d-4f83-962a-73576c07f7e0 None admin] Authenticating user token {{(pid=81408) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:09:19.554791 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 308/599] 10.222.0.27 () {58 vars in 1282 bytes} [Thu Oct 13 04:09:19 2022] GET /identity/v3/auth/tokens => generated 1002 bytes in 403 msecs (HTTP/1.1 200) 6 headers in 359 bytes (1 switches on core 0) Oct 13 04:09:19.560885 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-182afb7a-6265-4725-889c-9decdbdc0921 None admin] Authenticating user token {{(pid=81409) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:09:19.606721 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG dbcounter [-] [81409] Writing DB stats keystone:SELECT=100 {{(pid=81409) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:09:19.640972 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-68c62ee9-299d-4f83-962a-73576c07f7e0 None admin] Validating token access rules against request {{(pid=81408) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:09:19.669350 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-182afb7a-6265-4725-889c-9decdbdc0921 None admin] Validating token access rules against request {{(pid=81409) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:09:19.689950 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d344b12b-bec1-4b53-aefb-4ba787e87acb None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81408) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:09:19.690588 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d344b12b-bec1-4b53-aefb-4ba787e87acb None admin] REQUEST_METHOD: `GET` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:09:19.690867 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d344b12b-bec1-4b53-aefb-4ba787e87acb None admin] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:09:19.691136 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d344b12b-bec1-4b53-aefb-4ba787e87acb None admin] PATH_INFO: `/v3/services` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:09:19.691738 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d344b12b-bec1-4b53-aefb-4ba787e87acb None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:09:19.693196 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d344b12b-bec1-4b53-aefb-4ba787e87acb None admin] RBAC: Authorization granted {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:09:19.698895 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 292/600] 10.222.0.27 () {56 vars in 1087 bytes} [Thu Oct 13 04:09:19 2022] GET /identity/v3/services => generated 772 bytes in 244 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 13 04:09:19.699948 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d344b12b-bec1-4b53-aefb-4ba787e87acb None admin] Authenticating user token {{(pid=81408) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:09:19.707934 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8882acfd-4734-4d0e-bcb0-6f3272ec7d94 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81409) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:09:19.709714 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8882acfd-4734-4d0e-bcb0-6f3272ec7d94 None admin] REQUEST_METHOD: `GET` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:09:19.710251 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8882acfd-4734-4d0e-bcb0-6f3272ec7d94 None admin] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:09:19.710742 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8882acfd-4734-4d0e-bcb0-6f3272ec7d94 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:09:19.720489 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG dbcounter [-] [81408] Writing DB stats keystone:SELECT=100 {{(pid=81408) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:09:19.730337 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8882acfd-4734-4d0e-bcb0-6f3272ec7d94 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:09:19.731855 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8882acfd-4734-4d0e-bcb0-6f3272ec7d94 None admin] RBAC: Authorization granted {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:09:19.766860 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application [None req-8882acfd-4734-4d0e-bcb0-6f3272ec7d94 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 13 04:09:19.766860 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 13 04:09:19.766860 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 13 04:09:19.766860 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 13 04:09:19.766860 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 13 04:09:19.766860 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 13 04:09:19.766860 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 13 04:09:19.766860 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 13 04:09:19.766860 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 13 04:09:19.766860 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 13 04:09:19.766860 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 13 04:09:19.766860 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 13 04:09:19.766860 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 13 04:09:19.766860 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 13 04:09:19.766860 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 13 04:09:19.766860 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 13 04:09:19.767948 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 13 04:09:19.767948 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 13 04:09:19.767948 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 13 04:09:19.767948 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 13 04:09:19.767948 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 13 04:09:19.767948 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application return self.get_or_create( Oct 13 04:09:19.767948 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 13 04:09:19.767948 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application with Lock( Oct 13 04:09:19.767948 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 13 04:09:19.767948 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application return self._enter() Oct 13 04:09:19.767948 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 13 04:09:19.767948 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 13 04:09:19.767948 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 13 04:09:19.767948 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application return self.creator() Oct 13 04:09:19.767948 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 13 04:09:19.767948 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application created_value = creator( Oct 13 04:09:19.767948 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 13 04:09:19.768906 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 13 04:09:19.768906 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 13 04:09:19.768906 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application Oct 13 04:09:19.770623 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 309/601] 10.222.0.27 () {56 vars in 1108 bytes} [Thu Oct 13 04:09:19 2022] GET /identity/v3/domains/Default => generated 87 bytes in 210 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 13 04:09:19.777474 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8882acfd-4734-4d0e-bcb0-6f3272ec7d94 None admin] Authenticating user token {{(pid=81409) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:09:19.894493 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d344b12b-bec1-4b53-aefb-4ba787e87acb None admin] Validating token access rules against request {{(pid=81408) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:09:19.933488 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG dbcounter [-] [81409] Writing DB stats keystone:SELECT=100 {{(pid=81409) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:09:19.947692 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8882acfd-4734-4d0e-bcb0-6f3272ec7d94 None admin] Validating token access rules against request {{(pid=81409) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:09:19.948524 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d5682703-4d6f-47c3-9050-838ded1d9ee0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81408) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:09:19.949164 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5682703-4d6f-47c3-9050-838ded1d9ee0 None admin] REQUEST_METHOD: `GET` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:09:19.949443 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5682703-4d6f-47c3-9050-838ded1d9ee0 None admin] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:09:19.949714 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5682703-4d6f-47c3-9050-838ded1d9ee0 None admin] PATH_INFO: `/v3/roles/service` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:09:19.961241 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d5682703-4d6f-47c3-9050-838ded1d9ee0 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 13 04:09:19.966905 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application [None req-d5682703-4d6f-47c3-9050-838ded1d9ee0 None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Oct 13 04:09:19.966905 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 13 04:09:19.966905 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 13 04:09:19.966905 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 13 04:09:19.966905 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 13 04:09:19.966905 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 13 04:09:19.966905 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 13 04:09:19.966905 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 13 04:09:19.966905 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 13 04:09:19.966905 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 13 04:09:19.966905 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 13 04:09:19.966905 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 13 04:09:19.966905 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 13 04:09:19.966905 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 13 04:09:19.966905 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 13 04:09:19.966905 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 13 04:09:19.967835 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 13 04:09:19.967835 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 13 04:09:19.967835 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 13 04:09:19.967835 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 13 04:09:19.967835 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 13 04:09:19.967835 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 13 04:09:19.967835 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 13 04:09:19.967835 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 13 04:09:19.967835 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 13 04:09:19.967835 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application return self.get_or_create( Oct 13 04:09:19.967835 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 13 04:09:19.967835 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application with Lock( Oct 13 04:09:19.967835 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 13 04:09:19.967835 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application return self._enter() Oct 13 04:09:19.967835 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 13 04:09:19.967835 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 13 04:09:19.968657 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 13 04:09:19.968657 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application return self.creator() Oct 13 04:09:19.968657 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 13 04:09:19.968657 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application created_value = creator( Oct 13 04:09:19.968657 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 13 04:09:19.968657 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 13 04:09:19.968657 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 13 04:09:19.968657 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 13 04:09:19.968657 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 13 04:09:19.968657 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 13 04:09:19.968657 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Oct 13 04:09:19.968657 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application Oct 13 04:09:19.968657 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 293/602] 10.222.0.27 () {56 vars in 1102 bytes} [Thu Oct 13 04:09:19 2022] GET /identity/v3/roles/service => generated 85 bytes in 269 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 13 04:09:19.973920 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d5682703-4d6f-47c3-9050-838ded1d9ee0 None admin] Authenticating user token {{(pid=81408) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:09:19.997445 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c0dcffe8-8af0-4eb8-bcb8-f144ff307a1d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81409) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:09:19.998454 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0dcffe8-8af0-4eb8-bcb8-f144ff307a1d None admin] REQUEST_METHOD: `GET` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:09:19.998928 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0dcffe8-8af0-4eb8-bcb8-f144ff307a1d None admin] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:09:19.999946 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0dcffe8-8af0-4eb8-bcb8-f144ff307a1d None admin] PATH_INFO: `/v3/domains` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:09:20.000864 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c0dcffe8-8af0-4eb8-bcb8-f144ff307a1d None admin] RBAC: Adding query filter params (name=Default) {{(pid=81409) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 13 04:09:20.001782 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c0dcffe8-8af0-4eb8-bcb8-f144ff307a1d None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:09:20.003984 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c0dcffe8-8af0-4eb8-bcb8-f144ff307a1d None admin] RBAC: Authorization granted {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:09:20.020922 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 310/603] 10.222.0.27 () {56 vars in 1109 bytes} [Thu Oct 13 04:09:19 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 244 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 13 04:09:20.027669 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c0dcffe8-8af0-4eb8-bcb8-f144ff307a1d None admin] Authenticating user token {{(pid=81409) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:09:20.069193 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG dbcounter [-] [81408] Writing DB stats keystone:SELECT=100 {{(pid=81408) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:09:20.097143 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d5682703-4d6f-47c3-9050-838ded1d9ee0 None admin] Validating token access rules against request {{(pid=81408) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:09:20.134730 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8df63538-e81a-4856-9329-7b0859aa7e47 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81408) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:09:20.135640 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8df63538-e81a-4856-9329-7b0859aa7e47 None admin] REQUEST_METHOD: `GET` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:09:20.136113 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8df63538-e81a-4856-9329-7b0859aa7e47 None admin] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:09:20.136557 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8df63538-e81a-4856-9329-7b0859aa7e47 None admin] PATH_INFO: `/v3/roles` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:09:20.137403 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8df63538-e81a-4856-9329-7b0859aa7e47 None admin] RBAC: Adding query filter params (name=service) {{(pid=81408) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 13 04:09:20.137869 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8df63538-e81a-4856-9329-7b0859aa7e47 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:09:20.139437 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8df63538-e81a-4856-9329-7b0859aa7e47 None admin] RBAC: Authorization granted {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:09:20.148791 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 294/604] 10.222.0.27 () {56 vars in 1103 bytes} [Thu Oct 13 04:09:19 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 175 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 13 04:09:20.160673 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8df63538-e81a-4856-9329-7b0859aa7e47 None admin] Authenticating user token {{(pid=81408) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:09:20.253027 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c0dcffe8-8af0-4eb8-bcb8-f144ff307a1d None admin] Validating token access rules against request {{(pid=81409) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:09:20.307718 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8df63538-e81a-4856-9329-7b0859aa7e47 None admin] Validating token access rules against request {{(pid=81408) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:09:20.350923 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-718227fc-9f1c-4421-934c-321183274ee7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81408) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:09:20.351564 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-718227fc-9f1c-4421-934c-321183274ee7 None admin] REQUEST_METHOD: `GET` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:09:20.351843 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-718227fc-9f1c-4421-934c-321183274ee7 None admin] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:09:20.352640 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-718227fc-9f1c-4421-934c-321183274ee7 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:09:20.355025 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-833e1e75-b6d7-4513-a68d-bc5887877757 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81409) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:09:20.355658 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-833e1e75-b6d7-4513-a68d-bc5887877757 None admin] REQUEST_METHOD: `GET` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:09:20.355930 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-833e1e75-b6d7-4513-a68d-bc5887877757 None admin] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:09:20.356197 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-833e1e75-b6d7-4513-a68d-bc5887877757 None admin] PATH_INFO: `/v3/projects/service` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:09:20.369437 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG dbcounter [-] [81408] Writing DB stats keystone:SELECT=100 {{(pid=81408) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:09:20.369730 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-833e1e75-b6d7-4513-a68d-bc5887877757 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:09:20.371233 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-833e1e75-b6d7-4513-a68d-bc5887877757 None admin] RBAC: Authorization granted {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:09:20.377899 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-718227fc-9f1c-4421-934c-321183274ee7 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:09:20.379476 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-718227fc-9f1c-4421-934c-321183274ee7 None admin] RBAC: Authorization granted {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:09:20.392859 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application [None req-833e1e75-b6d7-4513-a68d-bc5887877757 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Oct 13 04:09:20.392859 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 13 04:09:20.392859 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 13 04:09:20.392859 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 13 04:09:20.392859 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 13 04:09:20.392859 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 13 04:09:20.392859 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 13 04:09:20.392859 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 13 04:09:20.392859 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 13 04:09:20.392859 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 13 04:09:20.392859 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 13 04:09:20.392859 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 13 04:09:20.392859 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Oct 13 04:09:20.392859 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application return self._get_project(project_id) Oct 13 04:09:20.392859 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Oct 13 04:09:20.393847 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Oct 13 04:09:20.393847 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 13 04:09:20.393847 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 13 04:09:20.393847 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 13 04:09:20.393847 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 13 04:09:20.393847 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 13 04:09:20.393847 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application return self.get_or_create( Oct 13 04:09:20.393847 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 13 04:09:20.393847 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application with Lock( Oct 13 04:09:20.393847 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 13 04:09:20.393847 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application return self._enter() Oct 13 04:09:20.393847 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 13 04:09:20.393847 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 13 04:09:20.393847 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 13 04:09:20.393847 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application return self.creator() Oct 13 04:09:20.393847 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 13 04:09:20.393847 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application created_value = creator( Oct 13 04:09:20.394802 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Oct 13 04:09:20.394802 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Oct 13 04:09:20.394802 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Oct 13 04:09:20.394802 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Oct 13 04:09:20.394802 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Oct 13 04:09:20.394802 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Oct 13 04:09:20.394802 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Oct 13 04:09:20.394802 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application Oct 13 04:09:20.399812 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 311/605] 10.222.0.27 () {56 vars in 1111 bytes} [Thu Oct 13 04:09:20 2022] GET /identity/v3/projects/service => generated 88 bytes in 372 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 13 04:09:20.403095 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-833e1e75-b6d7-4513-a68d-bc5887877757 None admin] Authenticating user token {{(pid=81409) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:09:20.413305 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application [None req-718227fc-9f1c-4421-934c-321183274ee7 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 13 04:09:20.413305 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 13 04:09:20.413305 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 13 04:09:20.413305 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 13 04:09:20.413305 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 13 04:09:20.413305 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 13 04:09:20.413305 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 13 04:09:20.413305 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 13 04:09:20.413305 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 13 04:09:20.413305 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 13 04:09:20.413305 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 13 04:09:20.413305 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 13 04:09:20.413305 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 13 04:09:20.413305 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 13 04:09:20.413305 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 13 04:09:20.413305 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 13 04:09:20.414763 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 13 04:09:20.414763 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 13 04:09:20.414763 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 13 04:09:20.414763 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 13 04:09:20.414763 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 13 04:09:20.414763 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application return self.get_or_create( Oct 13 04:09:20.414763 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 13 04:09:20.414763 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application with Lock( Oct 13 04:09:20.414763 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 13 04:09:20.414763 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application return self._enter() Oct 13 04:09:20.414763 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 13 04:09:20.414763 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 13 04:09:20.414763 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 13 04:09:20.414763 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application return self.creator() Oct 13 04:09:20.414763 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 13 04:09:20.414763 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application created_value = creator( Oct 13 04:09:20.414763 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 13 04:09:20.416013 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 13 04:09:20.416013 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 13 04:09:20.416013 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application Oct 13 04:09:20.416013 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 295/606] 10.222.0.27 () {56 vars in 1108 bytes} [Thu Oct 13 04:09:20 2022] GET /identity/v3/domains/Default => generated 87 bytes in 256 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 13 04:09:20.420828 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-718227fc-9f1c-4421-934c-321183274ee7 None admin] Authenticating user token {{(pid=81408) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:09:20.428558 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG dbcounter [-] [81409] Writing DB stats keystone:SELECT=100 {{(pid=81409) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:09:20.556851 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-718227fc-9f1c-4421-934c-321183274ee7 None admin] Validating token access rules against request {{(pid=81408) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:09:20.570348 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-833e1e75-b6d7-4513-a68d-bc5887877757 None admin] Validating token access rules against request {{(pid=81409) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:09:20.608820 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ed98a7d6-f164-41f2-87a0-22b1f6c27751 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81408) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:09:20.609625 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9dfe3c76-9f60-4d0b-b000-4d40c2719d0c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81409) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:09:20.610018 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed98a7d6-f164-41f2-87a0-22b1f6c27751 None admin] REQUEST_METHOD: `GET` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:09:20.610246 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9dfe3c76-9f60-4d0b-b000-4d40c2719d0c None admin] REQUEST_METHOD: `GET` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:09:20.610694 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9dfe3c76-9f60-4d0b-b000-4d40c2719d0c None admin] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:09:20.610937 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed98a7d6-f164-41f2-87a0-22b1f6c27751 None admin] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:09:20.611044 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9dfe3c76-9f60-4d0b-b000-4d40c2719d0c None admin] PATH_INFO: `/v3/projects` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:09:20.611541 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed98a7d6-f164-41f2-87a0-22b1f6c27751 None admin] PATH_INFO: `/v3/domains` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:09:20.611650 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9dfe3c76-9f60-4d0b-b000-4d40c2719d0c None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81409) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 13 04:09:20.611931 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9dfe3c76-9f60-4d0b-b000-4d40c2719d0c None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:09:20.612679 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ed98a7d6-f164-41f2-87a0-22b1f6c27751 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81408) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 13 04:09:20.613175 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ed98a7d6-f164-41f2-87a0-22b1f6c27751 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:09:20.613362 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9dfe3c76-9f60-4d0b-b000-4d40c2719d0c None admin] RBAC: Authorization granted {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:09:20.615244 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ed98a7d6-f164-41f2-87a0-22b1f6c27751 None admin] RBAC: Authorization granted {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:09:20.634062 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 312/607] 10.222.0.27 () {56 vars in 1148 bytes} [Thu Oct 13 04:09:20 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 226 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 13 04:09:20.634062 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49d8cbb4-eaa0-4a5d-95df-97f8e263e0b8 None None] REQUEST_METHOD: `GET` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:09:20.634062 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49d8cbb4-eaa0-4a5d-95df-97f8e263e0b8 None None] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:09:20.634062 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49d8cbb4-eaa0-4a5d-95df-97f8e263e0b8 None None] PATH_INFO: `/` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:09:20.634062 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 313/608] 10.222.0.27 () {54 vars in 927 bytes} [Thu Oct 13 04:09:20 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 13 04:09:20.635025 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-49d8cbb4-eaa0-4a5d-95df-97f8e263e0b8 None None] Authenticating user token {{(pid=81409) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:09:20.636533 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 296/609] 10.222.0.27 () {56 vars in 1109 bytes} [Thu Oct 13 04:09:20 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 216 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 13 04:09:20.642895 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-342789c0-22f4-44a3-aded-9c56f2fb3640 None None] REQUEST_METHOD: `POST` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:09:20.643414 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-342789c0-22f4-44a3-aded-9c56f2fb3640 None None] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:09:20.643898 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-342789c0-22f4-44a3-aded-9c56f2fb3640 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:09:20.668540 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.auth.core [None req-342789c0-22f4-44a3-aded-9c56f2fb3640 None None] MFA Rules not processed for user `0e2d979563d6421995451a0e86bf3022`. Rule list: `[]` (Enabled: `True`). {{(pid=81408) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 13 04:09:20.669738 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.fernet_utils [None req-342789c0-22f4-44a3-aded-9c56f2fb3640 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81408) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:09:20.716507 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG dbcounter [-] [81409] Writing DB stats keystone:SELECT=100 {{(pid=81409) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:09:20.721079 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG dbcounter [-] [81408] Writing DB stats keystone:SELECT=100 {{(pid=81408) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:09:20.754229 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 297/610] 10.222.0.27 () {58 vars in 1024 bytes} [Thu Oct 13 04:09:20 2022] POST /identity/v3/auth/tokens => generated 1002 bytes in 102 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Oct 13 04:09:20.754229 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-342789c0-22f4-44a3-aded-9c56f2fb3640 None None] Authenticating user token {{(pid=81408) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:09:20.764536 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-49d8cbb4-eaa0-4a5d-95df-97f8e263e0b8 None None] Validating token access rules against request {{(pid=81409) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:09:20.807620 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-36a6f6b0-334c-40b1-8663-82f186222923 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81409) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:09:20.808657 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36a6f6b0-334c-40b1-8663-82f186222923 None admin] REQUEST_METHOD: `GET` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:09:20.809018 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36a6f6b0-334c-40b1-8663-82f186222923 None admin] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:09:20.809351 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36a6f6b0-334c-40b1-8663-82f186222923 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:09:20.810148 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-36a6f6b0-334c-40b1-8663-82f186222923 None admin] RBAC: Adding query filter params (role.id=ef55fbd94e1040be83695754075a7ed5, scope.project.id=dcec0431fb6849499d6aec35a234daa6, user.id=90a0f466b664458898cf9dc4112ae651) {{(pid=81409) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 13 04:09:20.810547 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-36a6f6b0-334c-40b1-8663-82f186222923 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:09:20.812917 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-36a6f6b0-334c-40b1-8663-82f186222923 None admin] RBAC: Authorization granted {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:09:20.820652 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 314/611] 10.222.0.27 () {56 vars in 1374 bytes} [Thu Oct 13 04:09:20 2022] GET /identity/v3/role_assignments?user.id=90a0f466b664458898cf9dc4112ae651&scope.project.id=dcec0431fb6849499d6aec35a234daa6&role.id=ef55fbd94e1040be83695754075a7ed5 => generated 608 bytes in 186 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 13 04:09:20.822004 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d7c5bbc-85d3-401d-b518-aea31085c960 None None] REQUEST_METHOD: `GET` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:09:20.822317 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d7c5bbc-85d3-401d-b518-aea31085c960 None None] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:09:20.822595 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d7c5bbc-85d3-401d-b518-aea31085c960 None None] PATH_INFO: `/` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:09:20.824249 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 315/612] 10.222.0.27 () {54 vars in 927 bytes} [Thu Oct 13 04:09:20 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 13 04:09:20.830388 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79e4de21-7550-40fb-b2f1-8fd7b8ae67bf None None] REQUEST_METHOD: `POST` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:09:20.830798 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79e4de21-7550-40fb-b2f1-8fd7b8ae67bf None None] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:09:20.831172 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79e4de21-7550-40fb-b2f1-8fd7b8ae67bf None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:09:20.853625 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.auth.core [None req-79e4de21-7550-40fb-b2f1-8fd7b8ae67bf None None] MFA Rules not processed for user `0e2d979563d6421995451a0e86bf3022`. Rule list: `[]` (Enabled: `True`). {{(pid=81409) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 13 04:09:20.855147 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.fernet_utils [None req-79e4de21-7550-40fb-b2f1-8fd7b8ae67bf None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81409) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:09:20.910065 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-342789c0-22f4-44a3-aded-9c56f2fb3640 None None] Validating token access rules against request {{(pid=81408) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:09:20.988459 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b4babda8-5b27-4985-bf7d-eb1ff846458e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81408) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:09:20.989117 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4babda8-5b27-4985-bf7d-eb1ff846458e None admin] REQUEST_METHOD: `GET` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:09:20.989392 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4babda8-5b27-4985-bf7d-eb1ff846458e None admin] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:09:20.989678 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4babda8-5b27-4985-bf7d-eb1ff846458e None admin] PATH_INFO: `/v3/users/glance-swift` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:09:21.006722 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 316/613] 10.222.0.27 () {58 vars in 1024 bytes} [Thu Oct 13 04:09:20 2022] POST /identity/v3/auth/tokens => generated 1002 bytes in 177 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Oct 13 04:09:21.007628 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b4babda8-5b27-4985-bf7d-eb1ff846458e None admin] RBAC: Authorizing `identity:get_user(user_id=glance-swift)` {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:09:21.010013 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b4babda8-5b27-4985-bf7d-eb1ff846458e None admin] RBAC: Authorization granted {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:09:21.032346 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application [None req-b4babda8-5b27-4985-bf7d-eb1ff846458e None admin] Could not find user: glance-swift.: keystone.exception.UserNotFound: Could not find user: glance-swift. Oct 13 04:09:21.032346 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 13 04:09:21.032346 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 13 04:09:21.032346 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 13 04:09:21.032346 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 13 04:09:21.032346 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 13 04:09:21.032346 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 13 04:09:21.032346 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 13 04:09:21.032346 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 13 04:09:21.032346 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 13 04:09:21.032346 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 13 04:09:21.032346 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 13 04:09:21.032346 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Oct 13 04:09:21.032346 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application return self._get_user(user_id) Oct 13 04:09:21.032346 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Oct 13 04:09:21.032346 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Oct 13 04:09:21.033552 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 13 04:09:21.033552 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 13 04:09:21.033552 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Oct 13 04:09:21.033552 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 13 04:09:21.033552 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Oct 13 04:09:21.033552 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 13 04:09:21.033552 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 13 04:09:21.033552 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 13 04:09:21.033552 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 13 04:09:21.033552 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application return self.get_or_create( Oct 13 04:09:21.033552 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 13 04:09:21.033552 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application with Lock( Oct 13 04:09:21.033552 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 13 04:09:21.033552 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application return self._enter() Oct 13 04:09:21.033552 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 13 04:09:21.033552 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 13 04:09:21.033552 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 13 04:09:21.033552 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application return self.creator() Oct 13 04:09:21.034541 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 13 04:09:21.034541 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application created_value = creator( Oct 13 04:09:21.034541 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Oct 13 04:09:21.034541 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Oct 13 04:09:21.034541 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Oct 13 04:09:21.034541 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Oct 13 04:09:21.034541 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Oct 13 04:09:21.034541 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Oct 13 04:09:21.034541 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: glance-swift. Oct 13 04:09:21.034541 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application Oct 13 04:09:21.034541 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 298/614] 10.222.0.27 () {56 vars in 1117 bytes} [Thu Oct 13 04:09:20 2022] GET /identity/v3/users/glance-swift => generated 90 bytes in 289 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 13 04:09:21.040234 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-79e4de21-7550-40fb-b2f1-8fd7b8ae67bf None None] Authenticating user token {{(pid=81409) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:09:21.084735 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b4babda8-5b27-4985-bf7d-eb1ff846458e None admin] Authenticating user token {{(pid=81408) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:09:21.110435 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG dbcounter [-] [81409] Writing DB stats keystone:SELECT=100 {{(pid=81409) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:09:21.135073 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG dbcounter [-] [81408] Writing DB stats keystone:SELECT=100 {{(pid=81408) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:09:21.152581 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-79e4de21-7550-40fb-b2f1-8fd7b8ae67bf None None] Validating token access rules against request {{(pid=81409) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:09:21.188297 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c0837c03-8d6d-4860-a768-757c46e4f3a6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81409) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:09:21.188904 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0837c03-8d6d-4860-a768-757c46e4f3a6 None admin] REQUEST_METHOD: `GET` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:09:21.189179 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0837c03-8d6d-4860-a768-757c46e4f3a6 None admin] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:09:21.189453 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0837c03-8d6d-4860-a768-757c46e4f3a6 None admin] PATH_INFO: `/v3/users` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:09:21.190180 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c0837c03-8d6d-4860-a768-757c46e4f3a6 None admin] RBAC: Adding query filter params (domain_id=default, name=glance-swift) {{(pid=81409) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 13 04:09:21.190834 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c0837c03-8d6d-4860-a768-757c46e4f3a6 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:09:21.191968 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c0837c03-8d6d-4860-a768-757c46e4f3a6 None admin] RBAC: Authorization granted {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:09:21.204373 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b4babda8-5b27-4985-bf7d-eb1ff846458e None admin] Validating token access rules against request {{(pid=81408) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:09:21.213444 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 317/615] 10.222.0.27 () {56 vars in 1149 bytes} [Thu Oct 13 04:09:21 2022] GET /identity/v3/users?domain_id=default&name=glance-swift => generated 384 bytes in 171 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 13 04:09:21.217812 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c0837c03-8d6d-4860-a768-757c46e4f3a6 None admin] Authenticating user token {{(pid=81409) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:09:21.290697 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-73b7aa48-c63a-4f80-913b-81d446d1e565 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81408) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:09:21.291340 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-73b7aa48-c63a-4f80-913b-81d446d1e565 None admin] REQUEST_METHOD: `GET` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:09:21.291621 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-73b7aa48-c63a-4f80-913b-81d446d1e565 None admin] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:09:21.291896 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-73b7aa48-c63a-4f80-913b-81d446d1e565 None admin] PATH_INFO: `/v3/services/compute_legacy` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:09:21.292492 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-73b7aa48-c63a-4f80-913b-81d446d1e565 None admin] RBAC: Authorizing `identity:get_service(service_id=compute_legacy)` {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:09:21.293872 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-73b7aa48-c63a-4f80-913b-81d446d1e565 None admin] RBAC: Authorization granted {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:09:21.318049 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application [None req-73b7aa48-c63a-4f80-913b-81d446d1e565 None admin] Could not find service: compute_legacy.: keystone.exception.ServiceNotFound: Could not find service: compute_legacy. Oct 13 04:09:21.318049 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 13 04:09:21.318049 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Oct 13 04:09:21.318049 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Oct 13 04:09:21.318049 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Oct 13 04:09:21.318049 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Oct 13 04:09:21.318049 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Oct 13 04:09:21.318049 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 13 04:09:21.318049 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: compute_legacy. Oct 13 04:09:21.318049 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application Oct 13 04:09:21.318049 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Oct 13 04:09:21.318049 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application Oct 13 04:09:21.318049 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 13 04:09:21.318049 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 13 04:09:21.318049 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 13 04:09:21.318049 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 13 04:09:21.319127 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 13 04:09:21.319127 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 13 04:09:21.319127 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 13 04:09:21.319127 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 13 04:09:21.319127 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 13 04:09:21.319127 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 13 04:09:21.319127 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 13 04:09:21.319127 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Oct 13 04:09:21.319127 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application return self._get_service(service_id) Oct 13 04:09:21.319127 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Oct 13 04:09:21.319127 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Oct 13 04:09:21.319127 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 13 04:09:21.319127 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 13 04:09:21.319127 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 13 04:09:21.319127 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 13 04:09:21.319127 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 13 04:09:21.319127 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application return self.get_or_create( Oct 13 04:09:21.320040 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 13 04:09:21.320040 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application with Lock( Oct 13 04:09:21.320040 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 13 04:09:21.320040 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application return self._enter() Oct 13 04:09:21.320040 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 13 04:09:21.320040 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 13 04:09:21.320040 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 13 04:09:21.320040 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application return self.creator() Oct 13 04:09:21.320040 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 13 04:09:21.320040 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application created_value = creator( Oct 13 04:09:21.320040 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Oct 13 04:09:21.320040 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 13 04:09:21.320040 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: compute_legacy. Oct 13 04:09:21.320040 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application Oct 13 04:09:21.320040 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 299/616] 10.222.0.27 () {56 vars in 1132 bytes} [Thu Oct 13 04:09:21 2022] GET /identity/v3/services/compute_legacy => generated 95 bytes in 235 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 13 04:09:21.325238 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-73b7aa48-c63a-4f80-913b-81d446d1e565 None admin] Authenticating user token {{(pid=81408) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:09:21.344655 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c0837c03-8d6d-4860-a768-757c46e4f3a6 None admin] Validating token access rules against request {{(pid=81409) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:09:21.373240 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG dbcounter [-] [81409] Writing DB stats keystone:SELECT=100 {{(pid=81409) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:09:21.390108 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0f0469c6-d9d1-487e-b281-d57e7612af5c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81409) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:09:21.390774 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f0469c6-d9d1-487e-b281-d57e7612af5c None admin] REQUEST_METHOD: `GET` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:09:21.391051 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f0469c6-d9d1-487e-b281-d57e7612af5c None admin] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:09:21.391321 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f0469c6-d9d1-487e-b281-d57e7612af5c None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:09:21.451908 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-73b7aa48-c63a-4f80-913b-81d446d1e565 None admin] Validating token access rules against request {{(pid=81408) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:09:21.456071 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG dbcounter [-] [81408] Writing DB stats keystone:SELECT=100 {{(pid=81408) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:09:21.475361 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0f0469c6-d9d1-487e-b281-d57e7612af5c None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:09:21.477131 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0f0469c6-d9d1-487e-b281-d57e7612af5c None admin] RBAC: Authorization granted {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:09:21.492517 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0daada99-312d-4f77-8d9d-013dcaacc3a5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81408) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:09:21.494445 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0daada99-312d-4f77-8d9d-013dcaacc3a5 None admin] REQUEST_METHOD: `GET` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:09:21.494972 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0daada99-312d-4f77-8d9d-013dcaacc3a5 None admin] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:09:21.495435 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0daada99-312d-4f77-8d9d-013dcaacc3a5 None admin] PATH_INFO: `/v3/services/block-storage` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:09:21.496243 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0daada99-312d-4f77-8d9d-013dcaacc3a5 None admin] RBAC: Authorizing `identity:get_service(service_id=block-storage)` {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:09:21.498055 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0daada99-312d-4f77-8d9d-013dcaacc3a5 None admin] RBAC: Authorization granted {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:09:21.508006 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application [None req-0daada99-312d-4f77-8d9d-013dcaacc3a5 None admin] Could not find service: block-storage.: keystone.exception.ServiceNotFound: Could not find service: block-storage. Oct 13 04:09:21.508006 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 13 04:09:21.508006 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Oct 13 04:09:21.508006 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Oct 13 04:09:21.508006 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Oct 13 04:09:21.508006 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Oct 13 04:09:21.508006 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Oct 13 04:09:21.508006 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 13 04:09:21.508006 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: block-storage. Oct 13 04:09:21.508006 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application Oct 13 04:09:21.508006 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Oct 13 04:09:21.508006 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application Oct 13 04:09:21.508006 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 13 04:09:21.508006 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 13 04:09:21.508006 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 13 04:09:21.508006 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 13 04:09:21.508006 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 13 04:09:21.509790 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 13 04:09:21.509790 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 13 04:09:21.509790 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 13 04:09:21.509790 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 13 04:09:21.509790 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 13 04:09:21.509790 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 13 04:09:21.509790 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Oct 13 04:09:21.509790 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application return self._get_service(service_id) Oct 13 04:09:21.509790 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Oct 13 04:09:21.509790 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Oct 13 04:09:21.509790 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 13 04:09:21.509790 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 13 04:09:21.509790 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 13 04:09:21.509790 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 13 04:09:21.509790 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 13 04:09:21.509790 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application return self.get_or_create( Oct 13 04:09:21.511695 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 13 04:09:21.511695 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application with Lock( Oct 13 04:09:21.511695 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 13 04:09:21.511695 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application return self._enter() Oct 13 04:09:21.511695 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 13 04:09:21.511695 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 13 04:09:21.511695 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 13 04:09:21.511695 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application return self.creator() Oct 13 04:09:21.511695 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 13 04:09:21.511695 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application created_value = creator( Oct 13 04:09:21.511695 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Oct 13 04:09:21.511695 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 13 04:09:21.511695 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: block-storage. Oct 13 04:09:21.511695 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application Oct 13 04:09:21.521762 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 300/617] 10.222.0.27 () {56 vars in 1129 bytes} [Thu Oct 13 04:09:21 2022] GET /identity/v3/services/block-storage => generated 94 bytes in 192 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 13 04:09:21.523735 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0daada99-312d-4f77-8d9d-013dcaacc3a5 None admin] Authenticating user token {{(pid=81408) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:09:21.598547 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 318/618] 10.222.0.27 () {58 vars in 1282 bytes} [Thu Oct 13 04:09:21 2022] GET /identity/v3/auth/tokens => generated 1002 bytes in 381 msecs (HTTP/1.1 200) 6 headers in 359 bytes (1 switches on core 0) Oct 13 04:09:21.599492 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0f0469c6-d9d1-487e-b281-d57e7612af5c None admin] Authenticating user token {{(pid=81409) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:09:21.629397 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG dbcounter [-] [81409] Writing DB stats keystone:SELECT=100 {{(pid=81409) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:09:21.657830 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0daada99-312d-4f77-8d9d-013dcaacc3a5 None admin] Validating token access rules against request {{(pid=81408) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:09:21.698026 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2331dadd-587d-470b-b0c4-d94c6534ebb7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81408) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:09:21.699230 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2331dadd-587d-470b-b0c4-d94c6534ebb7 None admin] REQUEST_METHOD: `GET` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:09:21.699794 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2331dadd-587d-470b-b0c4-d94c6534ebb7 None admin] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:09:21.700256 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2331dadd-587d-470b-b0c4-d94c6534ebb7 None admin] PATH_INFO: `/v3/services` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:09:21.701146 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2331dadd-587d-470b-b0c4-d94c6534ebb7 None admin] RBAC: Adding query filter params (name=compute_legacy) {{(pid=81408) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 13 04:09:21.701729 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2331dadd-587d-470b-b0c4-d94c6534ebb7 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:09:21.703541 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2331dadd-587d-470b-b0c4-d94c6534ebb7 None admin] RBAC: Authorization granted {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:09:21.708900 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 301/619] 10.222.0.27 () {56 vars in 1126 bytes} [Thu Oct 13 04:09:21 2022] GET /identity/v3/services?name=compute_legacy => generated 131 bytes in 186 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 13 04:09:21.710050 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2331dadd-587d-470b-b0c4-d94c6534ebb7 None admin] Authenticating user token {{(pid=81408) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:09:21.719472 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0f0469c6-d9d1-487e-b281-d57e7612af5c None admin] Validating token access rules against request {{(pid=81409) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:09:21.746854 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG dbcounter [-] [81408] Writing DB stats keystone:SELECT=100 {{(pid=81408) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:09:21.761667 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2cb185de-5b5c-4f8f-923f-70497765d820 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81409) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:09:21.762430 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2cb185de-5b5c-4f8f-923f-70497765d820 None admin] REQUEST_METHOD: `GET` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:09:21.762709 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2cb185de-5b5c-4f8f-923f-70497765d820 None admin] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:09:21.762983 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2cb185de-5b5c-4f8f-923f-70497765d820 None admin] PATH_INFO: `/v3/services` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:09:21.763850 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2cb185de-5b5c-4f8f-923f-70497765d820 None admin] RBAC: Adding query filter params (name=block-storage) {{(pid=81409) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 13 04:09:21.764039 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2cb185de-5b5c-4f8f-923f-70497765d820 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:09:21.765405 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2cb185de-5b5c-4f8f-923f-70497765d820 None admin] RBAC: Authorization granted {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:09:21.775862 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 319/620] 10.222.0.27 () {56 vars in 1124 bytes} [Thu Oct 13 04:09:21 2022] GET /identity/v3/services?name=block-storage => generated 130 bytes in 172 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 13 04:09:21.775862 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2cb185de-5b5c-4f8f-923f-70497765d820 None admin] Authenticating user token {{(pid=81409) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:09:21.837826 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2331dadd-587d-470b-b0c4-d94c6534ebb7 None admin] Validating token access rules against request {{(pid=81408) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:09:21.879740 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3e211cae-0100-4458-af0b-3183c87031f1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81408) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:09:21.880750 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e211cae-0100-4458-af0b-3183c87031f1 None admin] REQUEST_METHOD: `GET` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:09:21.881232 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e211cae-0100-4458-af0b-3183c87031f1 None admin] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:09:21.881713 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e211cae-0100-4458-af0b-3183c87031f1 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:09:21.881850 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG dbcounter [-] [81409] Writing DB stats keystone:SELECT=100 {{(pid=81409) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:09:21.895203 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3e211cae-0100-4458-af0b-3183c87031f1 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:09:21.896779 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3e211cae-0100-4458-af0b-3183c87031f1 None admin] RBAC: Authorization granted {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:09:21.901769 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2cb185de-5b5c-4f8f-923f-70497765d820 None admin] Validating token access rules against request {{(pid=81409) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:09:21.913457 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application [None req-3e211cae-0100-4458-af0b-3183c87031f1 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 13 04:09:21.913457 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 13 04:09:21.913457 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 13 04:09:21.913457 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 13 04:09:21.913457 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 13 04:09:21.913457 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 13 04:09:21.913457 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 13 04:09:21.913457 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 13 04:09:21.913457 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 13 04:09:21.913457 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 13 04:09:21.913457 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 13 04:09:21.913457 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 13 04:09:21.913457 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 13 04:09:21.913457 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 13 04:09:21.913457 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 13 04:09:21.913457 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 13 04:09:21.914728 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 13 04:09:21.914728 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 13 04:09:21.914728 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 13 04:09:21.914728 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 13 04:09:21.914728 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 13 04:09:21.914728 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application return self.get_or_create( Oct 13 04:09:21.914728 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 13 04:09:21.914728 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application with Lock( Oct 13 04:09:21.914728 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 13 04:09:21.914728 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application return self._enter() Oct 13 04:09:21.914728 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 13 04:09:21.914728 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 13 04:09:21.914728 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 13 04:09:21.914728 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application return self.creator() Oct 13 04:09:21.914728 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 13 04:09:21.914728 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application created_value = creator( Oct 13 04:09:21.914728 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 13 04:09:21.915801 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 13 04:09:21.915801 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 13 04:09:21.915801 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application Oct 13 04:09:21.917203 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 302/621] 10.222.0.27 () {56 vars in 1108 bytes} [Thu Oct 13 04:09:21 2022] GET /identity/v3/domains/Default => generated 87 bytes in 207 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 13 04:09:21.918144 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3e211cae-0100-4458-af0b-3183c87031f1 None admin] Authenticating user token {{(pid=81408) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:09:21.949274 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-908e6789-2ae3-4c65-ba18-ab50b5c09bb9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81409) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:09:21.950274 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-908e6789-2ae3-4c65-ba18-ab50b5c09bb9 None admin] REQUEST_METHOD: `GET` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:09:21.950703 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-908e6789-2ae3-4c65-ba18-ab50b5c09bb9 None admin] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:09:21.951120 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-908e6789-2ae3-4c65-ba18-ab50b5c09bb9 None admin] PATH_INFO: `/v3/services` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:09:21.951981 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-908e6789-2ae3-4c65-ba18-ab50b5c09bb9 None admin] RBAC: Adding query filter params (type=compute_legacy) {{(pid=81409) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 13 04:09:21.952774 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-908e6789-2ae3-4c65-ba18-ab50b5c09bb9 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:09:21.955137 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-908e6789-2ae3-4c65-ba18-ab50b5c09bb9 None admin] RBAC: Authorization granted {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:09:21.962268 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 320/622] 10.222.0.27 () {56 vars in 1126 bytes} [Thu Oct 13 04:09:21 2022] GET /identity/v3/services?type=compute_legacy => generated 131 bytes in 190 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 13 04:09:21.964383 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-908e6789-2ae3-4c65-ba18-ab50b5c09bb9 None admin] Authenticating user token {{(pid=81409) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:09:22.027404 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG dbcounter [-] [81408] Writing DB stats keystone:SELECT=100 {{(pid=81408) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:09:22.045265 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3e211cae-0100-4458-af0b-3183c87031f1 None admin] Validating token access rules against request {{(pid=81408) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:09:22.083925 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c8adbe50-6fd3-4efe-a64a-ab37fbaaa7e0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81408) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:09:22.084830 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8adbe50-6fd3-4efe-a64a-ab37fbaaa7e0 None admin] REQUEST_METHOD: `GET` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:09:22.085204 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8adbe50-6fd3-4efe-a64a-ab37fbaaa7e0 None admin] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:09:22.085571 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8adbe50-6fd3-4efe-a64a-ab37fbaaa7e0 None admin] PATH_INFO: `/v3/services` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:09:22.086350 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c8adbe50-6fd3-4efe-a64a-ab37fbaaa7e0 None admin] RBAC: Adding query filter params (type=block-storage) {{(pid=81408) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 13 04:09:22.087503 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c8adbe50-6fd3-4efe-a64a-ab37fbaaa7e0 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:09:22.089544 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c8adbe50-6fd3-4efe-a64a-ab37fbaaa7e0 None admin] RBAC: Authorization granted {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:09:22.092474 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-908e6789-2ae3-4c65-ba18-ab50b5c09bb9 None admin] Validating token access rules against request {{(pid=81409) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:09:22.095871 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 303/623] 10.222.0.27 () {56 vars in 1124 bytes} [Thu Oct 13 04:09:21 2022] GET /identity/v3/services?type=block-storage => generated 367 bytes in 178 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 13 04:09:22.107122 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c8adbe50-6fd3-4efe-a64a-ab37fbaaa7e0 None admin] Authenticating user token {{(pid=81408) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:09:22.134070 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-82e2360c-736a-4d79-b539-29ad0ea6d3ec None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81409) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:09:22.135472 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82e2360c-736a-4d79-b539-29ad0ea6d3ec None admin] REQUEST_METHOD: `GET` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:09:22.135918 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82e2360c-736a-4d79-b539-29ad0ea6d3ec None admin] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:09:22.136339 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82e2360c-736a-4d79-b539-29ad0ea6d3ec None admin] PATH_INFO: `/v3/domains` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:09:22.137152 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-82e2360c-736a-4d79-b539-29ad0ea6d3ec None admin] RBAC: Adding query filter params (name=Default) {{(pid=81409) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 13 04:09:22.137607 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-82e2360c-736a-4d79-b539-29ad0ea6d3ec None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:09:22.139195 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-82e2360c-736a-4d79-b539-29ad0ea6d3ec None admin] RBAC: Authorization granted {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:09:22.154383 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 321/624] 10.222.0.27 () {56 vars in 1109 bytes} [Thu Oct 13 04:09:21 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 191 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 13 04:09:22.163553 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-82e2360c-736a-4d79-b539-29ad0ea6d3ec None admin] Authenticating user token {{(pid=81409) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:09:22.183503 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG dbcounter [-] [81409] Writing DB stats keystone:SELECT=100 {{(pid=81409) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:09:22.237148 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c8adbe50-6fd3-4efe-a64a-ab37fbaaa7e0 None admin] Validating token access rules against request {{(pid=81408) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:09:22.282717 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fd963640-fcf6-4f17-8c31-056fb4b40730 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81408) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:09:22.283548 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd963640-fcf6-4f17-8c31-056fb4b40730 None admin] REQUEST_METHOD: `POST` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:09:22.283897 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd963640-fcf6-4f17-8c31-056fb4b40730 None admin] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:09:22.284234 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd963640-fcf6-4f17-8c31-056fb4b40730 None admin] PATH_INFO: `/v3/endpoints` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:09:22.285176 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fd963640-fcf6-4f17-8c31-056fb4b40730 None admin] RBAC: Authorizing `identity:create_endpoint()` {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:09:22.286678 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fd963640-fcf6-4f17-8c31-056fb4b40730 None admin] RBAC: Authorization granted {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:09:22.288169 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-82e2360c-736a-4d79-b539-29ad0ea6d3ec None admin] Validating token access rules against request {{(pid=81409) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:09:22.318526 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 304/625] 10.222.0.27 () {60 vars in 1144 bytes} [Thu Oct 13 04:09:22 2022] POST /identity/v3/endpoints => generated 346 bytes in 208 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 13 04:09:22.360694 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff263e34-3cf1-4f9c-a040-e1791825d277 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81409) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:09:22.361354 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff263e34-3cf1-4f9c-a040-e1791825d277 None admin] REQUEST_METHOD: `GET` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:09:22.361628 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff263e34-3cf1-4f9c-a040-e1791825d277 None admin] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:09:22.361898 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff263e34-3cf1-4f9c-a040-e1791825d277 None admin] PATH_INFO: `/v3/projects/service` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:09:22.371430 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ff263e34-3cf1-4f9c-a040-e1791825d277 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:09:22.372909 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ff263e34-3cf1-4f9c-a040-e1791825d277 None admin] RBAC: Authorization granted {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:09:22.386856 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application [None req-ff263e34-3cf1-4f9c-a040-e1791825d277 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Oct 13 04:09:22.386856 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 13 04:09:22.386856 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 13 04:09:22.386856 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 13 04:09:22.386856 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 13 04:09:22.386856 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 13 04:09:22.386856 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 13 04:09:22.386856 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 13 04:09:22.386856 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 13 04:09:22.386856 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 13 04:09:22.386856 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 13 04:09:22.386856 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 13 04:09:22.386856 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Oct 13 04:09:22.386856 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application return self._get_project(project_id) Oct 13 04:09:22.386856 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Oct 13 04:09:22.387823 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Oct 13 04:09:22.387823 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 13 04:09:22.387823 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 13 04:09:22.387823 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 13 04:09:22.387823 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 13 04:09:22.387823 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 13 04:09:22.387823 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application return self.get_or_create( Oct 13 04:09:22.387823 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 13 04:09:22.387823 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application with Lock( Oct 13 04:09:22.387823 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 13 04:09:22.387823 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application return self._enter() Oct 13 04:09:22.387823 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 13 04:09:22.387823 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 13 04:09:22.387823 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 13 04:09:22.387823 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application return self.creator() Oct 13 04:09:22.387823 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 13 04:09:22.387823 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application created_value = creator( Oct 13 04:09:22.388745 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Oct 13 04:09:22.388745 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Oct 13 04:09:22.388745 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Oct 13 04:09:22.388745 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Oct 13 04:09:22.388745 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Oct 13 04:09:22.388745 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Oct 13 04:09:22.388745 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Oct 13 04:09:22.388745 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application Oct 13 04:09:22.388745 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 322/626] 10.222.0.27 () {56 vars in 1111 bytes} [Thu Oct 13 04:09:22 2022] GET /identity/v3/projects/service => generated 88 bytes in 225 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 13 04:09:22.394407 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fd963640-fcf6-4f17-8c31-056fb4b40730 None admin] Authenticating user token {{(pid=81408) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:09:22.398091 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG dbcounter [-] [81408] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81408) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:09:22.406323 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b49e2c4d-4e0e-4a2b-b9b7-25f255b5cad0 None None] REQUEST_METHOD: `GET` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:09:22.407239 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b49e2c4d-4e0e-4a2b-b9b7-25f255b5cad0 None None] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:09:22.407714 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b49e2c4d-4e0e-4a2b-b9b7-25f255b5cad0 None None] PATH_INFO: `/` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:09:22.410595 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 323/627] 10.222.0.27 () {54 vars in 927 bytes} [Thu Oct 13 04:09:22 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Oct 13 04:09:22.418324 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd54cd0a-7ce6-4077-89a4-17ce60eac693 None None] REQUEST_METHOD: `POST` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:09:22.418324 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd54cd0a-7ce6-4077-89a4-17ce60eac693 None None] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:09:22.421615 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd54cd0a-7ce6-4077-89a4-17ce60eac693 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:09:22.463854 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.auth.core [None req-dd54cd0a-7ce6-4077-89a4-17ce60eac693 None None] MFA Rules not processed for user `0e2d979563d6421995451a0e86bf3022`. Rule list: `[]` (Enabled: `True`). {{(pid=81409) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 13 04:09:22.467840 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.fernet_utils [None req-dd54cd0a-7ce6-4077-89a4-17ce60eac693 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81409) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:09:22.580662 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fd963640-fcf6-4f17-8c31-056fb4b40730 None admin] Validating token access rules against request {{(pid=81408) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:09:22.586259 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG dbcounter [-] [81409] Writing DB stats keystone:SELECT=100 {{(pid=81409) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:09:22.596497 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 324/628] 10.222.0.27 () {58 vars in 1024 bytes} [Thu Oct 13 04:09:22 2022] POST /identity/v3/auth/tokens => generated 1002 bytes in 181 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Oct 13 04:09:22.620776 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3e3ae667-8ef5-4810-ac8d-8c633b385878 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81408) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:09:22.621662 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e3ae667-8ef5-4810-ac8d-8c633b385878 None admin] REQUEST_METHOD: `GET` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:09:22.622257 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e3ae667-8ef5-4810-ac8d-8c633b385878 None admin] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:09:22.622707 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e3ae667-8ef5-4810-ac8d-8c633b385878 None admin] PATH_INFO: `/v3/projects` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:09:22.623591 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3e3ae667-8ef5-4810-ac8d-8c633b385878 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81408) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 13 04:09:22.624074 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3e3ae667-8ef5-4810-ac8d-8c633b385878 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:09:22.625709 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3e3ae667-8ef5-4810-ac8d-8c633b385878 None admin] RBAC: Authorization granted {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:09:22.639140 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 305/629] 10.222.0.27 () {56 vars in 1148 bytes} [Thu Oct 13 04:09:22 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 245 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 13 04:09:22.644828 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dd54cd0a-7ce6-4077-89a4-17ce60eac693 None None] Authenticating user token {{(pid=81409) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:09:22.756394 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dd54cd0a-7ce6-4077-89a4-17ce60eac693 None None] Validating token access rules against request {{(pid=81409) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:09:22.803583 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1ff01c51-3c95-435d-a6ba-e5be926d0fe7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81409) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:09:22.804490 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ff01c51-3c95-435d-a6ba-e5be926d0fe7 None admin] REQUEST_METHOD: `GET` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:09:22.804969 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ff01c51-3c95-435d-a6ba-e5be926d0fe7 None admin] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:09:22.805937 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ff01c51-3c95-435d-a6ba-e5be926d0fe7 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:09:22.806928 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1ff01c51-3c95-435d-a6ba-e5be926d0fe7 None admin] RBAC: Adding query filter params (role.id=4cfff1d3d68e49c9b04231a19cf76ede, scope.project.id=dcec0431fb6849499d6aec35a234daa6, user.id=487b1605fd4143cfaa422a3a7a0344c7) {{(pid=81409) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 13 04:09:22.807411 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1ff01c51-3c95-435d-a6ba-e5be926d0fe7 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:09:22.809142 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1ff01c51-3c95-435d-a6ba-e5be926d0fe7 None admin] RBAC: Authorization granted {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:09:22.816884 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 325/630] 10.222.0.27 () {56 vars in 1374 bytes} [Thu Oct 13 04:09:22 2022] GET /identity/v3/role_assignments?user.id=487b1605fd4143cfaa422a3a7a0344c7&scope.project.id=dcec0431fb6849499d6aec35a234daa6&role.id=4cfff1d3d68e49c9b04231a19cf76ede => generated 608 bytes in 173 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 13 04:09:22.894070 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3e3ae667-8ef5-4810-ac8d-8c633b385878 None admin] Authenticating user token {{(pid=81408) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:09:22.993294 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG dbcounter [-] [81408] Writing DB stats keystone:SELECT=100 {{(pid=81408) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:09:23.039339 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3e3ae667-8ef5-4810-ac8d-8c633b385878 None admin] Validating token access rules against request {{(pid=81408) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:09:23.076386 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5cae5418-25fd-4bc6-9154-37f0db8fcd37 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81408) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:09:23.077018 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5cae5418-25fd-4bc6-9154-37f0db8fcd37 None admin] REQUEST_METHOD: `GET` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:09:23.077293 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5cae5418-25fd-4bc6-9154-37f0db8fcd37 None admin] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:09:23.077560 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5cae5418-25fd-4bc6-9154-37f0db8fcd37 None admin] PATH_INFO: `/v3/services/network` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:09:23.078145 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5cae5418-25fd-4bc6-9154-37f0db8fcd37 None admin] RBAC: Authorizing `identity:get_service(service_id=network)` {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:09:23.079561 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5cae5418-25fd-4bc6-9154-37f0db8fcd37 None admin] RBAC: Authorization granted {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:09:23.089902 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application [None req-5cae5418-25fd-4bc6-9154-37f0db8fcd37 None admin] Could not find service: network.: keystone.exception.ServiceNotFound: Could not find service: network. Oct 13 04:09:23.089902 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 13 04:09:23.089902 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Oct 13 04:09:23.089902 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Oct 13 04:09:23.089902 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Oct 13 04:09:23.089902 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Oct 13 04:09:23.089902 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Oct 13 04:09:23.089902 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 13 04:09:23.089902 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: network. Oct 13 04:09:23.089902 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application Oct 13 04:09:23.089902 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Oct 13 04:09:23.089902 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application Oct 13 04:09:23.089902 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 13 04:09:23.089902 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 13 04:09:23.089902 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 13 04:09:23.089902 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 13 04:09:23.089902 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 13 04:09:23.090965 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 13 04:09:23.090965 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 13 04:09:23.090965 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 13 04:09:23.090965 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 13 04:09:23.090965 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 13 04:09:23.090965 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 13 04:09:23.090965 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Oct 13 04:09:23.090965 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application return self._get_service(service_id) Oct 13 04:09:23.090965 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Oct 13 04:09:23.090965 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Oct 13 04:09:23.090965 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 13 04:09:23.090965 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 13 04:09:23.090965 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 13 04:09:23.090965 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 13 04:09:23.090965 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 13 04:09:23.090965 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application return self.get_or_create( Oct 13 04:09:23.091831 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 13 04:09:23.091831 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application with Lock( Oct 13 04:09:23.091831 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 13 04:09:23.091831 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application return self._enter() Oct 13 04:09:23.091831 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 13 04:09:23.091831 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 13 04:09:23.091831 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 13 04:09:23.091831 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application return self.creator() Oct 13 04:09:23.091831 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 13 04:09:23.091831 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application created_value = creator( Oct 13 04:09:23.091831 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Oct 13 04:09:23.091831 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 13 04:09:23.091831 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: network. Oct 13 04:09:23.091831 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application Oct 13 04:09:23.091831 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 306/631] 10.222.0.27 () {56 vars in 1111 bytes} [Thu Oct 13 04:09:22 2022] GET /identity/v3/services/network => generated 88 bytes in 198 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 13 04:09:23.097401 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1ff01c51-3c95-435d-a6ba-e5be926d0fe7 None admin] Authenticating user token {{(pid=81409) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:09:23.166919 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG dbcounter [-] [81409] Writing DB stats keystone:SELECT=100 {{(pid=81409) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:09:23.214637 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1ff01c51-3c95-435d-a6ba-e5be926d0fe7 None admin] Validating token access rules against request {{(pid=81409) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:09:23.266669 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-543f29ca-3b58-42f6-a35c-2808046fa971 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81409) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:09:23.267279 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-543f29ca-3b58-42f6-a35c-2808046fa971 None admin] REQUEST_METHOD: `GET` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:09:23.267552 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-543f29ca-3b58-42f6-a35c-2808046fa971 None admin] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:09:23.267819 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-543f29ca-3b58-42f6-a35c-2808046fa971 None admin] PATH_INFO: `/v3/services` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:09:23.268443 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-543f29ca-3b58-42f6-a35c-2808046fa971 None admin] RBAC: Adding query filter params (name=network) {{(pid=81409) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 13 04:09:23.268742 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-543f29ca-3b58-42f6-a35c-2808046fa971 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:09:23.270105 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-543f29ca-3b58-42f6-a35c-2808046fa971 None admin] RBAC: Authorization granted {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:09:23.278016 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 326/632] 10.222.0.27 () {56 vars in 1112 bytes} [Thu Oct 13 04:09:23 2022] GET /identity/v3/services?name=network => generated 124 bytes in 182 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 13 04:09:23.283980 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5cae5418-25fd-4bc6-9154-37f0db8fcd37 None admin] Authenticating user token {{(pid=81408) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:09:23.405777 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5cae5418-25fd-4bc6-9154-37f0db8fcd37 None admin] Validating token access rules against request {{(pid=81408) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:09:23.456710 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG dbcounter [-] [81408] Writing DB stats keystone:SELECT=100 {{(pid=81408) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:09:23.469788 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-11eab5e3-b21a-4de4-b32b-996b1180d2f1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81408) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:09:23.470429 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11eab5e3-b21a-4de4-b32b-996b1180d2f1 None admin] REQUEST_METHOD: `GET` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:09:23.470920 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11eab5e3-b21a-4de4-b32b-996b1180d2f1 None admin] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:09:23.471195 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11eab5e3-b21a-4de4-b32b-996b1180d2f1 None admin] PATH_INFO: `/v3/services` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:09:23.471833 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-11eab5e3-b21a-4de4-b32b-996b1180d2f1 None admin] RBAC: Adding query filter params (type=network) {{(pid=81408) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 13 04:09:23.472136 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-11eab5e3-b21a-4de4-b32b-996b1180d2f1 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:09:23.473517 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-11eab5e3-b21a-4de4-b32b-996b1180d2f1 None admin] RBAC: Authorization granted {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:09:23.479506 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 307/633] 10.222.0.27 () {56 vars in 1112 bytes} [Thu Oct 13 04:09:23 2022] GET /identity/v3/services?type=network => generated 350 bytes in 196 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 13 04:09:23.485101 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-543f29ca-3b58-42f6-a35c-2808046fa971 None admin] Authenticating user token {{(pid=81409) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:09:23.562650 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d3b5a75-8e6b-4103-b6f6-f8b4ae81c397 None None] REQUEST_METHOD: `GET` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:09:23.562981 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d3b5a75-8e6b-4103-b6f6-f8b4ae81c397 None None] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:09:23.563207 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d3b5a75-8e6b-4103-b6f6-f8b4ae81c397 None None] PATH_INFO: `/` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:09:23.564834 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 308/634] 10.222.0.27 () {54 vars in 927 bytes} [Thu Oct 13 04:09:23 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 13 04:09:23.570682 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0fac49a-d49f-4830-a999-190fef4b2f91 None None] REQUEST_METHOD: `POST` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:09:23.571035 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0fac49a-d49f-4830-a999-190fef4b2f91 None None] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:09:23.571345 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0fac49a-d49f-4830-a999-190fef4b2f91 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:09:23.603860 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.auth.core [None req-a0fac49a-d49f-4830-a999-190fef4b2f91 None None] MFA Rules not processed for user `0e2d979563d6421995451a0e86bf3022`. Rule list: `[]` (Enabled: `True`). {{(pid=81408) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 13 04:09:23.605417 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.fernet_utils [None req-a0fac49a-d49f-4830-a999-190fef4b2f91 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81408) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:09:23.659201 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-543f29ca-3b58-42f6-a35c-2808046fa971 None admin] Validating token access rules against request {{(pid=81409) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:09:23.704785 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG dbcounter [-] [81409] Writing DB stats keystone:SELECT=100 {{(pid=81409) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:09:23.718473 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 309/635] 10.222.0.27 () {58 vars in 1024 bytes} [Thu Oct 13 04:09:23 2022] POST /identity/v3/auth/tokens => generated 1002 bytes in 149 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Oct 13 04:09:23.724237 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0ecbbe46-5d67-48ef-8d72-923a53c8f506 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81409) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:09:23.724854 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ecbbe46-5d67-48ef-8d72-923a53c8f506 None admin] REQUEST_METHOD: `POST` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:09:23.725128 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ecbbe46-5d67-48ef-8d72-923a53c8f506 None admin] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:09:23.725393 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ecbbe46-5d67-48ef-8d72-923a53c8f506 None admin] PATH_INFO: `/v3/endpoints` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:09:23.726458 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0ecbbe46-5d67-48ef-8d72-923a53c8f506 None admin] RBAC: Authorizing `identity:create_endpoint()` {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:09:23.728205 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0ecbbe46-5d67-48ef-8d72-923a53c8f506 None admin] RBAC: Authorization granted {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:09:23.756612 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 327/636] 10.222.0.27 () {60 vars in 1144 bytes} [Thu Oct 13 04:09:23 2022] POST /identity/v3/endpoints => generated 337 bytes in 272 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 13 04:09:23.995216 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a0fac49a-d49f-4830-a999-190fef4b2f91 None None] Authenticating user token {{(pid=81408) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:09:24.129917 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a0fac49a-d49f-4830-a999-190fef4b2f91 None None] Validating token access rules against request {{(pid=81408) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:09:24.153456 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG dbcounter [-] [81408] Writing DB stats keystone:SELECT=100 {{(pid=81408) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:09:24.174562 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9b58594a-aa43-4180-bd3d-71917350a8fe None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81408) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:09:24.175186 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b58594a-aa43-4180-bd3d-71917350a8fe None admin] REQUEST_METHOD: `GET` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:09:24.175457 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b58594a-aa43-4180-bd3d-71917350a8fe None admin] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:09:24.175724 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b58594a-aa43-4180-bd3d-71917350a8fe None admin] PATH_INFO: `/v3/services/object-store` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:09:24.176301 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9b58594a-aa43-4180-bd3d-71917350a8fe None admin] RBAC: Authorizing `identity:get_service(service_id=object-store)` {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:09:24.177666 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9b58594a-aa43-4180-bd3d-71917350a8fe None admin] RBAC: Authorization granted {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:09:24.187385 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application [None req-9b58594a-aa43-4180-bd3d-71917350a8fe None admin] Could not find service: object-store.: keystone.exception.ServiceNotFound: Could not find service: object-store. Oct 13 04:09:24.187385 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 13 04:09:24.187385 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Oct 13 04:09:24.187385 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Oct 13 04:09:24.187385 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Oct 13 04:09:24.187385 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Oct 13 04:09:24.187385 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Oct 13 04:09:24.187385 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 13 04:09:24.187385 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: object-store. Oct 13 04:09:24.187385 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application Oct 13 04:09:24.187385 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Oct 13 04:09:24.187385 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application Oct 13 04:09:24.187385 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 13 04:09:24.187385 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 13 04:09:24.187385 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 13 04:09:24.187385 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 13 04:09:24.187385 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 13 04:09:24.188342 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 13 04:09:24.188342 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 13 04:09:24.188342 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 13 04:09:24.188342 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 13 04:09:24.188342 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 13 04:09:24.188342 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 13 04:09:24.188342 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Oct 13 04:09:24.188342 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application return self._get_service(service_id) Oct 13 04:09:24.188342 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Oct 13 04:09:24.188342 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Oct 13 04:09:24.188342 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 13 04:09:24.188342 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 13 04:09:24.188342 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 13 04:09:24.188342 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 13 04:09:24.188342 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 13 04:09:24.188342 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application return self.get_or_create( Oct 13 04:09:24.189199 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 13 04:09:24.189199 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application with Lock( Oct 13 04:09:24.189199 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 13 04:09:24.189199 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application return self._enter() Oct 13 04:09:24.189199 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 13 04:09:24.189199 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 13 04:09:24.189199 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 13 04:09:24.189199 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application return self.creator() Oct 13 04:09:24.189199 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 13 04:09:24.189199 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application created_value = creator( Oct 13 04:09:24.189199 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Oct 13 04:09:24.189199 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 13 04:09:24.189199 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: object-store. Oct 13 04:09:24.189199 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application Oct 13 04:09:24.189199 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 310/637] 10.222.0.27 () {56 vars in 1126 bytes} [Thu Oct 13 04:09:23 2022] GET /identity/v3/services/object-store => generated 93 bytes in 194 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 13 04:09:24.194351 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0ecbbe46-5d67-48ef-8d72-923a53c8f506 None admin] Authenticating user token {{(pid=81409) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:09:24.315363 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0ecbbe46-5d67-48ef-8d72-923a53c8f506 None admin] Validating token access rules against request {{(pid=81409) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:09:24.355545 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eb73db11-9ab7-40f3-a0c4-0c42d11773de None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81409) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:09:24.356155 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb73db11-9ab7-40f3-a0c4-0c42d11773de None admin] REQUEST_METHOD: `GET` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:09:24.356433 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb73db11-9ab7-40f3-a0c4-0c42d11773de None admin] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:09:24.356701 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb73db11-9ab7-40f3-a0c4-0c42d11773de None admin] PATH_INFO: `/v3/services` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:09:24.357335 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eb73db11-9ab7-40f3-a0c4-0c42d11773de None admin] RBAC: Adding query filter params (name=object-store) {{(pid=81409) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 13 04:09:24.357637 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eb73db11-9ab7-40f3-a0c4-0c42d11773de None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:09:24.359054 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eb73db11-9ab7-40f3-a0c4-0c42d11773de None admin] RBAC: Authorization granted {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:09:24.364423 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 328/638] 10.222.0.27 () {56 vars in 1122 bytes} [Thu Oct 13 04:09:24 2022] GET /identity/v3/services?name=object-store => generated 129 bytes in 170 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 13 04:09:24.370160 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9b58594a-aa43-4180-bd3d-71917350a8fe None admin] Authenticating user token {{(pid=81408) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:09:24.487655 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9b58594a-aa43-4180-bd3d-71917350a8fe None admin] Validating token access rules against request {{(pid=81408) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:09:24.522570 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d12ed254-4c24-475d-880d-9a5375aa2753 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81408) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:09:24.523176 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d12ed254-4c24-475d-880d-9a5375aa2753 None admin] REQUEST_METHOD: `GET` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:09:24.523461 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d12ed254-4c24-475d-880d-9a5375aa2753 None admin] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:09:24.523728 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d12ed254-4c24-475d-880d-9a5375aa2753 None admin] PATH_INFO: `/v3/services` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:09:24.524489 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d12ed254-4c24-475d-880d-9a5375aa2753 None admin] RBAC: Adding query filter params (type=object-store) {{(pid=81408) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 13 04:09:24.524872 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d12ed254-4c24-475d-880d-9a5375aa2753 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:09:24.526265 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d12ed254-4c24-475d-880d-9a5375aa2753 None admin] RBAC: Authorization granted {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:09:24.531229 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 311/639] 10.222.0.27 () {56 vars in 1122 bytes} [Thu Oct 13 04:09:24 2022] GET /identity/v3/services?type=object-store => generated 129 bytes in 161 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 13 04:09:24.889886 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-583bb0dd-f59f-4dd6-9d9d-3a38a61542c1 None None] REQUEST_METHOD: `GET` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:09:24.890425 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-583bb0dd-f59f-4dd6-9d9d-3a38a61542c1 None None] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:09:24.890503 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-583bb0dd-f59f-4dd6-9d9d-3a38a61542c1 None None] PATH_INFO: `/` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:09:24.892054 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 329/640] 10.222.0.27 () {54 vars in 927 bytes} [Thu Oct 13 04:09:24 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 13 04:09:24.897632 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79ba1213-590a-4391-ab98-5497ff065240 None None] REQUEST_METHOD: `POST` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:09:24.897918 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79ba1213-590a-4391-ab98-5497ff065240 None None] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:09:24.898193 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79ba1213-590a-4391-ab98-5497ff065240 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:09:24.917459 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.auth.core [None req-79ba1213-590a-4391-ab98-5497ff065240 None None] MFA Rules not processed for user `0e2d979563d6421995451a0e86bf3022`. Rule list: `[]` (Enabled: `True`). {{(pid=81408) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 13 04:09:24.918752 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.fernet_utils [None req-79ba1213-590a-4391-ab98-5497ff065240 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81408) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:09:24.967406 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG dbcounter [-] [81408] Writing DB stats keystone:SELECT=100 {{(pid=81408) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:09:24.998334 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 312/641] 10.222.0.27 () {58 vars in 1024 bytes} [Thu Oct 13 04:09:24 2022] POST /identity/v3/auth/tokens => generated 1161 bytes in 98 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Oct 13 04:09:25.049845 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c9864c9-b2b7-4def-960c-468d7119625c None None] REQUEST_METHOD: `GET` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:09:25.050160 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c9864c9-b2b7-4def-960c-468d7119625c None None] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:09:25.050422 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c9864c9-b2b7-4def-960c-468d7119625c None None] PATH_INFO: `/` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:09:25.052060 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 330/642] 10.222.0.27 () {54 vars in 927 bytes} [Thu Oct 13 04:09:25 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 13 04:09:25.058043 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db23f607-3ca9-46ae-a328-70537334639c None None] REQUEST_METHOD: `POST` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:09:25.058624 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db23f607-3ca9-46ae-a328-70537334639c None None] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:09:25.059044 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db23f607-3ca9-46ae-a328-70537334639c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:09:25.078350 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.auth.core [None req-db23f607-3ca9-46ae-a328-70537334639c None None] MFA Rules not processed for user `0e2d979563d6421995451a0e86bf3022`. Rule list: `[]` (Enabled: `True`). {{(pid=81408) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 13 04:09:25.079787 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.fernet_utils [None req-db23f607-3ca9-46ae-a328-70537334639c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81408) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:09:25.148552 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 313/643] 10.222.0.27 () {58 vars in 1024 bytes} [Thu Oct 13 04:09:25 2022] POST /identity/v3/auth/tokens => generated 1161 bytes in 91 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Oct 13 04:09:25.277483 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1c9864c9-b2b7-4def-960c-468d7119625c None None] Authenticating user token {{(pid=81409) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:09:25.330262 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG dbcounter [-] [81409] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81409) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:09:25.389976 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1c9864c9-b2b7-4def-960c-468d7119625c None None] Validating token access rules against request {{(pid=81409) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:09:25.427701 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-64bacc63-12e9-458e-99ba-876533242653 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81409) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:09:25.428324 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64bacc63-12e9-458e-99ba-876533242653 None admin] REQUEST_METHOD: `POST` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:09:25.428599 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64bacc63-12e9-458e-99ba-876533242653 None admin] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:09:25.428870 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64bacc63-12e9-458e-99ba-876533242653 None admin] PATH_INFO: `/v3/services` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:09:25.429537 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-64bacc63-12e9-458e-99ba-876533242653 None admin] RBAC: Authorizing `identity:create_service()` {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:09:25.432115 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-64bacc63-12e9-458e-99ba-876533242653 None admin] RBAC: Authorization granted {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:09:25.438177 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-db23f607-3ca9-46ae-a328-70537334639c None None] Authenticating user token {{(pid=81408) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:09:25.454934 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 331/644] 10.222.0.27 () {60 vars in 1141 bytes} [Thu Oct 13 04:09:25 2022] POST /identity/v3/services => generated 269 bytes in 178 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 13 04:09:25.456374 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9286018b-ddf0-4ef2-a828-4c3b13846aba None None] REQUEST_METHOD: `GET` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:09:25.456485 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9286018b-ddf0-4ef2-a828-4c3b13846aba None None] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:09:25.456708 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9286018b-ddf0-4ef2-a828-4c3b13846aba None None] PATH_INFO: `/` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:09:25.458184 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 332/645] 10.222.0.27 () {54 vars in 927 bytes} [Thu Oct 13 04:09:25 2022] GET /identity => generated 270 bytes in 2 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 13 04:09:25.463846 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3a20b9b-a0bb-4188-9598-62f1eed88b40 None None] REQUEST_METHOD: `POST` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:09:25.464090 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3a20b9b-a0bb-4188-9598-62f1eed88b40 None None] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:09:25.464395 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3a20b9b-a0bb-4188-9598-62f1eed88b40 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:09:25.487972 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.auth.core [None req-d3a20b9b-a0bb-4188-9598-62f1eed88b40 None None] MFA Rules not processed for user `0e2d979563d6421995451a0e86bf3022`. Rule list: `[]` (Enabled: `True`). {{(pid=81409) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 13 04:09:25.489544 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.fernet_utils [None req-d3a20b9b-a0bb-4188-9598-62f1eed88b40 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81409) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:09:25.579388 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-db23f607-3ca9-46ae-a328-70537334639c None None] Validating token access rules against request {{(pid=81408) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:09:25.588178 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG dbcounter [-] [81408] Writing DB stats keystone:SELECT=100 {{(pid=81408) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:09:25.591773 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 333/646] 10.222.0.27 () {58 vars in 1024 bytes} [Thu Oct 13 04:09:25 2022] POST /identity/v3/auth/tokens => generated 1271 bytes in 129 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Oct 13 04:09:25.623489 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-71e307cf-bbab-4a88-bbc5-2e22abb20622 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81408) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:09:25.624181 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71e307cf-bbab-4a88-bbc5-2e22abb20622 None admin] REQUEST_METHOD: `GET` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:09:25.624467 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71e307cf-bbab-4a88-bbc5-2e22abb20622 None admin] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:09:25.624731 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71e307cf-bbab-4a88-bbc5-2e22abb20622 None admin] PATH_INFO: `/v3/services/volumev3` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:09:25.625317 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-71e307cf-bbab-4a88-bbc5-2e22abb20622 None admin] RBAC: Authorizing `identity:get_service(service_id=volumev3)` {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:09:25.626785 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-71e307cf-bbab-4a88-bbc5-2e22abb20622 None admin] RBAC: Authorization granted {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:09:25.636805 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application [None req-71e307cf-bbab-4a88-bbc5-2e22abb20622 None admin] Could not find service: volumev3.: keystone.exception.ServiceNotFound: Could not find service: volumev3. Oct 13 04:09:25.636805 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 13 04:09:25.636805 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Oct 13 04:09:25.636805 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Oct 13 04:09:25.636805 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Oct 13 04:09:25.636805 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Oct 13 04:09:25.636805 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Oct 13 04:09:25.636805 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 13 04:09:25.636805 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: volumev3. Oct 13 04:09:25.636805 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application Oct 13 04:09:25.636805 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Oct 13 04:09:25.636805 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application Oct 13 04:09:25.636805 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 13 04:09:25.636805 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 13 04:09:25.636805 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 13 04:09:25.636805 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 13 04:09:25.636805 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 13 04:09:25.637851 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 13 04:09:25.637851 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 13 04:09:25.637851 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 13 04:09:25.637851 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 13 04:09:25.637851 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 13 04:09:25.637851 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 13 04:09:25.637851 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Oct 13 04:09:25.637851 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application return self._get_service(service_id) Oct 13 04:09:25.637851 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Oct 13 04:09:25.637851 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Oct 13 04:09:25.637851 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 13 04:09:25.637851 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 13 04:09:25.637851 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 13 04:09:25.637851 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 13 04:09:25.637851 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 13 04:09:25.637851 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application return self.get_or_create( Oct 13 04:09:25.638725 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 13 04:09:25.638725 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application with Lock( Oct 13 04:09:25.638725 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 13 04:09:25.638725 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application return self._enter() Oct 13 04:09:25.638725 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 13 04:09:25.638725 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 13 04:09:25.638725 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 13 04:09:25.638725 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application return self.creator() Oct 13 04:09:25.638725 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 13 04:09:25.638725 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application created_value = creator( Oct 13 04:09:25.638725 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Oct 13 04:09:25.638725 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 13 04:09:25.638725 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: volumev3. Oct 13 04:09:25.638725 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application Oct 13 04:09:25.638725 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 314/647] 10.222.0.27 () {56 vars in 1114 bytes} [Thu Oct 13 04:09:25 2022] GET /identity/v3/services/volumev3 => generated 89 bytes in 200 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Oct 13 04:09:25.643897 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d3a20b9b-a0bb-4188-9598-62f1eed88b40 None None] Authenticating user token {{(pid=81409) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:09:25.680304 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG dbcounter [-] [81409] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81409) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:09:25.755735 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d3a20b9b-a0bb-4188-9598-62f1eed88b40 None None] Validating token access rules against request {{(pid=81409) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:09:25.791930 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d4b8f003-04b5-457c-8246-8e96d6885e06 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81409) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:09:25.792542 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4b8f003-04b5-457c-8246-8e96d6885e06 None admin] REQUEST_METHOD: `GET` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:09:25.792814 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4b8f003-04b5-457c-8246-8e96d6885e06 None admin] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:09:25.793082 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4b8f003-04b5-457c-8246-8e96d6885e06 None admin] PATH_INFO: `/v3/services` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:09:25.793708 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d4b8f003-04b5-457c-8246-8e96d6885e06 None admin] RBAC: Adding query filter params (name=volumev3) {{(pid=81409) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 13 04:09:25.794007 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d4b8f003-04b5-457c-8246-8e96d6885e06 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:09:25.795454 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d4b8f003-04b5-457c-8246-8e96d6885e06 None admin] RBAC: Authorization granted {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:09:25.800583 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 334/648] 10.222.0.27 () {56 vars in 1114 bytes} [Thu Oct 13 04:09:25 2022] GET /identity/v3/services?name=volumev3 => generated 125 bytes in 157 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 13 04:09:25.807116 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-71e307cf-bbab-4a88-bbc5-2e22abb20622 None admin] Authenticating user token {{(pid=81408) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:09:25.865727 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d4b8f003-04b5-457c-8246-8e96d6885e06 None admin] Authenticating user token {{(pid=81409) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:09:25.917384 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-71e307cf-bbab-4a88-bbc5-2e22abb20622 None admin] Validating token access rules against request {{(pid=81408) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:09:25.960191 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5151a52c-4e60-49eb-86ea-27e1cae83f42 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81408) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:09:25.961628 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5151a52c-4e60-49eb-86ea-27e1cae83f42 None admin] REQUEST_METHOD: `GET` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:09:25.962099 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5151a52c-4e60-49eb-86ea-27e1cae83f42 None admin] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:09:25.962601 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5151a52c-4e60-49eb-86ea-27e1cae83f42 None admin] PATH_INFO: `/v3/services` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:09:25.963723 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5151a52c-4e60-49eb-86ea-27e1cae83f42 None admin] RBAC: Adding query filter params (type=volumev3) {{(pid=81408) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 13 04:09:25.964209 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5151a52c-4e60-49eb-86ea-27e1cae83f42 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:09:25.966783 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5151a52c-4e60-49eb-86ea-27e1cae83f42 None admin] RBAC: Authorization granted {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:09:25.968049 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG dbcounter [-] [81409] Writing DB stats keystone:SELECT=100 {{(pid=81409) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:09:25.976111 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 315/649] 10.222.0.27 () {56 vars in 1114 bytes} [Thu Oct 13 04:09:25 2022] GET /identity/v3/services?type=volumev3 => generated 125 bytes in 170 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 13 04:09:25.983591 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d4b8f003-04b5-457c-8246-8e96d6885e06 None admin] Validating token access rules against request {{(pid=81409) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:09:26.031471 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2ce6b499-8c4d-4fac-9580-3b1ca097659a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81409) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:09:26.032562 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ce6b499-8c4d-4fac-9580-3b1ca097659a None admin] REQUEST_METHOD: `GET` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:09:26.033106 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ce6b499-8c4d-4fac-9580-3b1ca097659a None admin] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:09:26.033609 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ce6b499-8c4d-4fac-9580-3b1ca097659a None admin] PATH_INFO: `/v3/roles/ResellerAdmin` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:09:26.047449 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2ce6b499-8c4d-4fac-9580-3b1ca097659a None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 13 04:09:26.052876 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application [None req-2ce6b499-8c4d-4fac-9580-3b1ca097659a None admin] Could not find role: ResellerAdmin.: keystone.exception.RoleNotFound: Could not find role: ResellerAdmin. Oct 13 04:09:26.052876 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 13 04:09:26.052876 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 13 04:09:26.052876 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 13 04:09:26.052876 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 13 04:09:26.052876 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 13 04:09:26.052876 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 13 04:09:26.052876 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 13 04:09:26.052876 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 13 04:09:26.052876 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 13 04:09:26.052876 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 13 04:09:26.052876 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 13 04:09:26.052876 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 13 04:09:26.052876 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 13 04:09:26.052876 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 13 04:09:26.052876 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 13 04:09:26.053900 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 13 04:09:26.053900 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 13 04:09:26.053900 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 13 04:09:26.053900 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 13 04:09:26.053900 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 13 04:09:26.053900 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 13 04:09:26.053900 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 13 04:09:26.053900 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 13 04:09:26.053900 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 13 04:09:26.053900 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application return self.get_or_create( Oct 13 04:09:26.053900 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 13 04:09:26.053900 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application with Lock( Oct 13 04:09:26.053900 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 13 04:09:26.053900 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application return self._enter() Oct 13 04:09:26.053900 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 13 04:09:26.053900 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 13 04:09:26.054794 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 13 04:09:26.054794 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application return self.creator() Oct 13 04:09:26.054794 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 13 04:09:26.054794 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application created_value = creator( Oct 13 04:09:26.054794 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 13 04:09:26.054794 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 13 04:09:26.054794 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 13 04:09:26.054794 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 13 04:09:26.054794 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 13 04:09:26.054794 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 13 04:09:26.054794 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: ResellerAdmin. Oct 13 04:09:26.054794 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application Oct 13 04:09:26.057932 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 335/650] 10.222.0.27 () {56 vars in 1120 bytes} [Thu Oct 13 04:09:25 2022] GET /identity/v3/roles/ResellerAdmin => generated 91 bytes in 193 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Oct 13 04:09:26.064231 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5151a52c-4e60-49eb-86ea-27e1cae83f42 None admin] Authenticating user token {{(pid=81408) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:09:26.105207 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG dbcounter [-] [81408] Writing DB stats keystone:SELECT=100 {{(pid=81408) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:09:26.179253 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5151a52c-4e60-49eb-86ea-27e1cae83f42 None admin] Validating token access rules against request {{(pid=81408) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:09:26.216420 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-866f1209-3d1a-43ce-b285-a6b90fc7e60a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81408) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:09:26.216945 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-866f1209-3d1a-43ce-b285-a6b90fc7e60a None admin] REQUEST_METHOD: `GET` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:09:26.217218 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-866f1209-3d1a-43ce-b285-a6b90fc7e60a None admin] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:09:26.217485 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-866f1209-3d1a-43ce-b285-a6b90fc7e60a None admin] PATH_INFO: `/v3/roles` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:09:26.218246 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-866f1209-3d1a-43ce-b285-a6b90fc7e60a None admin] RBAC: Adding query filter params (name=ResellerAdmin) {{(pid=81408) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 13 04:09:26.218533 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-866f1209-3d1a-43ce-b285-a6b90fc7e60a None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:09:26.219968 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-866f1209-3d1a-43ce-b285-a6b90fc7e60a None admin] RBAC: Authorization granted {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:09:26.230296 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 316/651] 10.222.0.27 () {56 vars in 1115 bytes} [Thu Oct 13 04:09:26 2022] GET /identity/v3/roles?name=ResellerAdmin => generated 338 bytes in 166 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 13 04:09:26.235746 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2ce6b499-8c4d-4fac-9580-3b1ca097659a None admin] Authenticating user token {{(pid=81409) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:09:26.346394 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2ce6b499-8c4d-4fac-9580-3b1ca097659a None admin] Validating token access rules against request {{(pid=81409) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:09:26.388270 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0fe7f369-4172-40df-a225-e9a1daaf91f8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81409) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:09:26.388905 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0fe7f369-4172-40df-a225-e9a1daaf91f8 None admin] REQUEST_METHOD: `GET` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:09:26.389178 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0fe7f369-4172-40df-a225-e9a1daaf91f8 None admin] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:09:26.389449 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0fe7f369-4172-40df-a225-e9a1daaf91f8 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:09:26.400237 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0fe7f369-4172-40df-a225-e9a1daaf91f8 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:09:26.401682 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0fe7f369-4172-40df-a225-e9a1daaf91f8 None admin] RBAC: Authorization granted {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:09:26.416809 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application [None req-0fe7f369-4172-40df-a225-e9a1daaf91f8 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 13 04:09:26.416809 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 13 04:09:26.416809 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 13 04:09:26.416809 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 13 04:09:26.416809 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 13 04:09:26.416809 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 13 04:09:26.416809 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 13 04:09:26.416809 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 13 04:09:26.416809 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 13 04:09:26.416809 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 13 04:09:26.416809 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 13 04:09:26.416809 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 13 04:09:26.416809 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 13 04:09:26.416809 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 13 04:09:26.416809 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 13 04:09:26.416809 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 13 04:09:26.417741 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 13 04:09:26.417741 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 13 04:09:26.417741 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 13 04:09:26.417741 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 13 04:09:26.417741 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 13 04:09:26.417741 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application return self.get_or_create( Oct 13 04:09:26.417741 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 13 04:09:26.417741 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application with Lock( Oct 13 04:09:26.417741 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 13 04:09:26.417741 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application return self._enter() Oct 13 04:09:26.417741 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 13 04:09:26.417741 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 13 04:09:26.417741 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 13 04:09:26.417741 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application return self.creator() Oct 13 04:09:26.417741 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 13 04:09:26.417741 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application created_value = creator( Oct 13 04:09:26.417741 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 13 04:09:26.418660 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 13 04:09:26.418660 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 13 04:09:26.418660 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application Oct 13 04:09:26.418660 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 336/652] 10.222.0.27 () {56 vars in 1108 bytes} [Thu Oct 13 04:09:26 2022] GET /identity/v3/domains/Default => generated 87 bytes in 183 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 13 04:09:26.425041 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-866f1209-3d1a-43ce-b285-a6b90fc7e60a None admin] Authenticating user token {{(pid=81408) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:09:26.528872 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG dbcounter [-] [81408] Writing DB stats keystone:SELECT=100 {{(pid=81408) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:09:26.535151 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-866f1209-3d1a-43ce-b285-a6b90fc7e60a None admin] Validating token access rules against request {{(pid=81408) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:09:26.570892 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-023b2929-8ee8-446e-a330-2ae0ca0987fe None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81408) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:09:26.571751 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-023b2929-8ee8-446e-a330-2ae0ca0987fe None admin] REQUEST_METHOD: `GET` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:09:26.572087 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-023b2929-8ee8-446e-a330-2ae0ca0987fe None admin] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:09:26.572358 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-023b2929-8ee8-446e-a330-2ae0ca0987fe None admin] PATH_INFO: `/v3/domains` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:09:26.572993 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-023b2929-8ee8-446e-a330-2ae0ca0987fe None admin] RBAC: Adding query filter params (name=Default) {{(pid=81408) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 13 04:09:26.573290 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-023b2929-8ee8-446e-a330-2ae0ca0987fe None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:09:26.574671 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-023b2929-8ee8-446e-a330-2ae0ca0987fe None admin] RBAC: Authorization granted {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:09:26.587288 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 317/653] 10.222.0.27 () {56 vars in 1109 bytes} [Thu Oct 13 04:09:26 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 162 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 13 04:09:26.592550 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0fe7f369-4172-40df-a225-e9a1daaf91f8 None admin] Authenticating user token {{(pid=81409) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:09:26.602682 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG dbcounter [-] [81409] Writing DB stats keystone:SELECT=100 {{(pid=81409) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:09:26.697511 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0fe7f369-4172-40df-a225-e9a1daaf91f8 None admin] Validating token access rules against request {{(pid=81409) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:09:26.737800 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-65a3cacd-933c-4ece-8a7f-ec92b81e96e6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81409) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:09:26.738495 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65a3cacd-933c-4ece-8a7f-ec92b81e96e6 None admin] REQUEST_METHOD: `GET` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:09:26.738770 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65a3cacd-933c-4ece-8a7f-ec92b81e96e6 None admin] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:09:26.739041 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65a3cacd-933c-4ece-8a7f-ec92b81e96e6 None admin] PATH_INFO: `/v3/users/glance-swift` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:09:26.748622 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-65a3cacd-933c-4ece-8a7f-ec92b81e96e6 None admin] RBAC: Authorizing `identity:get_user(user_id=glance-swift)` {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:09:26.750082 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-65a3cacd-933c-4ece-8a7f-ec92b81e96e6 None admin] RBAC: Authorization granted {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:09:26.761955 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application [None req-65a3cacd-933c-4ece-8a7f-ec92b81e96e6 None admin] Could not find user: glance-swift.: keystone.exception.UserNotFound: Could not find user: glance-swift. Oct 13 04:09:26.761955 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 13 04:09:26.761955 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 13 04:09:26.761955 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 13 04:09:26.761955 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 13 04:09:26.761955 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 13 04:09:26.761955 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 13 04:09:26.761955 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 13 04:09:26.761955 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 13 04:09:26.761955 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 13 04:09:26.761955 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 13 04:09:26.761955 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 13 04:09:26.761955 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Oct 13 04:09:26.761955 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application return self._get_user(user_id) Oct 13 04:09:26.761955 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Oct 13 04:09:26.761955 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Oct 13 04:09:26.762976 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 13 04:09:26.762976 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 13 04:09:26.762976 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Oct 13 04:09:26.762976 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 13 04:09:26.762976 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Oct 13 04:09:26.762976 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 13 04:09:26.762976 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 13 04:09:26.762976 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 13 04:09:26.762976 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 13 04:09:26.762976 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application return self.get_or_create( Oct 13 04:09:26.762976 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 13 04:09:26.762976 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application with Lock( Oct 13 04:09:26.762976 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 13 04:09:26.762976 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application return self._enter() Oct 13 04:09:26.762976 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 13 04:09:26.762976 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 13 04:09:26.762976 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 13 04:09:26.762976 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application return self.creator() Oct 13 04:09:26.763893 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 13 04:09:26.763893 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application created_value = creator( Oct 13 04:09:26.763893 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Oct 13 04:09:26.763893 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Oct 13 04:09:26.763893 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Oct 13 04:09:26.763893 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Oct 13 04:09:26.763893 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Oct 13 04:09:26.763893 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Oct 13 04:09:26.763893 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: glance-swift. Oct 13 04:09:26.763893 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application Oct 13 04:09:26.763893 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 337/654] 10.222.0.27 () {56 vars in 1117 bytes} [Thu Oct 13 04:09:26 2022] GET /identity/v3/users/glance-swift => generated 90 bytes in 171 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 13 04:09:26.770727 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-023b2929-8ee8-446e-a330-2ae0ca0987fe None admin] Authenticating user token {{(pid=81408) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:09:26.844141 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-657234a4-9487-4cd9-a317-1a0c0364be5e None None] REQUEST_METHOD: `GET` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:09:26.844316 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-657234a4-9487-4cd9-a317-1a0c0364be5e None None] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:09:26.844588 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-657234a4-9487-4cd9-a317-1a0c0364be5e None None] PATH_INFO: `/` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:09:26.846066 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 338/655] 10.222.0.27 () {54 vars in 927 bytes} [Thu Oct 13 04:09:26 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Oct 13 04:09:26.852813 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ba65d1d-750a-47ad-8c03-3b37189cc5c9 None None] REQUEST_METHOD: `POST` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:09:26.853294 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ba65d1d-750a-47ad-8c03-3b37189cc5c9 None None] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:09:26.853715 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ba65d1d-750a-47ad-8c03-3b37189cc5c9 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:09:26.872918 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.auth.core [None req-1ba65d1d-750a-47ad-8c03-3b37189cc5c9 None None] MFA Rules not processed for user `0e2d979563d6421995451a0e86bf3022`. Rule list: `[]` (Enabled: `True`). {{(pid=81409) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 13 04:09:26.874118 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.fernet_utils [None req-1ba65d1d-750a-47ad-8c03-3b37189cc5c9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81409) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:09:26.884953 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-023b2929-8ee8-446e-a330-2ae0ca0987fe None admin] Validating token access rules against request {{(pid=81408) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:09:26.923339 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9f5f2316-422e-4c12-b58a-b6d74e50f12b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81408) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:09:26.924262 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f5f2316-422e-4c12-b58a-b6d74e50f12b None admin] REQUEST_METHOD: `GET` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:09:26.924721 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f5f2316-422e-4c12-b58a-b6d74e50f12b None admin] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:09:26.925187 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f5f2316-422e-4c12-b58a-b6d74e50f12b None admin] PATH_INFO: `/v3/users` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:09:26.926114 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9f5f2316-422e-4c12-b58a-b6d74e50f12b None admin] RBAC: Adding query filter params (domain_id=default, name=glance-swift) {{(pid=81408) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 13 04:09:26.926698 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9f5f2316-422e-4c12-b58a-b6d74e50f12b None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:09:26.928486 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9f5f2316-422e-4c12-b58a-b6d74e50f12b None admin] RBAC: Authorization granted {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:09:26.945243 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG dbcounter [-] [81409] Writing DB stats keystone:SELECT=100 {{(pid=81409) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:09:26.948054 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 318/656] 10.222.0.27 () {56 vars in 1149 bytes} [Thu Oct 13 04:09:26 2022] GET /identity/v3/users?domain_id=default&name=glance-swift => generated 384 bytes in 178 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 13 04:09:26.953941 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9f5f2316-422e-4c12-b58a-b6d74e50f12b None admin] Authenticating user token {{(pid=81408) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:09:26.965747 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 339/657] 10.222.0.27 () {58 vars in 1024 bytes} [Thu Oct 13 04:09:26 2022] POST /identity/v3/auth/tokens => generated 1271 bytes in 114 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Oct 13 04:09:26.974636 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG dbcounter [-] [81408] Writing DB stats keystone:SELECT=100 {{(pid=81408) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:09:27.060136 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9f5f2316-422e-4c12-b58a-b6d74e50f12b None admin] Validating token access rules against request {{(pid=81408) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:09:27.095426 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-81d8138d-3708-4a44-9488-dd05a32a0652 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81408) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:09:27.096269 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81d8138d-3708-4a44-9488-dd05a32a0652 None admin] REQUEST_METHOD: `GET` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:09:27.096789 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81d8138d-3708-4a44-9488-dd05a32a0652 None admin] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:09:27.097228 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81d8138d-3708-4a44-9488-dd05a32a0652 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:09:27.108417 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-81d8138d-3708-4a44-9488-dd05a32a0652 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:09:27.110128 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-81d8138d-3708-4a44-9488-dd05a32a0652 None admin] RBAC: Authorization granted {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:09:27.125390 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application [None req-81d8138d-3708-4a44-9488-dd05a32a0652 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 13 04:09:27.125390 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 13 04:09:27.125390 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 13 04:09:27.125390 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 13 04:09:27.125390 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 13 04:09:27.125390 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 13 04:09:27.125390 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 13 04:09:27.125390 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 13 04:09:27.125390 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 13 04:09:27.125390 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 13 04:09:27.125390 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 13 04:09:27.125390 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 13 04:09:27.125390 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 13 04:09:27.125390 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 13 04:09:27.125390 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 13 04:09:27.125390 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 13 04:09:27.126364 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 13 04:09:27.126364 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 13 04:09:27.126364 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 13 04:09:27.126364 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 13 04:09:27.126364 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 13 04:09:27.126364 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application return self.get_or_create( Oct 13 04:09:27.126364 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 13 04:09:27.126364 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application with Lock( Oct 13 04:09:27.126364 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 13 04:09:27.126364 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application return self._enter() Oct 13 04:09:27.126364 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 13 04:09:27.126364 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 13 04:09:27.126364 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 13 04:09:27.126364 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application return self.creator() Oct 13 04:09:27.126364 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 13 04:09:27.126364 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application created_value = creator( Oct 13 04:09:27.126364 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 13 04:09:27.127258 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 13 04:09:27.127258 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 13 04:09:27.127258 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application Oct 13 04:09:27.129027 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 319/658] 10.222.0.27 () {56 vars in 1108 bytes} [Thu Oct 13 04:09:26 2022] GET /identity/v3/domains/Default => generated 87 bytes in 175 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Oct 13 04:09:27.134878 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1ba65d1d-750a-47ad-8c03-3b37189cc5c9 None None] Authenticating user token {{(pid=81409) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:09:27.240203 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1ba65d1d-750a-47ad-8c03-3b37189cc5c9 None None] Validating token access rules against request {{(pid=81409) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:09:27.255619 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-81d8138d-3708-4a44-9488-dd05a32a0652 None admin] Authenticating user token {{(pid=81408) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:09:27.279976 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-daddc82b-35b4-4a6c-b8c9-3d9b3a3a5ee3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81409) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:09:27.280778 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-daddc82b-35b4-4a6c-b8c9-3d9b3a3a5ee3 None admin] REQUEST_METHOD: `GET` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:09:27.281065 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-daddc82b-35b4-4a6c-b8c9-3d9b3a3a5ee3 None admin] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:09:27.281334 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-daddc82b-35b4-4a6c-b8c9-3d9b3a3a5ee3 None admin] PATH_INFO: `/v3/domains` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:09:27.281987 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-daddc82b-35b4-4a6c-b8c9-3d9b3a3a5ee3 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81409) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 13 04:09:27.282321 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-daddc82b-35b4-4a6c-b8c9-3d9b3a3a5ee3 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:09:27.284385 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-daddc82b-35b4-4a6c-b8c9-3d9b3a3a5ee3 None admin] RBAC: Authorization granted {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:09:27.299100 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 340/659] 10.222.0.27 () {56 vars in 1109 bytes} [Thu Oct 13 04:09:27 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 165 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 13 04:09:27.304549 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-daddc82b-35b4-4a6c-b8c9-3d9b3a3a5ee3 None admin] Authenticating user token {{(pid=81409) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:09:27.330598 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG dbcounter [-] [81408] Writing DB stats keystone:SELECT=100 {{(pid=81408) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:09:27.380280 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-81d8138d-3708-4a44-9488-dd05a32a0652 None admin] Validating token access rules against request {{(pid=81408) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:09:27.380884 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG dbcounter [-] [81409] Writing DB stats keystone:SELECT=100 {{(pid=81409) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:09:27.425667 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-daddc82b-35b4-4a6c-b8c9-3d9b3a3a5ee3 None admin] Validating token access rules against request {{(pid=81409) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:09:27.430630 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-054e9ff6-c06c-40c2-a04e-7f6f13484bc2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81408) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:09:27.431616 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-054e9ff6-c06c-40c2-a04e-7f6f13484bc2 None admin] REQUEST_METHOD: `POST` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:09:27.432045 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-054e9ff6-c06c-40c2-a04e-7f6f13484bc2 None admin] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:09:27.432460 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-054e9ff6-c06c-40c2-a04e-7f6f13484bc2 None admin] PATH_INFO: `/v3/services` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:09:27.433287 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-054e9ff6-c06c-40c2-a04e-7f6f13484bc2 None admin] RBAC: Authorizing `identity:create_service()` {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:09:27.435296 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-054e9ff6-c06c-40c2-a04e-7f6f13484bc2 None admin] RBAC: Authorization granted {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:09:27.458564 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 320/660] 10.222.0.27 () {60 vars in 1141 bytes} [Thu Oct 13 04:09:27 2022] POST /identity/v3/services => generated 241 bytes in 203 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 13 04:09:27.485602 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bf13e1e8-75be-41c4-8718-80646e855e6a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81409) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:09:27.486297 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf13e1e8-75be-41c4-8718-80646e855e6a None admin] REQUEST_METHOD: `GET` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:09:27.486620 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf13e1e8-75be-41c4-8718-80646e855e6a None admin] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:09:27.486892 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf13e1e8-75be-41c4-8718-80646e855e6a None admin] PATH_INFO: `/v3/projects/service` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:09:27.496074 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bf13e1e8-75be-41c4-8718-80646e855e6a None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:09:27.497569 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bf13e1e8-75be-41c4-8718-80646e855e6a None admin] RBAC: Authorization granted {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:09:27.511203 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application [None req-bf13e1e8-75be-41c4-8718-80646e855e6a None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Oct 13 04:09:27.511203 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 13 04:09:27.511203 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 13 04:09:27.511203 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 13 04:09:27.511203 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 13 04:09:27.511203 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 13 04:09:27.511203 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 13 04:09:27.511203 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 13 04:09:27.511203 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 13 04:09:27.511203 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 13 04:09:27.511203 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 13 04:09:27.511203 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 13 04:09:27.511203 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Oct 13 04:09:27.511203 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application return self._get_project(project_id) Oct 13 04:09:27.511203 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Oct 13 04:09:27.512174 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Oct 13 04:09:27.512174 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 13 04:09:27.512174 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 13 04:09:27.512174 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 13 04:09:27.512174 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 13 04:09:27.512174 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 13 04:09:27.512174 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application return self.get_or_create( Oct 13 04:09:27.512174 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 13 04:09:27.512174 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application with Lock( Oct 13 04:09:27.512174 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 13 04:09:27.512174 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application return self._enter() Oct 13 04:09:27.512174 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 13 04:09:27.512174 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 13 04:09:27.512174 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 13 04:09:27.512174 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application return self.creator() Oct 13 04:09:27.512174 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 13 04:09:27.512174 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application created_value = creator( Oct 13 04:09:27.513043 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Oct 13 04:09:27.513043 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Oct 13 04:09:27.513043 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Oct 13 04:09:27.513043 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Oct 13 04:09:27.513043 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Oct 13 04:09:27.513043 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Oct 13 04:09:27.513043 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Oct 13 04:09:27.513043 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application Oct 13 04:09:27.513043 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 341/661] 10.222.0.27 () {56 vars in 1111 bytes} [Thu Oct 13 04:09:27 2022] GET /identity/v3/projects/service => generated 88 bytes in 208 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Oct 13 04:09:27.519979 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-054e9ff6-c06c-40c2-a04e-7f6f13484bc2 None admin] Authenticating user token {{(pid=81408) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:09:27.638458 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-054e9ff6-c06c-40c2-a04e-7f6f13484bc2 None admin] Validating token access rules against request {{(pid=81408) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:09:27.667702 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG dbcounter [-] [81408] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81408) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:09:27.679236 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8be9a8ff-70dc-42d0-b9f1-4873e035c844 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81408) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:09:27.679854 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8be9a8ff-70dc-42d0-b9f1-4873e035c844 None admin] REQUEST_METHOD: `GET` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:09:27.680195 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8be9a8ff-70dc-42d0-b9f1-4873e035c844 None admin] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:09:27.680471 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8be9a8ff-70dc-42d0-b9f1-4873e035c844 None admin] PATH_INFO: `/v3/projects` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:09:27.681215 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8be9a8ff-70dc-42d0-b9f1-4873e035c844 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81408) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 13 04:09:27.681523 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8be9a8ff-70dc-42d0-b9f1-4873e035c844 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:09:27.683076 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8be9a8ff-70dc-42d0-b9f1-4873e035c844 None admin] RBAC: Authorization granted {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:09:27.697150 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 321/662] 10.222.0.27 () {56 vars in 1148 bytes} [Thu Oct 13 04:09:27 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 178 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 13 04:09:27.703472 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bf13e1e8-75be-41c4-8718-80646e855e6a None admin] Authenticating user token {{(pid=81409) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:09:27.818809 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bf13e1e8-75be-41c4-8718-80646e855e6a None admin] Validating token access rules against request {{(pid=81409) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:09:27.819382 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e53f86c-5fde-4079-92f0-83cb64a4b784 None None] REQUEST_METHOD: `GET` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:09:27.819663 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e53f86c-5fde-4079-92f0-83cb64a4b784 None None] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:09:27.819934 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e53f86c-5fde-4079-92f0-83cb64a4b784 None None] PATH_INFO: `/` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:09:27.822462 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 322/663] 10.222.0.27 () {54 vars in 927 bytes} [Thu Oct 13 04:09:27 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Oct 13 04:09:27.827632 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31b12394-8e30-42e7-9bb7-5518c5fb7fad None None] REQUEST_METHOD: `POST` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:09:27.828079 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31b12394-8e30-42e7-9bb7-5518c5fb7fad None None] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:09:27.828446 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31b12394-8e30-42e7-9bb7-5518c5fb7fad None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:09:27.833667 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG dbcounter [-] [81409] Writing DB stats keystone:SELECT=100 {{(pid=81409) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:09:27.849276 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.auth.core [None req-31b12394-8e30-42e7-9bb7-5518c5fb7fad None None] MFA Rules not processed for user `0e2d979563d6421995451a0e86bf3022`. Rule list: `[]` (Enabled: `True`). {{(pid=81408) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 13 04:09:27.851073 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.fernet_utils [None req-31b12394-8e30-42e7-9bb7-5518c5fb7fad None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81408) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:09:27.859321 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c2ef6248-f9a0-4f2e-ba10-07f9d293a036 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81409) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:09:27.859987 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2ef6248-f9a0-4f2e-ba10-07f9d293a036 None admin] REQUEST_METHOD: `PUT` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:09:27.860264 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2ef6248-f9a0-4f2e-ba10-07f9d293a036 None admin] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:09:27.860536 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2ef6248-f9a0-4f2e-ba10-07f9d293a036 None admin] PATH_INFO: `/v3/projects/dcec0431fb6849499d6aec35a234daa6/users/487b1605fd4143cfaa422a3a7a0344c7/roles/0b094166c73d4276a8e8eec82da7f5ff` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:09:27.862768 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c2ef6248-f9a0-4f2e-ba10-07f9d293a036 None admin] RBAC: Authorizing `identity:create_grant(project_id=dcec0431fb6849499d6aec35a234daa6, user_id=487b1605fd4143cfaa422a3a7a0344c7, role_id=0b094166c73d4276a8e8eec82da7f5ff)` {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:09:27.864208 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c2ef6248-f9a0-4f2e-ba10-07f9d293a036 None admin] RBAC: Authorization granted {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:09:27.881902 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 342/664] 10.222.0.27 () {58 vars in 1439 bytes} [Thu Oct 13 04:09:27 2022] PUT /identity/v3/projects/dcec0431fb6849499d6aec35a234daa6/users/487b1605fd4143cfaa422a3a7a0344c7/roles/0b094166c73d4276a8e8eec82da7f5ff => generated 0 bytes in 179 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 13 04:09:27.937051 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 323/665] 10.222.0.27 () {58 vars in 1024 bytes} [Thu Oct 13 04:09:27 2022] POST /identity/v3/auth/tokens => generated 1373 bytes in 110 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Oct 13 04:09:28.206541 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c2ef6248-f9a0-4f2e-ba10-07f9d293a036 None admin] Authenticating user token {{(pid=81409) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:09:28.316947 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c2ef6248-f9a0-4f2e-ba10-07f9d293a036 None admin] Validating token access rules against request {{(pid=81409) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:09:28.342874 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b5bfb13-bea5-406a-95e5-44cadac207ad None None] REQUEST_METHOD: `GET` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:09:28.343199 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b5bfb13-bea5-406a-95e5-44cadac207ad None None] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:09:28.343411 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b5bfb13-bea5-406a-95e5-44cadac207ad None None] PATH_INFO: `/` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:09:28.346322 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 324/666] 10.222.0.27 () {54 vars in 927 bytes} [Thu Oct 13 04:09:28 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 13 04:09:28.350836 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37b00da0-2858-4973-9eeb-d5e89169cd7b None None] REQUEST_METHOD: `POST` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:09:28.351109 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37b00da0-2858-4973-9eeb-d5e89169cd7b None None] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:09:28.351387 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37b00da0-2858-4973-9eeb-d5e89169cd7b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:09:28.354549 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8595bb1e-50b5-450e-afd3-bad6dde00914 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81409) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:09:28.355185 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8595bb1e-50b5-450e-afd3-bad6dde00914 None admin] REQUEST_METHOD: `GET` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:09:28.355455 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8595bb1e-50b5-450e-afd3-bad6dde00914 None admin] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:09:28.355720 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8595bb1e-50b5-450e-afd3-bad6dde00914 None admin] PATH_INFO: `/v3/services/compute_legacy` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:09:28.356304 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8595bb1e-50b5-450e-afd3-bad6dde00914 None admin] RBAC: Authorizing `identity:get_service(service_id=compute_legacy)` {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:09:28.357659 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8595bb1e-50b5-450e-afd3-bad6dde00914 None admin] RBAC: Authorization granted {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:09:28.372015 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application [None req-8595bb1e-50b5-450e-afd3-bad6dde00914 None admin] Could not find service: compute_legacy.: keystone.exception.ServiceNotFound: Could not find service: compute_legacy. Oct 13 04:09:28.372015 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 13 04:09:28.372015 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Oct 13 04:09:28.372015 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Oct 13 04:09:28.372015 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Oct 13 04:09:28.372015 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Oct 13 04:09:28.372015 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Oct 13 04:09:28.372015 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 13 04:09:28.372015 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: compute_legacy. Oct 13 04:09:28.372015 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application Oct 13 04:09:28.372015 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Oct 13 04:09:28.372015 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application Oct 13 04:09:28.372015 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 13 04:09:28.372015 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 13 04:09:28.372015 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 13 04:09:28.372015 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 13 04:09:28.373008 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 13 04:09:28.373008 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 13 04:09:28.373008 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 13 04:09:28.373008 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 13 04:09:28.373008 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 13 04:09:28.373008 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 13 04:09:28.373008 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 13 04:09:28.373008 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Oct 13 04:09:28.373008 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application return self._get_service(service_id) Oct 13 04:09:28.373008 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Oct 13 04:09:28.373008 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Oct 13 04:09:28.373008 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 13 04:09:28.373008 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 13 04:09:28.373008 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 13 04:09:28.373008 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 13 04:09:28.373008 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 13 04:09:28.373008 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application return self.get_or_create( Oct 13 04:09:28.373933 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 13 04:09:28.373933 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application with Lock( Oct 13 04:09:28.373933 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 13 04:09:28.373933 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application return self._enter() Oct 13 04:09:28.373933 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 13 04:09:28.373933 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 13 04:09:28.373933 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 13 04:09:28.373933 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application return self.creator() Oct 13 04:09:28.373933 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 13 04:09:28.373933 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application created_value = creator( Oct 13 04:09:28.373933 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Oct 13 04:09:28.373933 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 13 04:09:28.373933 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: compute_legacy. Oct 13 04:09:28.373933 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application Oct 13 04:09:28.373933 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 343/667] 10.222.0.27 () {56 vars in 1132 bytes} [Thu Oct 13 04:09:28 2022] GET /identity/v3/services/compute_legacy => generated 95 bytes in 167 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 13 04:09:28.379242 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.auth.core [None req-37b00da0-2858-4973-9eeb-d5e89169cd7b None None] MFA Rules not processed for user `0e2d979563d6421995451a0e86bf3022`. Rule list: `[]` (Enabled: `True`). {{(pid=81408) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 13 04:09:28.380638 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8595bb1e-50b5-450e-afd3-bad6dde00914 None admin] Authenticating user token {{(pid=81409) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:09:28.380845 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.fernet_utils [None req-37b00da0-2858-4973-9eeb-d5e89169cd7b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81408) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:09:28.422196 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG dbcounter [-] [81409] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81409) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:09:28.478363 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 325/668] 10.222.0.27 () {58 vars in 1024 bytes} [Thu Oct 13 04:09:28 2022] POST /identity/v3/auth/tokens => generated 1373 bytes in 128 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Oct 13 04:09:28.510222 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8595bb1e-50b5-450e-afd3-bad6dde00914 None admin] Validating token access rules against request {{(pid=81409) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:09:28.546467 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b89143c2-728d-422b-9f27-402da51d1a58 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81409) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:09:28.547076 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b89143c2-728d-422b-9f27-402da51d1a58 None admin] REQUEST_METHOD: `GET` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:09:28.547354 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b89143c2-728d-422b-9f27-402da51d1a58 None admin] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:09:28.547628 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b89143c2-728d-422b-9f27-402da51d1a58 None admin] PATH_INFO: `/v3/services` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:09:28.548255 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b89143c2-728d-422b-9f27-402da51d1a58 None admin] RBAC: Adding query filter params (name=compute_legacy) {{(pid=81409) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 13 04:09:28.548558 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b89143c2-728d-422b-9f27-402da51d1a58 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:09:28.549929 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b89143c2-728d-422b-9f27-402da51d1a58 None admin] RBAC: Authorization granted {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:09:28.555124 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 344/669] 10.222.0.27 () {56 vars in 1126 bytes} [Thu Oct 13 04:09:28 2022] GET /identity/v3/services?name=compute_legacy => generated 131 bytes in 175 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 13 04:09:28.563314 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-37b00da0-2858-4973-9eeb-d5e89169cd7b None None] Authenticating user token {{(pid=81408) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:09:28.611719 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG dbcounter [-] [81408] Writing DB stats keystone:SELECT=100 {{(pid=81408) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:09:28.671692 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-37b00da0-2858-4973-9eeb-d5e89169cd7b None None] Validating token access rules against request {{(pid=81408) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:09:28.707258 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4f9d8064-c0ba-4923-8523-d6a7686b0f2b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81408) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:09:28.707860 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f9d8064-c0ba-4923-8523-d6a7686b0f2b None admin] REQUEST_METHOD: `GET` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:09:28.708131 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f9d8064-c0ba-4923-8523-d6a7686b0f2b None admin] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:09:28.708398 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f9d8064-c0ba-4923-8523-d6a7686b0f2b None admin] PATH_INFO: `/v3/services` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:09:28.709020 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4f9d8064-c0ba-4923-8523-d6a7686b0f2b None admin] RBAC: Adding query filter params (type=compute_legacy) {{(pid=81408) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 13 04:09:28.709315 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4f9d8064-c0ba-4923-8523-d6a7686b0f2b None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:09:28.710716 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4f9d8064-c0ba-4923-8523-d6a7686b0f2b None admin] RBAC: Authorization granted {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:09:28.716880 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 326/670] 10.222.0.27 () {56 vars in 1126 bytes} [Thu Oct 13 04:09:28 2022] GET /identity/v3/services?type=compute_legacy => generated 386 bytes in 154 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 13 04:09:28.722514 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b89143c2-728d-422b-9f27-402da51d1a58 None admin] Authenticating user token {{(pid=81409) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:09:28.778481 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4f9d8064-c0ba-4923-8523-d6a7686b0f2b None admin] Authenticating user token {{(pid=81408) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:09:28.831855 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG dbcounter [-] [81409] Writing DB stats keystone:SELECT=100 {{(pid=81409) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:09:28.836145 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b89143c2-728d-422b-9f27-402da51d1a58 None admin] Validating token access rules against request {{(pid=81409) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:09:28.887953 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-46f6f9b2-e83e-405e-ba36-9023a84436d4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81409) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:09:28.888647 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46f6f9b2-e83e-405e-ba36-9023a84436d4 None admin] REQUEST_METHOD: `GET` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:09:28.889023 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4f9d8064-c0ba-4923-8523-d6a7686b0f2b None admin] Validating token access rules against request {{(pid=81408) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:09:28.889134 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46f6f9b2-e83e-405e-ba36-9023a84436d4 None admin] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:09:28.889360 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46f6f9b2-e83e-405e-ba36-9023a84436d4 None admin] PATH_INFO: `/v3/endpoints` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:09:28.890118 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-46f6f9b2-e83e-405e-ba36-9023a84436d4 None admin] RBAC: Adding query filter params (interface=public, service_id=8f4048733ab7474aad5942b7f62040cd, region_id=RegionOne) {{(pid=81409) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 13 04:09:28.890521 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-46f6f9b2-e83e-405e-ba36-9023a84436d4 None admin] RBAC: Authorizing `identity:list_endpoints()` {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:09:28.891955 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-46f6f9b2-e83e-405e-ba36-9023a84436d4 None admin] RBAC: Authorization granted {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:09:28.900890 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 345/671] 10.222.0.27 () {56 vars in 1251 bytes} [Thu Oct 13 04:09:28 2022] GET /identity/v3/endpoints?service_id=8f4048733ab7474aad5942b7f62040cd&interface=public®ion_id=RegionOne => generated 194 bytes in 179 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 13 04:09:28.901837 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG dbcounter [-] [81408] Writing DB stats keystone:SELECT=100 {{(pid=81408) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:09:28.906481 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-46f6f9b2-e83e-405e-ba36-9023a84436d4 None admin] Authenticating user token {{(pid=81409) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:09:28.950562 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-74f53c0b-1c66-4038-b2b3-cfadec9501ab None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81408) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:09:28.951179 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74f53c0b-1c66-4038-b2b3-cfadec9501ab None admin] REQUEST_METHOD: `POST` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:09:28.951458 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74f53c0b-1c66-4038-b2b3-cfadec9501ab None admin] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:09:28.951731 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74f53c0b-1c66-4038-b2b3-cfadec9501ab None admin] PATH_INFO: `/v3/services` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:09:28.952396 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-74f53c0b-1c66-4038-b2b3-cfadec9501ab None admin] RBAC: Authorizing `identity:create_service()` {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:09:28.954923 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-74f53c0b-1c66-4038-b2b3-cfadec9501ab None admin] RBAC: Authorization granted {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:09:28.975044 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 327/672] 10.222.0.27 () {60 vars in 1141 bytes} [Thu Oct 13 04:09:28 2022] POST /identity/v3/services => generated 251 bytes in 197 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 13 04:09:29.037476 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-46f6f9b2-e83e-405e-ba36-9023a84436d4 None admin] Validating token access rules against request {{(pid=81409) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:09:29.085387 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7cc34d6a-c7d3-40dd-8142-74da6977e1b0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81409) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:09:29.087029 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7cc34d6a-c7d3-40dd-8142-74da6977e1b0 None admin] REQUEST_METHOD: `GET` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:09:29.087674 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7cc34d6a-c7d3-40dd-8142-74da6977e1b0 None admin] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:09:29.088144 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7cc34d6a-c7d3-40dd-8142-74da6977e1b0 None admin] PATH_INFO: `/v3/services` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:09:29.088949 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7cc34d6a-c7d3-40dd-8142-74da6977e1b0 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:09:29.090583 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7cc34d6a-c7d3-40dd-8142-74da6977e1b0 None admin] RBAC: Authorization granted {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:09:29.098184 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 346/673] 10.222.0.27 () {56 vars in 1087 bytes} [Thu Oct 13 04:09:28 2022] GET /identity/v3/services => generated 1497 bytes in 192 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Oct 13 04:09:29.761260 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07784203-6e37-459c-a59d-180d144e7158 None None] REQUEST_METHOD: `GET` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:09:29.761785 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07784203-6e37-459c-a59d-180d144e7158 None None] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:09:29.761785 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07784203-6e37-459c-a59d-180d144e7158 None None] PATH_INFO: `/` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:09:29.763432 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 328/674] 10.222.0.27 () {54 vars in 927 bytes} [Thu Oct 13 04:09:29 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 13 04:09:29.768928 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47143470-69f8-4a15-b3b6-ecb0950c0898 None None] REQUEST_METHOD: `POST` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:09:29.769335 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47143470-69f8-4a15-b3b6-ecb0950c0898 None None] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:09:29.769505 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47143470-69f8-4a15-b3b6-ecb0950c0898 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:09:29.788264 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.auth.core [None req-47143470-69f8-4a15-b3b6-ecb0950c0898 None None] MFA Rules not processed for user `0e2d979563d6421995451a0e86bf3022`. Rule list: `[]` (Enabled: `True`). {{(pid=81409) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 13 04:09:29.790009 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.fernet_utils [None req-47143470-69f8-4a15-b3b6-ecb0950c0898 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81409) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:09:29.809416 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG dbcounter [-] [81409] Writing DB stats keystone:SELECT=100 {{(pid=81409) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:09:29.861615 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 347/675] 10.222.0.27 () {58 vars in 1024 bytes} [Thu Oct 13 04:09:29 2022] POST /identity/v3/auth/tokens => generated 1474 bytes in 93 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Oct 13 04:09:30.134675 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-07784203-6e37-459c-a59d-180d144e7158 None None] Authenticating user token {{(pid=81408) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:09:30.142334 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dcab50d1-d816-4d43-baa6-7f0ec4509d7b None None] REQUEST_METHOD: `GET` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:09:30.142911 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dcab50d1-d816-4d43-baa6-7f0ec4509d7b None None] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:09:30.143393 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dcab50d1-d816-4d43-baa6-7f0ec4509d7b None None] PATH_INFO: `/` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:09:30.145201 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 348/676] 10.222.0.27 () {54 vars in 927 bytes} [Thu Oct 13 04:09:30 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Oct 13 04:09:30.151121 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1f3065b-7247-4fa1-95d5-3ea5824275dc None None] REQUEST_METHOD: `POST` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:09:30.151626 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1f3065b-7247-4fa1-95d5-3ea5824275dc None None] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:09:30.152103 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1f3065b-7247-4fa1-95d5-3ea5824275dc None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:09:30.171132 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.auth.core [None req-a1f3065b-7247-4fa1-95d5-3ea5824275dc None None] MFA Rules not processed for user `0e2d979563d6421995451a0e86bf3022`. Rule list: `[]` (Enabled: `True`). {{(pid=81409) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 13 04:09:30.172818 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.fernet_utils [None req-a1f3065b-7247-4fa1-95d5-3ea5824275dc None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81409) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:09:30.246248 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 349/677] 10.222.0.27 () {58 vars in 1024 bytes} [Thu Oct 13 04:09:30 2022] POST /identity/v3/auth/tokens => generated 1474 bytes in 96 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Oct 13 04:09:30.259620 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-07784203-6e37-459c-a59d-180d144e7158 None None] Validating token access rules against request {{(pid=81408) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:09:30.296224 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7dc7f416-fb82-4796-a3e5-8c74986d0a4f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81408) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:09:30.297183 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7dc7f416-fb82-4796-a3e5-8c74986d0a4f None admin] REQUEST_METHOD: `GET` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:09:30.297652 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7dc7f416-fb82-4796-a3e5-8c74986d0a4f None admin] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:09:30.298101 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7dc7f416-fb82-4796-a3e5-8c74986d0a4f None admin] PATH_INFO: `/v3/services/object-store` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:09:30.298893 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7dc7f416-fb82-4796-a3e5-8c74986d0a4f None admin] RBAC: Authorizing `identity:get_service(service_id=object-store)` {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:09:30.300456 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7dc7f416-fb82-4796-a3e5-8c74986d0a4f None admin] RBAC: Authorization granted {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:09:30.310602 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application [None req-7dc7f416-fb82-4796-a3e5-8c74986d0a4f None admin] Could not find service: object-store.: keystone.exception.ServiceNotFound: Could not find service: object-store. Oct 13 04:09:30.310602 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 13 04:09:30.310602 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Oct 13 04:09:30.310602 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Oct 13 04:09:30.310602 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Oct 13 04:09:30.310602 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Oct 13 04:09:30.310602 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Oct 13 04:09:30.310602 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 13 04:09:30.310602 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: object-store. Oct 13 04:09:30.310602 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application Oct 13 04:09:30.310602 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Oct 13 04:09:30.310602 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application Oct 13 04:09:30.310602 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 13 04:09:30.310602 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 13 04:09:30.310602 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 13 04:09:30.310602 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 13 04:09:30.310602 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 13 04:09:30.311617 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 13 04:09:30.311617 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 13 04:09:30.311617 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 13 04:09:30.311617 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 13 04:09:30.311617 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 13 04:09:30.311617 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 13 04:09:30.311617 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Oct 13 04:09:30.311617 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application return self._get_service(service_id) Oct 13 04:09:30.311617 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Oct 13 04:09:30.311617 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Oct 13 04:09:30.311617 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 13 04:09:30.311617 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 13 04:09:30.311617 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 13 04:09:30.311617 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 13 04:09:30.311617 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 13 04:09:30.311617 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application return self.get_or_create( Oct 13 04:09:30.312467 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 13 04:09:30.312467 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application with Lock( Oct 13 04:09:30.312467 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 13 04:09:30.312467 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application return self._enter() Oct 13 04:09:30.312467 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 13 04:09:30.312467 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 13 04:09:30.312467 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 13 04:09:30.312467 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application return self.creator() Oct 13 04:09:30.312467 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 13 04:09:30.312467 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application created_value = creator( Oct 13 04:09:30.312467 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Oct 13 04:09:30.312467 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 13 04:09:30.312467 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: object-store. Oct 13 04:09:30.312467 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application Oct 13 04:09:30.314812 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 329/678] 10.222.0.27 () {56 vars in 1126 bytes} [Thu Oct 13 04:09:30 2022] GET /identity/v3/services/object-store => generated 93 bytes in 181 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 13 04:09:30.320985 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a1f3065b-7247-4fa1-95d5-3ea5824275dc None None] Authenticating user token {{(pid=81409) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:09:30.409701 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG dbcounter [-] [81409] Writing DB stats keystone:SELECT=100 {{(pid=81409) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:09:30.431280 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a1f3065b-7247-4fa1-95d5-3ea5824275dc None None] Validating token access rules against request {{(pid=81409) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:09:30.471202 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e9b1ac41-8026-442b-a908-d3093f379cfc None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81409) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:09:30.472081 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9b1ac41-8026-442b-a908-d3093f379cfc None admin] REQUEST_METHOD: `GET` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:09:30.472504 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9b1ac41-8026-442b-a908-d3093f379cfc None admin] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:09:30.472919 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9b1ac41-8026-442b-a908-d3093f379cfc None admin] PATH_INFO: `/v3/services` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:09:30.473689 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e9b1ac41-8026-442b-a908-d3093f379cfc None admin] RBAC: Adding query filter params (name=object-store) {{(pid=81409) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 13 04:09:30.474134 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e9b1ac41-8026-442b-a908-d3093f379cfc None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:09:30.476006 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e9b1ac41-8026-442b-a908-d3093f379cfc None admin] RBAC: Authorization granted {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:09:30.481445 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 350/679] 10.222.0.27 () {56 vars in 1122 bytes} [Thu Oct 13 04:09:30 2022] GET /identity/v3/services?name=object-store => generated 129 bytes in 161 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 13 04:09:30.486793 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7dc7f416-fb82-4796-a3e5-8c74986d0a4f None admin] Authenticating user token {{(pid=81408) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:09:30.522978 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG dbcounter [-] [81408] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81408) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:09:30.528117 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e9b1ac41-8026-442b-a908-d3093f379cfc None admin] Authenticating user token {{(pid=81409) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:09:30.612412 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7dc7f416-fb82-4796-a3e5-8c74986d0a4f None admin] Validating token access rules against request {{(pid=81408) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:09:30.641693 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e9b1ac41-8026-442b-a908-d3093f379cfc None admin] Validating token access rules against request {{(pid=81409) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:09:30.658555 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d0d8e4a7-7e7e-4a7b-ab47-92c5d2e519f5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81408) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:09:30.659437 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0d8e4a7-7e7e-4a7b-ab47-92c5d2e519f5 None admin] REQUEST_METHOD: `GET` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:09:30.659778 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0d8e4a7-7e7e-4a7b-ab47-92c5d2e519f5 None admin] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:09:30.659997 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0d8e4a7-7e7e-4a7b-ab47-92c5d2e519f5 None admin] PATH_INFO: `/v3/services` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:09:30.660635 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d0d8e4a7-7e7e-4a7b-ab47-92c5d2e519f5 None admin] RBAC: Adding query filter params (type=object-store) {{(pid=81408) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 13 04:09:30.660939 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d0d8e4a7-7e7e-4a7b-ab47-92c5d2e519f5 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:09:30.662966 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d0d8e4a7-7e7e-4a7b-ab47-92c5d2e519f5 None admin] RBAC: Authorization granted {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:09:30.670351 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 330/680] 10.222.0.27 () {56 vars in 1122 bytes} [Thu Oct 13 04:09:30 2022] GET /identity/v3/services?type=object-store => generated 356 bytes in 184 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 13 04:09:30.675353 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d0d8e4a7-7e7e-4a7b-ab47-92c5d2e519f5 None admin] Authenticating user token {{(pid=81408) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:09:30.687520 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-852aa110-6944-4097-9625-5d76c2b80eda None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81409) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:09:30.688441 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-852aa110-6944-4097-9625-5d76c2b80eda None admin] REQUEST_METHOD: `GET` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:09:30.689041 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-852aa110-6944-4097-9625-5d76c2b80eda None admin] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:09:30.689501 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-852aa110-6944-4097-9625-5d76c2b80eda None admin] PATH_INFO: `/v3/roles/ResellerAdmin` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:09:30.701757 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-852aa110-6944-4097-9625-5d76c2b80eda None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 13 04:09:30.707293 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application [None req-852aa110-6944-4097-9625-5d76c2b80eda None admin] Could not find role: ResellerAdmin.: keystone.exception.RoleNotFound: Could not find role: ResellerAdmin. Oct 13 04:09:30.707293 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 13 04:09:30.707293 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 13 04:09:30.707293 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 13 04:09:30.707293 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 13 04:09:30.707293 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 13 04:09:30.707293 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 13 04:09:30.707293 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 13 04:09:30.707293 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 13 04:09:30.707293 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 13 04:09:30.707293 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 13 04:09:30.707293 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 13 04:09:30.707293 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 13 04:09:30.707293 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 13 04:09:30.707293 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 13 04:09:30.707293 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 13 04:09:30.708192 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 13 04:09:30.708192 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 13 04:09:30.708192 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 13 04:09:30.708192 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 13 04:09:30.708192 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 13 04:09:30.708192 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 13 04:09:30.708192 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 13 04:09:30.708192 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 13 04:09:30.708192 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 13 04:09:30.708192 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application return self.get_or_create( Oct 13 04:09:30.708192 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 13 04:09:30.708192 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application with Lock( Oct 13 04:09:30.708192 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 13 04:09:30.708192 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application return self._enter() Oct 13 04:09:30.708192 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 13 04:09:30.708192 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 13 04:09:30.709037 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 13 04:09:30.709037 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application return self.creator() Oct 13 04:09:30.709037 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 13 04:09:30.709037 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application created_value = creator( Oct 13 04:09:30.709037 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 13 04:09:30.709037 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 13 04:09:30.709037 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 13 04:09:30.709037 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 13 04:09:30.709037 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 13 04:09:30.709037 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 13 04:09:30.709037 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: ResellerAdmin. Oct 13 04:09:30.709037 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application Oct 13 04:09:30.716740 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 351/681] 10.222.0.27 () {56 vars in 1120 bytes} [Thu Oct 13 04:09:30 2022] GET /identity/v3/roles/ResellerAdmin => generated 91 bytes in 184 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Oct 13 04:09:30.718174 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-852aa110-6944-4097-9625-5d76c2b80eda None admin] Authenticating user token {{(pid=81409) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:09:30.732026 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG dbcounter [-] [81409] Writing DB stats keystone:SELECT=100 {{(pid=81409) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:09:30.786695 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG dbcounter [-] [81408] Writing DB stats keystone:SELECT=100 {{(pid=81408) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:09:30.791167 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d0d8e4a7-7e7e-4a7b-ab47-92c5d2e519f5 None admin] Validating token access rules against request {{(pid=81408) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:09:30.827398 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f3f41d11-44e5-461a-b5c4-1cfd045c137d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81408) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:09:30.828030 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3f41d11-44e5-461a-b5c4-1cfd045c137d None admin] REQUEST_METHOD: `GET` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:09:30.828309 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3f41d11-44e5-461a-b5c4-1cfd045c137d None admin] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:09:30.828564 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3f41d11-44e5-461a-b5c4-1cfd045c137d None admin] PATH_INFO: `/v3/endpoints` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:09:30.829254 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f3f41d11-44e5-461a-b5c4-1cfd045c137d None admin] RBAC: Adding query filter params (interface=public, service_id=b505f599e7d44052a99c9638280cdad3, region_id=RegionOne) {{(pid=81408) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 13 04:09:30.829559 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f3f41d11-44e5-461a-b5c4-1cfd045c137d None admin] RBAC: Authorizing `identity:list_endpoints()` {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:09:30.830978 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f3f41d11-44e5-461a-b5c4-1cfd045c137d None admin] RBAC: Authorization granted {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:09:30.841174 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 331/682] 10.222.0.27 () {56 vars in 1251 bytes} [Thu Oct 13 04:09:30 2022] GET /identity/v3/endpoints?service_id=b505f599e7d44052a99c9638280cdad3&interface=public®ion_id=RegionOne => generated 194 bytes in 161 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 13 04:09:30.842022 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-852aa110-6944-4097-9625-5d76c2b80eda None admin] Validating token access rules against request {{(pid=81409) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:09:30.842503 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f3f41d11-44e5-461a-b5c4-1cfd045c137d None admin] Authenticating user token {{(pid=81408) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:09:30.883091 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5efb7584-a26c-48d6-b750-47c449165023 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81409) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:09:30.884545 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5efb7584-a26c-48d6-b750-47c449165023 None admin] REQUEST_METHOD: `GET` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:09:30.885022 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5efb7584-a26c-48d6-b750-47c449165023 None admin] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:09:30.885510 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5efb7584-a26c-48d6-b750-47c449165023 None admin] PATH_INFO: `/v3/roles` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:09:30.886497 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5efb7584-a26c-48d6-b750-47c449165023 None admin] RBAC: Adding query filter params (name=ResellerAdmin) {{(pid=81409) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 13 04:09:30.886996 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5efb7584-a26c-48d6-b750-47c449165023 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:09:30.888785 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5efb7584-a26c-48d6-b750-47c449165023 None admin] RBAC: Authorization granted {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:09:30.903954 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 352/683] 10.222.0.27 () {56 vars in 1115 bytes} [Thu Oct 13 04:09:30 2022] GET /identity/v3/roles?name=ResellerAdmin => generated 338 bytes in 186 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 13 04:09:30.905678 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5efb7584-a26c-48d6-b750-47c449165023 None admin] Authenticating user token {{(pid=81409) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:09:30.946360 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f3f41d11-44e5-461a-b5c4-1cfd045c137d None admin] Validating token access rules against request {{(pid=81408) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:09:30.982171 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-07c06067-386e-4140-a1c8-6a7df194a0c1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81408) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:09:30.982821 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07c06067-386e-4140-a1c8-6a7df194a0c1 None admin] REQUEST_METHOD: `GET` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:09:30.983099 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07c06067-386e-4140-a1c8-6a7df194a0c1 None admin] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:09:30.983369 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07c06067-386e-4140-a1c8-6a7df194a0c1 None admin] PATH_INFO: `/v3/services` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:09:30.983975 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-07c06067-386e-4140-a1c8-6a7df194a0c1 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:09:30.985350 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-07c06067-386e-4140-a1c8-6a7df194a0c1 None admin] RBAC: Authorization granted {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:09:30.986339 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG dbcounter [-] [81409] Writing DB stats keystone:SELECT=100 {{(pid=81409) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:09:31.002364 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 332/684] 10.222.0.27 () {56 vars in 1087 bytes} [Thu Oct 13 04:09:30 2022] GET /identity/v3/services => generated 1497 bytes in 151 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 13 04:09:31.027593 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5efb7584-a26c-48d6-b750-47c449165023 None admin] Validating token access rules against request {{(pid=81409) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:09:31.069118 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d7d58e5b-0beb-4220-a20b-c1a870824ad9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81409) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:09:31.070181 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7d58e5b-0beb-4220-a20b-c1a870824ad9 None admin] REQUEST_METHOD: `GET` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:09:31.070859 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7d58e5b-0beb-4220-a20b-c1a870824ad9 None admin] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:09:31.071459 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7d58e5b-0beb-4220-a20b-c1a870824ad9 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:09:31.084517 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d7d58e5b-0beb-4220-a20b-c1a870824ad9 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:09:31.087145 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d7d58e5b-0beb-4220-a20b-c1a870824ad9 None admin] RBAC: Authorization granted {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:09:31.104750 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application [None req-d7d58e5b-0beb-4220-a20b-c1a870824ad9 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 13 04:09:31.104750 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 13 04:09:31.104750 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 13 04:09:31.104750 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 13 04:09:31.104750 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 13 04:09:31.104750 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 13 04:09:31.104750 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 13 04:09:31.104750 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 13 04:09:31.104750 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 13 04:09:31.104750 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 13 04:09:31.104750 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 13 04:09:31.104750 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 13 04:09:31.104750 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 13 04:09:31.104750 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 13 04:09:31.104750 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 13 04:09:31.104750 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 13 04:09:31.105812 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 13 04:09:31.105812 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 13 04:09:31.105812 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 13 04:09:31.105812 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 13 04:09:31.105812 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 13 04:09:31.105812 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application return self.get_or_create( Oct 13 04:09:31.105812 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 13 04:09:31.105812 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application with Lock( Oct 13 04:09:31.105812 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 13 04:09:31.105812 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application return self._enter() Oct 13 04:09:31.105812 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 13 04:09:31.105812 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 13 04:09:31.105812 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 13 04:09:31.105812 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application return self.creator() Oct 13 04:09:31.105812 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 13 04:09:31.105812 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application created_value = creator( Oct 13 04:09:31.105812 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 13 04:09:31.106712 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 13 04:09:31.106712 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 13 04:09:31.106712 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application Oct 13 04:09:31.113633 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 353/685] 10.222.0.27 () {56 vars in 1108 bytes} [Thu Oct 13 04:09:30 2022] GET /identity/v3/domains/Default => generated 87 bytes in 203 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Oct 13 04:09:31.115510 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-07c06067-386e-4140-a1c8-6a7df194a0c1 None admin] Authenticating user token {{(pid=81408) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:09:31.152363 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG dbcounter [-] [81408] Writing DB stats keystone:SELECT=100 {{(pid=81408) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:09:31.235342 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-07c06067-386e-4140-a1c8-6a7df194a0c1 None admin] Validating token access rules against request {{(pid=81408) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:09:31.246135 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ffe4cc6d-265e-4c04-a747-7884745d293e None None] REQUEST_METHOD: `GET` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:09:31.246505 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ffe4cc6d-265e-4c04-a747-7884745d293e None None] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:09:31.246717 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ffe4cc6d-265e-4c04-a747-7884745d293e None None] PATH_INFO: `/` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:09:31.248272 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 354/686] 10.222.0.27 () {54 vars in 927 bytes} [Thu Oct 13 04:09:31 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Oct 13 04:09:31.254256 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36d27245-2a81-4a55-a812-f6427fd88962 None None] REQUEST_METHOD: `POST` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:09:31.254873 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36d27245-2a81-4a55-a812-f6427fd88962 None None] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:09:31.255346 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36d27245-2a81-4a55-a812-f6427fd88962 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:09:31.275863 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.auth.core [None req-36d27245-2a81-4a55-a812-f6427fd88962 None None] MFA Rules not processed for user `0e2d979563d6421995451a0e86bf3022`. Rule list: `[]` (Enabled: `True`). {{(pid=81409) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 13 04:09:31.277220 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.fernet_utils [None req-36d27245-2a81-4a55-a812-f6427fd88962 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81409) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:09:31.292485 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a82ad89f-ba5a-43f6-adcf-d3de70b65dfd None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81408) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:09:31.293331 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a82ad89f-ba5a-43f6-adcf-d3de70b65dfd None admin] REQUEST_METHOD: `GET` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:09:31.294029 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a82ad89f-ba5a-43f6-adcf-d3de70b65dfd None admin] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:09:31.294504 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a82ad89f-ba5a-43f6-adcf-d3de70b65dfd None admin] PATH_INFO: `/v3/domains` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:09:31.295730 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a82ad89f-ba5a-43f6-adcf-d3de70b65dfd None admin] RBAC: Adding query filter params (name=Default) {{(pid=81408) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 13 04:09:31.296410 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a82ad89f-ba5a-43f6-adcf-d3de70b65dfd None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:09:31.298184 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a82ad89f-ba5a-43f6-adcf-d3de70b65dfd None admin] RBAC: Authorization granted {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:09:31.316191 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 333/687] 10.222.0.27 () {56 vars in 1109 bytes} [Thu Oct 13 04:09:31 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 197 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 13 04:09:31.317770 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a82ad89f-ba5a-43f6-adcf-d3de70b65dfd None admin] Authenticating user token {{(pid=81408) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:09:31.360306 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 355/688] 10.222.0.27 () {58 vars in 1024 bytes} [Thu Oct 13 04:09:31 2022] POST /identity/v3/auth/tokens => generated 1474 bytes in 107 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Oct 13 04:09:31.369038 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4fbc6d2-59e6-4f5c-8831-491510ce7315 None None] REQUEST_METHOD: `GET` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:09:31.369315 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4fbc6d2-59e6-4f5c-8831-491510ce7315 None None] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:09:31.369875 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4fbc6d2-59e6-4f5c-8831-491510ce7315 None None] PATH_INFO: `/` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:09:31.371192 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 356/689] 10.222.0.27 () {54 vars in 927 bytes} [Thu Oct 13 04:09:31 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 13 04:09:31.380022 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca3f43a4-7d44-4139-90ce-451fd9a21b9b None None] REQUEST_METHOD: `POST` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:09:31.380617 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca3f43a4-7d44-4139-90ce-451fd9a21b9b None None] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:09:31.381116 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca3f43a4-7d44-4139-90ce-451fd9a21b9b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:09:31.405237 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.auth.core [None req-ca3f43a4-7d44-4139-90ce-451fd9a21b9b None None] MFA Rules not processed for user `0e2d979563d6421995451a0e86bf3022`. Rule list: `[]` (Enabled: `True`). {{(pid=81409) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 13 04:09:31.406884 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.fernet_utils [None req-ca3f43a4-7d44-4139-90ce-451fd9a21b9b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81409) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:09:31.433268 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG dbcounter [-] [81408] Writing DB stats keystone:SELECT=100 {{(pid=81408) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:09:31.454241 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a82ad89f-ba5a-43f6-adcf-d3de70b65dfd None admin] Validating token access rules against request {{(pid=81408) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:09:31.468249 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG dbcounter [-] [81409] Writing DB stats keystone:SELECT=100 {{(pid=81409) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:09:31.500449 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 357/690] 10.222.0.27 () {58 vars in 1024 bytes} [Thu Oct 13 04:09:31 2022] POST /identity/v3/auth/tokens => generated 1474 bytes in 116 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Oct 13 04:09:31.504250 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e8e00b88-970a-4017-b203-9b7176773687 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81408) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:09:31.504887 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8e00b88-970a-4017-b203-9b7176773687 None admin] REQUEST_METHOD: `GET` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:09:31.505166 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8e00b88-970a-4017-b203-9b7176773687 None admin] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:09:31.505436 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8e00b88-970a-4017-b203-9b7176773687 None admin] PATH_INFO: `/v3/users/glance-swift` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:09:31.514226 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e8e00b88-970a-4017-b203-9b7176773687 None admin] RBAC: Authorizing `identity:get_user(user_id=glance-swift)` {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:09:31.515689 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e8e00b88-970a-4017-b203-9b7176773687 None admin] RBAC: Authorization granted {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:09:31.527842 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application [None req-e8e00b88-970a-4017-b203-9b7176773687 None admin] Could not find user: glance-swift.: keystone.exception.UserNotFound: Could not find user: glance-swift. Oct 13 04:09:31.527842 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 13 04:09:31.527842 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 13 04:09:31.527842 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 13 04:09:31.527842 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 13 04:09:31.527842 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 13 04:09:31.527842 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 13 04:09:31.527842 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 13 04:09:31.527842 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 13 04:09:31.527842 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 13 04:09:31.527842 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 13 04:09:31.527842 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 13 04:09:31.527842 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Oct 13 04:09:31.527842 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application return self._get_user(user_id) Oct 13 04:09:31.527842 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Oct 13 04:09:31.527842 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Oct 13 04:09:31.529177 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 13 04:09:31.529177 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 13 04:09:31.529177 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Oct 13 04:09:31.529177 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 13 04:09:31.529177 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Oct 13 04:09:31.529177 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 13 04:09:31.529177 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 13 04:09:31.529177 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 13 04:09:31.529177 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 13 04:09:31.529177 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application return self.get_or_create( Oct 13 04:09:31.529177 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 13 04:09:31.529177 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application with Lock( Oct 13 04:09:31.529177 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 13 04:09:31.529177 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application return self._enter() Oct 13 04:09:31.529177 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 13 04:09:31.529177 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 13 04:09:31.529177 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 13 04:09:31.529177 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application return self.creator() Oct 13 04:09:31.530519 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 13 04:09:31.530519 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application created_value = creator( Oct 13 04:09:31.530519 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Oct 13 04:09:31.530519 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Oct 13 04:09:31.530519 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Oct 13 04:09:31.530519 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Oct 13 04:09:31.530519 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Oct 13 04:09:31.530519 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Oct 13 04:09:31.530519 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: glance-swift. Oct 13 04:09:31.530519 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application Oct 13 04:09:31.530519 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 334/691] 10.222.0.27 () {56 vars in 1117 bytes} [Thu Oct 13 04:09:31 2022] GET /identity/v3/users/glance-swift => generated 90 bytes in 212 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Oct 13 04:09:31.536091 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ca3f43a4-7d44-4139-90ce-451fd9a21b9b None None] Authenticating user token {{(pid=81409) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:09:31.633640 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e8e00b88-970a-4017-b203-9b7176773687 None admin] Authenticating user token {{(pid=81408) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:09:31.642922 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ca3f43a4-7d44-4139-90ce-451fd9a21b9b None None] Validating token access rules against request {{(pid=81409) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:09:31.679546 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0f1b1800-09d9-4299-9357-57456e2deda0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81409) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:09:31.680733 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f1b1800-09d9-4299-9357-57456e2deda0 None admin] REQUEST_METHOD: `GET` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:09:31.681008 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f1b1800-09d9-4299-9357-57456e2deda0 None admin] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:09:31.681257 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f1b1800-09d9-4299-9357-57456e2deda0 None admin] PATH_INFO: `/v3/users` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:09:31.681987 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0f1b1800-09d9-4299-9357-57456e2deda0 None admin] RBAC: Adding query filter params (domain_id=default, name=glance-swift) {{(pid=81409) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 13 04:09:31.682315 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0f1b1800-09d9-4299-9357-57456e2deda0 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:09:31.683762 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0f1b1800-09d9-4299-9357-57456e2deda0 None admin] RBAC: Authorization granted {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:09:31.701644 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 358/692] 10.222.0.27 () {56 vars in 1149 bytes} [Thu Oct 13 04:09:31 2022] GET /identity/v3/users?domain_id=default&name=glance-swift => generated 384 bytes in 166 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 13 04:09:31.707863 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0f1b1800-09d9-4299-9357-57456e2deda0 None admin] Authenticating user token {{(pid=81409) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:09:31.762111 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e8e00b88-970a-4017-b203-9b7176773687 None admin] Validating token access rules against request {{(pid=81408) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:09:31.764290 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG dbcounter [-] [81409] Writing DB stats keystone:SELECT=100 {{(pid=81409) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:09:31.806114 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e5a05b0d-9f76-4c32-9800-eb36f0e1b515 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81408) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:09:31.806814 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e5a05b0d-9f76-4c32-9800-eb36f0e1b515 None admin] REQUEST_METHOD: `GET` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:09:31.807055 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e5a05b0d-9f76-4c32-9800-eb36f0e1b515 None admin] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:09:31.807649 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e5a05b0d-9f76-4c32-9800-eb36f0e1b515 None admin] PATH_INFO: `/v3/services/volumev3` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:09:31.808521 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e5a05b0d-9f76-4c32-9800-eb36f0e1b515 None admin] RBAC: Authorizing `identity:get_service(service_id=volumev3)` {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:09:31.810064 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e5a05b0d-9f76-4c32-9800-eb36f0e1b515 None admin] RBAC: Authorization granted {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:09:31.819921 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application [None req-e5a05b0d-9f76-4c32-9800-eb36f0e1b515 None admin] Could not find service: volumev3.: keystone.exception.ServiceNotFound: Could not find service: volumev3. Oct 13 04:09:31.819921 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 13 04:09:31.819921 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Oct 13 04:09:31.819921 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Oct 13 04:09:31.819921 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Oct 13 04:09:31.819921 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Oct 13 04:09:31.819921 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Oct 13 04:09:31.819921 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 13 04:09:31.819921 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: volumev3. Oct 13 04:09:31.819921 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application Oct 13 04:09:31.819921 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Oct 13 04:09:31.819921 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application Oct 13 04:09:31.819921 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 13 04:09:31.819921 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 13 04:09:31.819921 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 13 04:09:31.819921 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 13 04:09:31.819921 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 13 04:09:31.821710 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 13 04:09:31.821710 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 13 04:09:31.821710 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 13 04:09:31.821710 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 13 04:09:31.821710 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 13 04:09:31.821710 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 13 04:09:31.821710 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Oct 13 04:09:31.821710 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application return self._get_service(service_id) Oct 13 04:09:31.821710 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Oct 13 04:09:31.821710 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Oct 13 04:09:31.821710 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 13 04:09:31.821710 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 13 04:09:31.821710 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 13 04:09:31.821710 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 13 04:09:31.821710 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 13 04:09:31.821710 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application return self.get_or_create( Oct 13 04:09:31.822557 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 13 04:09:31.822557 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application with Lock( Oct 13 04:09:31.822557 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 13 04:09:31.822557 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application return self._enter() Oct 13 04:09:31.822557 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 13 04:09:31.822557 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 13 04:09:31.822557 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 13 04:09:31.822557 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application return self.creator() Oct 13 04:09:31.822557 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 13 04:09:31.822557 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application created_value = creator( Oct 13 04:09:31.822557 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Oct 13 04:09:31.822557 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 13 04:09:31.822557 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: volumev3. Oct 13 04:09:31.822557 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application Oct 13 04:09:31.822557 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 335/693] 10.222.0.27 () {56 vars in 1114 bytes} [Thu Oct 13 04:09:31 2022] GET /identity/v3/services/volumev3 => generated 89 bytes in 189 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Oct 13 04:09:31.823343 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e5a05b0d-9f76-4c32-9800-eb36f0e1b515 None admin] Authenticating user token {{(pid=81408) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:09:31.835201 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0f1b1800-09d9-4299-9357-57456e2deda0 None admin] Validating token access rules against request {{(pid=81409) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:09:31.841324 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG dbcounter [-] [81408] Writing DB stats keystone:SELECT=100 {{(pid=81408) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:09:31.874857 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-de8dfe26-e2e1-4fe0-9394-7e7baf7931d4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81409) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:09:31.875540 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de8dfe26-e2e1-4fe0-9394-7e7baf7931d4 None admin] REQUEST_METHOD: `GET` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:09:31.875732 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de8dfe26-e2e1-4fe0-9394-7e7baf7931d4 None admin] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:09:31.876003 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de8dfe26-e2e1-4fe0-9394-7e7baf7931d4 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:09:31.934001 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e5a05b0d-9f76-4c32-9800-eb36f0e1b515 None admin] Validating token access rules against request {{(pid=81408) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:09:31.949550 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-de8dfe26-e2e1-4fe0-9394-7e7baf7931d4 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:09:31.951402 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-de8dfe26-e2e1-4fe0-9394-7e7baf7931d4 None admin] RBAC: Authorization granted {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:09:31.971671 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b5a731d4-5929-442d-9b3d-e4f01381b237 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81408) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:09:31.972379 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5a731d4-5929-442d-9b3d-e4f01381b237 None admin] REQUEST_METHOD: `GET` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:09:31.972649 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5a731d4-5929-442d-9b3d-e4f01381b237 None admin] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:09:31.973234 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5a731d4-5929-442d-9b3d-e4f01381b237 None admin] PATH_INFO: `/v3/services/compute_legacy` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:09:31.973491 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b5a731d4-5929-442d-9b3d-e4f01381b237 None admin] RBAC: Authorizing `identity:get_service(service_id=compute_legacy)` {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:09:31.974876 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b5a731d4-5929-442d-9b3d-e4f01381b237 None admin] RBAC: Authorization granted {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:09:31.984648 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application [None req-b5a731d4-5929-442d-9b3d-e4f01381b237 None admin] Could not find service: compute_legacy.: keystone.exception.ServiceNotFound: Could not find service: compute_legacy. Oct 13 04:09:31.984648 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 13 04:09:31.984648 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Oct 13 04:09:31.984648 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Oct 13 04:09:31.984648 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Oct 13 04:09:31.984648 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Oct 13 04:09:31.984648 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Oct 13 04:09:31.984648 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 13 04:09:31.984648 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: compute_legacy. Oct 13 04:09:31.984648 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application Oct 13 04:09:31.984648 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Oct 13 04:09:31.984648 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application Oct 13 04:09:31.984648 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 13 04:09:31.984648 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 13 04:09:31.984648 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 13 04:09:31.984648 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 13 04:09:31.985704 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 13 04:09:31.985704 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 13 04:09:31.985704 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 13 04:09:31.985704 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 13 04:09:31.985704 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 13 04:09:31.985704 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 13 04:09:31.985704 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 13 04:09:31.985704 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Oct 13 04:09:31.985704 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application return self._get_service(service_id) Oct 13 04:09:31.985704 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Oct 13 04:09:31.985704 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Oct 13 04:09:31.985704 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 13 04:09:31.985704 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 13 04:09:31.985704 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 13 04:09:31.985704 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 13 04:09:31.985704 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 13 04:09:31.985704 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application return self.get_or_create( Oct 13 04:09:31.991199 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 13 04:09:31.991199 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application with Lock( Oct 13 04:09:31.991199 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 13 04:09:31.991199 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application return self._enter() Oct 13 04:09:31.991199 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 13 04:09:31.991199 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 13 04:09:31.991199 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 13 04:09:31.991199 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application return self.creator() Oct 13 04:09:31.991199 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 13 04:09:31.991199 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application created_value = creator( Oct 13 04:09:31.991199 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Oct 13 04:09:31.991199 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 13 04:09:31.991199 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: compute_legacy. Oct 13 04:09:31.991199 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application Oct 13 04:09:31.991199 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 336/694] 10.222.0.27 () {56 vars in 1132 bytes} [Thu Oct 13 04:09:31 2022] GET /identity/v3/services/compute_legacy => generated 95 bytes in 164 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 13 04:09:31.991989 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b5a731d4-5929-442d-9b3d-e4f01381b237 None admin] Authenticating user token {{(pid=81408) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:09:32.004116 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG dbcounter [-] [81409] Writing DB stats keystone:SELECT=100 {{(pid=81409) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:09:32.069862 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 359/695] 10.222.0.27 () {58 vars in 1282 bytes} [Thu Oct 13 04:09:31 2022] GET /identity/v3/auth/tokens => generated 1474 bytes in 362 msecs (HTTP/1.1 200) 6 headers in 359 bytes (1 switches on core 0) Oct 13 04:09:32.070500 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-de8dfe26-e2e1-4fe0-9394-7e7baf7931d4 None admin] Authenticating user token {{(pid=81409) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:09:32.071481 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG dbcounter [-] [81408] Writing DB stats keystone:SELECT=100 {{(pid=81408) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:09:32.098045 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b5a731d4-5929-442d-9b3d-e4f01381b237 None admin] Validating token access rules against request {{(pid=81408) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:09:32.134805 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3a2ea253-3433-46b7-b6e8-febb66092315 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81408) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:09:32.135642 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a2ea253-3433-46b7-b6e8-febb66092315 None admin] REQUEST_METHOD: `GET` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:09:32.136123 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a2ea253-3433-46b7-b6e8-febb66092315 None admin] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:09:32.136489 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a2ea253-3433-46b7-b6e8-febb66092315 None admin] PATH_INFO: `/v3/services` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:09:32.137209 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3a2ea253-3433-46b7-b6e8-febb66092315 None admin] RBAC: Adding query filter params (name=volumev3) {{(pid=81408) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 13 04:09:32.137604 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3a2ea253-3433-46b7-b6e8-febb66092315 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:09:32.139251 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3a2ea253-3433-46b7-b6e8-febb66092315 None admin] RBAC: Authorization granted {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:09:32.144440 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 337/696] 10.222.0.27 () {56 vars in 1114 bytes} [Thu Oct 13 04:09:31 2022] GET /identity/v3/services?name=volumev3 => generated 125 bytes in 158 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 13 04:09:32.149700 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3a2ea253-3433-46b7-b6e8-febb66092315 None admin] Authenticating user token {{(pid=81408) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:09:32.174325 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-de8dfe26-e2e1-4fe0-9394-7e7baf7931d4 None admin] Validating token access rules against request {{(pid=81409) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:09:32.209317 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-54259445-6769-4073-a6f8-7dc393d471cf None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81409) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:09:32.209912 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54259445-6769-4073-a6f8-7dc393d471cf None admin] REQUEST_METHOD: `GET` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:09:32.210185 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54259445-6769-4073-a6f8-7dc393d471cf None admin] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:09:32.210476 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54259445-6769-4073-a6f8-7dc393d471cf None admin] PATH_INFO: `/v3/services` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:09:32.211092 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-54259445-6769-4073-a6f8-7dc393d471cf None admin] RBAC: Adding query filter params (name=compute_legacy) {{(pid=81409) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 13 04:09:32.211448 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-54259445-6769-4073-a6f8-7dc393d471cf None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:09:32.212978 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-54259445-6769-4073-a6f8-7dc393d471cf None admin] RBAC: Authorization granted {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:09:32.217729 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 360/697] 10.222.0.27 () {56 vars in 1126 bytes} [Thu Oct 13 04:09:32 2022] GET /identity/v3/services?name=compute_legacy => generated 131 bytes in 148 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 13 04:09:32.218556 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-54259445-6769-4073-a6f8-7dc393d471cf None admin] Authenticating user token {{(pid=81409) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:09:32.226798 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG dbcounter [-] [81409] Writing DB stats keystone:SELECT=100 {{(pid=81409) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:09:32.254811 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3a2ea253-3433-46b7-b6e8-febb66092315 None admin] Validating token access rules against request {{(pid=81408) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:09:32.295719 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8c7b423a-4295-4911-8a96-6a7585868196 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81408) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:09:32.297322 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c7b423a-4295-4911-8a96-6a7585868196 None admin] REQUEST_METHOD: `GET` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:09:32.297818 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c7b423a-4295-4911-8a96-6a7585868196 None admin] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:09:32.298296 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c7b423a-4295-4911-8a96-6a7585868196 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:09:32.308118 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG dbcounter [-] [81408] Writing DB stats keystone:SELECT=100 {{(pid=81408) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:09:32.312511 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8c7b423a-4295-4911-8a96-6a7585868196 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:09:32.314479 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8c7b423a-4295-4911-8a96-6a7585868196 None admin] RBAC: Authorization granted {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:09:32.327513 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-54259445-6769-4073-a6f8-7dc393d471cf None admin] Validating token access rules against request {{(pid=81409) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:09:32.330195 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application [None req-8c7b423a-4295-4911-8a96-6a7585868196 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 13 04:09:32.330195 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 13 04:09:32.330195 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 13 04:09:32.330195 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 13 04:09:32.330195 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 13 04:09:32.330195 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 13 04:09:32.330195 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 13 04:09:32.330195 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 13 04:09:32.330195 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 13 04:09:32.330195 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 13 04:09:32.330195 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 13 04:09:32.330195 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 13 04:09:32.330195 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 13 04:09:32.330195 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 13 04:09:32.330195 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 13 04:09:32.330195 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 13 04:09:32.331122 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 13 04:09:32.331122 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 13 04:09:32.331122 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 13 04:09:32.331122 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 13 04:09:32.331122 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 13 04:09:32.331122 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application return self.get_or_create( Oct 13 04:09:32.331122 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 13 04:09:32.331122 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application with Lock( Oct 13 04:09:32.331122 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 13 04:09:32.331122 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application return self._enter() Oct 13 04:09:32.331122 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 13 04:09:32.331122 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 13 04:09:32.331122 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 13 04:09:32.331122 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application return self.creator() Oct 13 04:09:32.331122 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 13 04:09:32.331122 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application created_value = creator( Oct 13 04:09:32.331122 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 13 04:09:32.331994 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 13 04:09:32.331994 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 13 04:09:32.331994 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application Oct 13 04:09:32.331994 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 338/698] 10.222.0.27 () {56 vars in 1108 bytes} [Thu Oct 13 04:09:32 2022] GET /identity/v3/domains/Default => generated 87 bytes in 187 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 13 04:09:32.332213 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8c7b423a-4295-4911-8a96-6a7585868196 None admin] Authenticating user token {{(pid=81408) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:09:32.368946 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-af075be4-6e9b-45d5-b8a6-744d5dbd219a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81409) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:09:32.369862 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af075be4-6e9b-45d5-b8a6-744d5dbd219a None admin] REQUEST_METHOD: `GET` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:09:32.370357 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af075be4-6e9b-45d5-b8a6-744d5dbd219a None admin] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:09:32.370805 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af075be4-6e9b-45d5-b8a6-744d5dbd219a None admin] PATH_INFO: `/v3/services` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:09:32.371623 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-af075be4-6e9b-45d5-b8a6-744d5dbd219a None admin] RBAC: Adding query filter params (type=volumev3) {{(pid=81409) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 13 04:09:32.372098 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-af075be4-6e9b-45d5-b8a6-744d5dbd219a None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:09:32.373639 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-af075be4-6e9b-45d5-b8a6-744d5dbd219a None admin] RBAC: Authorization granted {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:09:32.378913 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 361/699] 10.222.0.27 () {56 vars in 1114 bytes} [Thu Oct 13 04:09:32 2022] GET /identity/v3/services?type=volumev3 => generated 362 bytes in 160 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 13 04:09:32.379726 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-af075be4-6e9b-45d5-b8a6-744d5dbd219a None admin] Authenticating user token {{(pid=81409) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:09:32.447344 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8c7b423a-4295-4911-8a96-6a7585868196 None admin] Validating token access rules against request {{(pid=81408) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:09:32.458385 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG dbcounter [-] [81409] Writing DB stats keystone:SELECT=100 {{(pid=81409) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:09:32.483259 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1fc491a8-ece6-4b36-b9d0-22f0530cf34f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81408) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:09:32.483863 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1fc491a8-ece6-4b36-b9d0-22f0530cf34f None admin] REQUEST_METHOD: `GET` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:09:32.484136 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1fc491a8-ece6-4b36-b9d0-22f0530cf34f None admin] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:09:32.484402 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1fc491a8-ece6-4b36-b9d0-22f0530cf34f None admin] PATH_INFO: `/v3/services` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:09:32.485016 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1fc491a8-ece6-4b36-b9d0-22f0530cf34f None admin] RBAC: Adding query filter params (type=compute_legacy) {{(pid=81408) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 13 04:09:32.485326 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1fc491a8-ece6-4b36-b9d0-22f0530cf34f None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:09:32.486715 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1fc491a8-ece6-4b36-b9d0-22f0530cf34f None admin] RBAC: Authorization granted {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:09:32.496691 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 339/700] 10.222.0.27 () {56 vars in 1126 bytes} [Thu Oct 13 04:09:32 2022] GET /identity/v3/services?type=compute_legacy => generated 386 bytes in 160 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 13 04:09:32.496691 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1fc491a8-ece6-4b36-b9d0-22f0530cf34f None admin] Authenticating user token {{(pid=81408) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:09:32.500290 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-af075be4-6e9b-45d5-b8a6-744d5dbd219a None admin] Validating token access rules against request {{(pid=81409) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:09:32.536951 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-24dae152-3ce9-4fd3-a1d2-76c88835a4ec None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81409) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:09:32.537560 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24dae152-3ce9-4fd3-a1d2-76c88835a4ec None admin] REQUEST_METHOD: `GET` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:09:32.537860 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24dae152-3ce9-4fd3-a1d2-76c88835a4ec None admin] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:09:32.538153 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24dae152-3ce9-4fd3-a1d2-76c88835a4ec None admin] PATH_INFO: `/v3/domains` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:09:32.538813 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-24dae152-3ce9-4fd3-a1d2-76c88835a4ec None admin] RBAC: Adding query filter params (name=Default) {{(pid=81409) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 13 04:09:32.539187 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-24dae152-3ce9-4fd3-a1d2-76c88835a4ec None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:09:32.540803 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-24dae152-3ce9-4fd3-a1d2-76c88835a4ec None admin] RBAC: Authorization granted {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:09:32.551129 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG dbcounter [-] [81408] Writing DB stats keystone:SELECT=100 {{(pid=81408) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:09:32.553825 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 362/701] 10.222.0.27 () {56 vars in 1109 bytes} [Thu Oct 13 04:09:32 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 174 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 13 04:09:32.556237 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-24dae152-3ce9-4fd3-a1d2-76c88835a4ec None admin] Authenticating user token {{(pid=81409) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:09:32.604951 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1fc491a8-ece6-4b36-b9d0-22f0530cf34f None admin] Validating token access rules against request {{(pid=81408) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:09:32.640363 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0cdce8ab-8205-4b75-82a5-6c7cfa403275 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81408) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:09:32.641504 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0cdce8ab-8205-4b75-82a5-6c7cfa403275 None admin] REQUEST_METHOD: `GET` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:09:32.641919 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0cdce8ab-8205-4b75-82a5-6c7cfa403275 None admin] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:09:32.642189 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0cdce8ab-8205-4b75-82a5-6c7cfa403275 None admin] PATH_INFO: `/v3/endpoints` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:09:32.642920 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0cdce8ab-8205-4b75-82a5-6c7cfa403275 None admin] RBAC: Adding query filter params (interface=public, service_id=56a60a25ba4a4231ba93a51f3d58a72f, region_id=RegionOne) {{(pid=81408) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 13 04:09:32.643230 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0cdce8ab-8205-4b75-82a5-6c7cfa403275 None admin] RBAC: Authorizing `identity:list_endpoints()` {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:09:32.644608 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0cdce8ab-8205-4b75-82a5-6c7cfa403275 None admin] RBAC: Authorization granted {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:09:32.655144 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 340/702] 10.222.0.27 () {56 vars in 1251 bytes} [Thu Oct 13 04:09:32 2022] GET /identity/v3/endpoints?service_id=56a60a25ba4a4231ba93a51f3d58a72f&interface=public®ion_id=RegionOne => generated 194 bytes in 158 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 13 04:09:32.655144 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0cdce8ab-8205-4b75-82a5-6c7cfa403275 None admin] Authenticating user token {{(pid=81408) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:09:32.665615 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-24dae152-3ce9-4fd3-a1d2-76c88835a4ec None admin] Validating token access rules against request {{(pid=81409) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:09:32.694717 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG dbcounter [-] [81409] Writing DB stats keystone:SELECT=100 {{(pid=81409) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:09:32.704611 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-42ec5cfd-772f-4b6d-9292-111b4d4ec4ae None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81409) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:09:32.705432 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42ec5cfd-772f-4b6d-9292-111b4d4ec4ae None admin] REQUEST_METHOD: `POST` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:09:32.705860 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42ec5cfd-772f-4b6d-9292-111b4d4ec4ae None admin] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:09:32.706324 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42ec5cfd-772f-4b6d-9292-111b4d4ec4ae None admin] PATH_INFO: `/v3/endpoints` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:09:32.707126 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-42ec5cfd-772f-4b6d-9292-111b4d4ec4ae None admin] RBAC: Authorizing `identity:create_endpoint()` {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:09:32.708621 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-42ec5cfd-772f-4b6d-9292-111b4d4ec4ae None admin] RBAC: Authorization granted {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:09:32.738412 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 363/703] 10.222.0.27 () {60 vars in 1144 bytes} [Thu Oct 13 04:09:32 2022] POST /identity/v3/endpoints => generated 347 bytes in 183 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 13 04:09:32.739497 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-42ec5cfd-772f-4b6d-9292-111b4d4ec4ae None admin] Authenticating user token {{(pid=81409) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:09:32.772459 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0cdce8ab-8205-4b75-82a5-6c7cfa403275 None admin] Validating token access rules against request {{(pid=81408) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:09:32.799424 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG dbcounter [-] [81408] Writing DB stats keystone:SELECT=100 {{(pid=81408) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:09:32.835214 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6e5eb3b5-1429-4ea5-b143-26fae2d092ea None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81408) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:09:32.836907 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e5eb3b5-1429-4ea5-b143-26fae2d092ea None admin] REQUEST_METHOD: `GET` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:09:32.837652 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e5eb3b5-1429-4ea5-b143-26fae2d092ea None admin] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:09:32.838150 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e5eb3b5-1429-4ea5-b143-26fae2d092ea None admin] PATH_INFO: `/v3/projects/service` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:09:32.848940 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6e5eb3b5-1429-4ea5-b143-26fae2d092ea None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:09:32.851635 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6e5eb3b5-1429-4ea5-b143-26fae2d092ea None admin] RBAC: Authorization granted {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:09:32.870290 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application [None req-6e5eb3b5-1429-4ea5-b143-26fae2d092ea None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Oct 13 04:09:32.870290 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 13 04:09:32.870290 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 13 04:09:32.870290 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 13 04:09:32.870290 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 13 04:09:32.870290 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 13 04:09:32.870290 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 13 04:09:32.870290 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 13 04:09:32.870290 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 13 04:09:32.870290 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 13 04:09:32.870290 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 13 04:09:32.870290 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 13 04:09:32.870290 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Oct 13 04:09:32.870290 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application return self._get_project(project_id) Oct 13 04:09:32.870290 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Oct 13 04:09:32.871221 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Oct 13 04:09:32.871221 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 13 04:09:32.871221 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 13 04:09:32.871221 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 13 04:09:32.871221 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 13 04:09:32.871221 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 13 04:09:32.871221 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application return self.get_or_create( Oct 13 04:09:32.871221 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 13 04:09:32.871221 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application with Lock( Oct 13 04:09:32.871221 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 13 04:09:32.871221 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application return self._enter() Oct 13 04:09:32.871221 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 13 04:09:32.871221 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 13 04:09:32.871221 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 13 04:09:32.871221 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application return self.creator() Oct 13 04:09:32.871221 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 13 04:09:32.871221 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application created_value = creator( Oct 13 04:09:32.872104 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Oct 13 04:09:32.872104 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Oct 13 04:09:32.872104 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Oct 13 04:09:32.872104 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Oct 13 04:09:32.872104 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Oct 13 04:09:32.872104 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Oct 13 04:09:32.872104 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Oct 13 04:09:32.872104 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application Oct 13 04:09:32.872523 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-42ec5cfd-772f-4b6d-9292-111b4d4ec4ae None admin] Validating token access rules against request {{(pid=81409) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:09:32.879345 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 341/704] 10.222.0.27 () {56 vars in 1111 bytes} [Thu Oct 13 04:09:32 2022] GET /identity/v3/projects/service => generated 88 bytes in 223 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 13 04:09:32.886139 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6e5eb3b5-1429-4ea5-b143-26fae2d092ea None admin] Authenticating user token {{(pid=81408) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:09:32.924186 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-500e4f0e-6f13-4eb6-92a1-192e2279fba4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81409) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:09:32.924749 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-500e4f0e-6f13-4eb6-92a1-192e2279fba4 None admin] REQUEST_METHOD: `GET` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:09:32.924837 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-500e4f0e-6f13-4eb6-92a1-192e2279fba4 None admin] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:09:32.925085 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-500e4f0e-6f13-4eb6-92a1-192e2279fba4 None admin] PATH_INFO: `/v3/services` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:09:32.925696 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-500e4f0e-6f13-4eb6-92a1-192e2279fba4 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:09:32.927106 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-500e4f0e-6f13-4eb6-92a1-192e2279fba4 None admin] RBAC: Authorization granted {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:09:32.935392 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 364/705] 10.222.0.27 () {56 vars in 1087 bytes} [Thu Oct 13 04:09:32 2022] GET /identity/v3/services => generated 1497 bytes in 196 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 13 04:09:33.018981 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6e5eb3b5-1429-4ea5-b143-26fae2d092ea None admin] Validating token access rules against request {{(pid=81408) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:09:33.058692 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3c87180e-ac74-4e58-9ea0-19b52a28d8b6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81408) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:09:33.059312 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c87180e-ac74-4e58-9ea0-19b52a28d8b6 None admin] REQUEST_METHOD: `GET` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:09:33.059584 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c87180e-ac74-4e58-9ea0-19b52a28d8b6 None admin] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:09:33.059849 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c87180e-ac74-4e58-9ea0-19b52a28d8b6 None admin] PATH_INFO: `/v3/projects` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:09:33.060551 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3c87180e-ac74-4e58-9ea0-19b52a28d8b6 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81408) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 13 04:09:33.060848 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3c87180e-ac74-4e58-9ea0-19b52a28d8b6 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:09:33.062316 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3c87180e-ac74-4e58-9ea0-19b52a28d8b6 None admin] RBAC: Authorization granted {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:09:33.077196 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 342/706] 10.222.0.27 () {56 vars in 1148 bytes} [Thu Oct 13 04:09:32 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 191 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 13 04:09:33.083020 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-500e4f0e-6f13-4eb6-92a1-192e2279fba4 None admin] Authenticating user token {{(pid=81409) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:09:33.140392 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG dbcounter [-] [81409] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81409) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:09:33.198517 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-500e4f0e-6f13-4eb6-92a1-192e2279fba4 None admin] Validating token access rules against request {{(pid=81409) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:09:33.236824 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b61f2e06-9bf3-4716-b3b0-644295068661 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81409) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:09:33.237431 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b61f2e06-9bf3-4716-b3b0-644295068661 None admin] REQUEST_METHOD: `GET` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:09:33.237709 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b61f2e06-9bf3-4716-b3b0-644295068661 None admin] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:09:33.237978 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b61f2e06-9bf3-4716-b3b0-644295068661 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:09:33.238732 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b61f2e06-9bf3-4716-b3b0-644295068661 None admin] RBAC: Adding query filter params (role.id=0b094166c73d4276a8e8eec82da7f5ff, scope.project.id=dcec0431fb6849499d6aec35a234daa6, user.id=487b1605fd4143cfaa422a3a7a0344c7) {{(pid=81409) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 13 04:09:33.239038 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b61f2e06-9bf3-4716-b3b0-644295068661 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:09:33.240461 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b61f2e06-9bf3-4716-b3b0-644295068661 None admin] RBAC: Authorization granted {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:09:33.247841 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 365/707] 10.222.0.27 () {56 vars in 1374 bytes} [Thu Oct 13 04:09:33 2022] GET /identity/v3/role_assignments?user.id=487b1605fd4143cfaa422a3a7a0344c7&scope.project.id=dcec0431fb6849499d6aec35a234daa6&role.id=0b094166c73d4276a8e8eec82da7f5ff => generated 608 bytes in 165 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 13 04:09:33.313825 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a15e5780-d3f6-4145-8250-7a6e40ba5026 None None] REQUEST_METHOD: `GET` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:09:33.314166 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a15e5780-d3f6-4145-8250-7a6e40ba5026 None None] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:09:33.314418 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a15e5780-d3f6-4145-8250-7a6e40ba5026 None None] PATH_INFO: `/` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:09:33.316045 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 343/708] 10.222.0.27 () {54 vars in 927 bytes} [Thu Oct 13 04:09:33 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 13 04:09:33.322502 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8435200-8342-4238-9dc0-f62ec54dd8aa None None] REQUEST_METHOD: `POST` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:09:33.322814 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8435200-8342-4238-9dc0-f62ec54dd8aa None None] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:09:33.323061 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8435200-8342-4238-9dc0-f62ec54dd8aa None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:09:33.342842 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.auth.core [None req-c8435200-8342-4238-9dc0-f62ec54dd8aa None None] MFA Rules not processed for user `0e2d979563d6421995451a0e86bf3022`. Rule list: `[]` (Enabled: `True`). {{(pid=81409) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 13 04:09:33.344124 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.fernet_utils [None req-c8435200-8342-4238-9dc0-f62ec54dd8aa None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81409) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:09:33.423363 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 366/709] 10.222.0.27 () {58 vars in 1024 bytes} [Thu Oct 13 04:09:33 2022] POST /identity/v3/auth/tokens => generated 1474 bytes in 102 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Oct 13 04:09:33.699957 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a15e5780-d3f6-4145-8250-7a6e40ba5026 None None] Authenticating user token {{(pid=81408) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:09:33.739875 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG dbcounter [-] [81408] Writing DB stats keystone:SELECT=100 {{(pid=81408) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:09:33.811632 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a15e5780-d3f6-4145-8250-7a6e40ba5026 None None] Validating token access rules against request {{(pid=81408) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:09:33.847122 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eb371fc9-d5f3-4c35-86bb-24eaef645e68 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81408) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:09:33.847734 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb371fc9-d5f3-4c35-86bb-24eaef645e68 None admin] REQUEST_METHOD: `GET` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:09:33.848011 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb371fc9-d5f3-4c35-86bb-24eaef645e68 None admin] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:09:33.848276 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb371fc9-d5f3-4c35-86bb-24eaef645e68 None admin] PATH_INFO: `/v3/services/object-store` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:09:33.848855 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eb371fc9-d5f3-4c35-86bb-24eaef645e68 None admin] RBAC: Authorizing `identity:get_service(service_id=object-store)` {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:09:33.850304 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eb371fc9-d5f3-4c35-86bb-24eaef645e68 None admin] RBAC: Authorization granted {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:09:33.859974 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application [None req-eb371fc9-d5f3-4c35-86bb-24eaef645e68 None admin] Could not find service: object-store.: keystone.exception.ServiceNotFound: Could not find service: object-store. Oct 13 04:09:33.859974 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 13 04:09:33.859974 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Oct 13 04:09:33.859974 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Oct 13 04:09:33.859974 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Oct 13 04:09:33.859974 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Oct 13 04:09:33.859974 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Oct 13 04:09:33.859974 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 13 04:09:33.859974 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: object-store. Oct 13 04:09:33.859974 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application Oct 13 04:09:33.859974 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Oct 13 04:09:33.859974 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application Oct 13 04:09:33.859974 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 13 04:09:33.859974 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 13 04:09:33.859974 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 13 04:09:33.859974 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 13 04:09:33.859974 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 13 04:09:33.860984 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 13 04:09:33.860984 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 13 04:09:33.860984 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 13 04:09:33.860984 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 13 04:09:33.860984 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 13 04:09:33.860984 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 13 04:09:33.860984 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Oct 13 04:09:33.860984 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application return self._get_service(service_id) Oct 13 04:09:33.860984 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Oct 13 04:09:33.860984 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Oct 13 04:09:33.860984 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 13 04:09:33.860984 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 13 04:09:33.860984 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 13 04:09:33.860984 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 13 04:09:33.860984 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 13 04:09:33.860984 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application return self.get_or_create( Oct 13 04:09:33.861842 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 13 04:09:33.861842 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application with Lock( Oct 13 04:09:33.861842 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 13 04:09:33.861842 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application return self._enter() Oct 13 04:09:33.861842 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 13 04:09:33.861842 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 13 04:09:33.861842 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 13 04:09:33.861842 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application return self.creator() Oct 13 04:09:33.861842 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 13 04:09:33.861842 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application created_value = creator( Oct 13 04:09:33.861842 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Oct 13 04:09:33.861842 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 13 04:09:33.861842 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: object-store. Oct 13 04:09:33.861842 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application Oct 13 04:09:33.861842 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 344/710] 10.222.0.27 () {56 vars in 1126 bytes} [Thu Oct 13 04:09:33 2022] GET /identity/v3/services/object-store => generated 93 bytes in 162 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 13 04:09:33.867013 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c8435200-8342-4238-9dc0-f62ec54dd8aa None None] Authenticating user token {{(pid=81409) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:09:33.913679 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG dbcounter [-] [81409] Writing DB stats keystone:SELECT=100 {{(pid=81409) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:09:33.972181 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c8435200-8342-4238-9dc0-f62ec54dd8aa None None] Validating token access rules against request {{(pid=81409) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:09:34.009042 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-98e8660f-0d5f-4913-97af-55060777ddfa None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81409) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:09:34.009657 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98e8660f-0d5f-4913-97af-55060777ddfa None admin] REQUEST_METHOD: `GET` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:09:34.009934 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98e8660f-0d5f-4913-97af-55060777ddfa None admin] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:09:34.010249 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98e8660f-0d5f-4913-97af-55060777ddfa None admin] PATH_INFO: `/v3/services` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:09:34.010869 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-98e8660f-0d5f-4913-97af-55060777ddfa None admin] RBAC: Adding query filter params (name=object-store) {{(pid=81409) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 13 04:09:34.011172 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-98e8660f-0d5f-4913-97af-55060777ddfa None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:09:34.012583 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-98e8660f-0d5f-4913-97af-55060777ddfa None admin] RBAC: Authorization granted {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:09:34.023062 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 367/711] 10.222.0.27 () {56 vars in 1122 bytes} [Thu Oct 13 04:09:33 2022] GET /identity/v3/services?name=object-store => generated 129 bytes in 156 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 13 04:09:34.029030 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eb371fc9-d5f3-4c35-86bb-24eaef645e68 None admin] Authenticating user token {{(pid=81408) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:09:34.130637 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG dbcounter [-] [81408] Writing DB stats keystone:SELECT=100 {{(pid=81408) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:09:34.136381 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eb371fc9-d5f3-4c35-86bb-24eaef645e68 None admin] Validating token access rules against request {{(pid=81408) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:09:34.172597 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5acfe47c-7155-43df-9f84-e535c922c882 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81408) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:09:34.173257 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5acfe47c-7155-43df-9f84-e535c922c882 None admin] REQUEST_METHOD: `GET` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:09:34.173533 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5acfe47c-7155-43df-9f84-e535c922c882 None admin] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:09:34.173804 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5acfe47c-7155-43df-9f84-e535c922c882 None admin] PATH_INFO: `/v3/services` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:09:34.174514 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5acfe47c-7155-43df-9f84-e535c922c882 None admin] RBAC: Adding query filter params (type=object-store) {{(pid=81408) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 13 04:09:34.174818 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5acfe47c-7155-43df-9f84-e535c922c882 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:09:34.176188 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5acfe47c-7155-43df-9f84-e535c922c882 None admin] RBAC: Authorization granted {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:09:34.182438 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 345/712] 10.222.0.27 () {56 vars in 1122 bytes} [Thu Oct 13 04:09:34 2022] GET /identity/v3/services?type=object-store => generated 356 bytes in 154 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 13 04:09:34.188501 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-98e8660f-0d5f-4913-97af-55060777ddfa None admin] Authenticating user token {{(pid=81409) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:09:34.298589 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-98e8660f-0d5f-4913-97af-55060777ddfa None admin] Validating token access rules against request {{(pid=81409) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:09:34.306134 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG dbcounter [-] [81409] Writing DB stats keystone:SELECT=100 {{(pid=81409) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:09:34.333995 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4cd4bbc0-1d0f-451a-9d09-5569ab0eb411 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81409) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:09:34.334628 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4cd4bbc0-1d0f-451a-9d09-5569ab0eb411 None admin] REQUEST_METHOD: `POST` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:09:34.334899 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4cd4bbc0-1d0f-451a-9d09-5569ab0eb411 None admin] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:09:34.335167 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4cd4bbc0-1d0f-451a-9d09-5569ab0eb411 None admin] PATH_INFO: `/v3/endpoints` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:09:34.335821 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4cd4bbc0-1d0f-451a-9d09-5569ab0eb411 None admin] RBAC: Authorizing `identity:create_endpoint()` {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:09:34.337234 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4cd4bbc0-1d0f-451a-9d09-5569ab0eb411 None admin] RBAC: Authorization granted {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:09:34.362164 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 368/713] 10.222.0.27 () {60 vars in 1144 bytes} [Thu Oct 13 04:09:34 2022] POST /identity/v3/endpoints => generated 349 bytes in 174 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 13 04:09:35.065759 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65c22913-b640-4c40-b561-38119cbcb1ea None None] REQUEST_METHOD: `GET` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:09:35.066306 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65c22913-b640-4c40-b561-38119cbcb1ea None None] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:09:35.066306 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65c22913-b640-4c40-b561-38119cbcb1ea None None] PATH_INFO: `/` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:09:35.067934 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 346/714] 10.222.0.27 () {54 vars in 927 bytes} [Thu Oct 13 04:09:35 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 13 04:09:35.073842 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3889dea8-08f3-46b6-8f09-dba87a612a9d None None] REQUEST_METHOD: `POST` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:09:35.074669 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3889dea8-08f3-46b6-8f09-dba87a612a9d None None] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:09:35.075145 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3889dea8-08f3-46b6-8f09-dba87a612a9d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:09:35.094771 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.auth.core [None req-3889dea8-08f3-46b6-8f09-dba87a612a9d None None] MFA Rules not processed for user `0e2d979563d6421995451a0e86bf3022`. Rule list: `[]` (Enabled: `True`). {{(pid=81409) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 13 04:09:35.096371 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.fernet_utils [None req-3889dea8-08f3-46b6-8f09-dba87a612a9d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81409) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:09:35.178311 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 369/715] 10.222.0.27 () {58 vars in 1024 bytes} [Thu Oct 13 04:09:35 2022] POST /identity/v3/auth/tokens => generated 1474 bytes in 101 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Oct 13 04:09:35.213177 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-490094fe-7160-4a88-8d8f-6ed89cb98f9f None None] REQUEST_METHOD: `GET` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:09:35.213482 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-490094fe-7160-4a88-8d8f-6ed89cb98f9f None None] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:09:35.213718 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-490094fe-7160-4a88-8d8f-6ed89cb98f9f None None] PATH_INFO: `/` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:09:35.215322 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 347/716] 10.222.0.27 () {54 vars in 927 bytes} [Thu Oct 13 04:09:35 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 13 04:09:35.220512 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c2d8b0b-0e08-48e6-b2a5-38a09ded5570 None None] REQUEST_METHOD: `POST` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:09:35.220795 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c2d8b0b-0e08-48e6-b2a5-38a09ded5570 None None] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:09:35.221068 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c2d8b0b-0e08-48e6-b2a5-38a09ded5570 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:09:35.240701 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.auth.core [None req-3c2d8b0b-0e08-48e6-b2a5-38a09ded5570 None None] MFA Rules not processed for user `0e2d979563d6421995451a0e86bf3022`. Rule list: `[]` (Enabled: `True`). {{(pid=81409) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 13 04:09:35.241873 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.fernet_utils [None req-3c2d8b0b-0e08-48e6-b2a5-38a09ded5570 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81409) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:09:35.312587 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 370/717] 10.222.0.27 () {58 vars in 1024 bytes} [Thu Oct 13 04:09:35 2022] POST /identity/v3/auth/tokens => generated 1474 bytes in 93 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Oct 13 04:09:35.447316 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-490094fe-7160-4a88-8d8f-6ed89cb98f9f None None] Authenticating user token {{(pid=81408) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:09:35.556655 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-490094fe-7160-4a88-8d8f-6ed89cb98f9f None None] Validating token access rules against request {{(pid=81408) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:09:35.577172 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1d436b0-3895-41a4-9fab-8d1e7b65b802 None None] REQUEST_METHOD: `GET` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:09:35.577809 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1d436b0-3895-41a4-9fab-8d1e7b65b802 None None] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:09:35.578309 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1d436b0-3895-41a4-9fab-8d1e7b65b802 None None] PATH_INFO: `/` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:09:35.580054 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 371/718] 10.222.0.27 () {54 vars in 927 bytes} [Thu Oct 13 04:09:35 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 13 04:09:35.585121 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da2d10db-bf10-40a2-9474-659834ff7ec6 None None] REQUEST_METHOD: `POST` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:09:35.585619 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da2d10db-bf10-40a2-9474-659834ff7ec6 None None] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:09:35.586138 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da2d10db-bf10-40a2-9474-659834ff7ec6 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:09:35.592713 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dc3908d9-3de4-4b3c-b757-38e4ab8c4c2e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81408) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:09:35.593342 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc3908d9-3de4-4b3c-b757-38e4ab8c4c2e None admin] REQUEST_METHOD: `GET` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:09:35.593666 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc3908d9-3de4-4b3c-b757-38e4ab8c4c2e None admin] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:09:35.593939 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc3908d9-3de4-4b3c-b757-38e4ab8c4c2e None admin] PATH_INFO: `/v3/services/compute` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:09:35.594603 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dc3908d9-3de4-4b3c-b757-38e4ab8c4c2e None admin] RBAC: Authorizing `identity:get_service(service_id=compute)` {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:09:35.596500 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dc3908d9-3de4-4b3c-b757-38e4ab8c4c2e None admin] RBAC: Authorization granted {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:09:35.606001 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application [None req-dc3908d9-3de4-4b3c-b757-38e4ab8c4c2e None admin] Could not find service: compute.: keystone.exception.ServiceNotFound: Could not find service: compute. Oct 13 04:09:35.606001 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 13 04:09:35.606001 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Oct 13 04:09:35.606001 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Oct 13 04:09:35.606001 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Oct 13 04:09:35.606001 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Oct 13 04:09:35.606001 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Oct 13 04:09:35.606001 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 13 04:09:35.606001 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: compute. Oct 13 04:09:35.606001 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application Oct 13 04:09:35.606001 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Oct 13 04:09:35.606001 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application Oct 13 04:09:35.606001 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 13 04:09:35.606001 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 13 04:09:35.606001 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 13 04:09:35.606001 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 13 04:09:35.606001 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 13 04:09:35.607012 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 13 04:09:35.607012 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 13 04:09:35.607012 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 13 04:09:35.607012 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 13 04:09:35.607012 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 13 04:09:35.607012 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 13 04:09:35.607012 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Oct 13 04:09:35.607012 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application return self._get_service(service_id) Oct 13 04:09:35.607012 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Oct 13 04:09:35.607012 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Oct 13 04:09:35.607012 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 13 04:09:35.607012 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 13 04:09:35.607012 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 13 04:09:35.607012 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 13 04:09:35.607012 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 13 04:09:35.607012 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application return self.get_or_create( Oct 13 04:09:35.607832 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 13 04:09:35.607832 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application with Lock( Oct 13 04:09:35.607832 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 13 04:09:35.607832 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application return self._enter() Oct 13 04:09:35.607832 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 13 04:09:35.607832 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 13 04:09:35.607832 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 13 04:09:35.607832 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application return self.creator() Oct 13 04:09:35.607832 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 13 04:09:35.607832 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application created_value = creator( Oct 13 04:09:35.607832 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Oct 13 04:09:35.607832 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 13 04:09:35.607832 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: compute. Oct 13 04:09:35.607832 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application Oct 13 04:09:35.610300 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 348/719] 10.222.0.27 () {56 vars in 1111 bytes} [Thu Oct 13 04:09:35 2022] GET /identity/v3/services/compute => generated 88 bytes in 161 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 13 04:09:35.611000 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.auth.core [None req-da2d10db-bf10-40a2-9474-659834ff7ec6 None None] MFA Rules not processed for user `0e2d979563d6421995451a0e86bf3022`. Rule list: `[]` (Enabled: `True`). {{(pid=81409) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 13 04:09:35.611182 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dc3908d9-3de4-4b3c-b757-38e4ab8c4c2e None admin] Authenticating user token {{(pid=81408) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:09:35.612615 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.fernet_utils [None req-da2d10db-bf10-40a2-9474-659834ff7ec6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81409) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:09:35.627588 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG dbcounter [-] [81409] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81409) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:09:35.642103 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG dbcounter [-] [81408] Writing DB stats keystone:SELECT=100 {{(pid=81408) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:09:35.687133 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 372/720] 10.222.0.27 () {58 vars in 1024 bytes} [Thu Oct 13 04:09:35 2022] POST /identity/v3/auth/tokens => generated 1474 bytes in 102 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Oct 13 04:09:35.695170 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-da2d10db-bf10-40a2-9474-659834ff7ec6 None None] Authenticating user token {{(pid=81409) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:09:35.722387 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dc3908d9-3de4-4b3c-b757-38e4ab8c4c2e None admin] Validating token access rules against request {{(pid=81408) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:09:35.760227 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2ce3ade1-5816-4ac3-9752-ac5b97675b9f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81408) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:09:35.761308 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ce3ade1-5816-4ac3-9752-ac5b97675b9f None admin] REQUEST_METHOD: `GET` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:09:35.761894 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ce3ade1-5816-4ac3-9752-ac5b97675b9f None admin] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:09:35.762403 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ce3ade1-5816-4ac3-9752-ac5b97675b9f None admin] PATH_INFO: `/v3/services/volumev3` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:09:35.763177 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2ce3ade1-5816-4ac3-9752-ac5b97675b9f None admin] RBAC: Authorizing `identity:get_service(service_id=volumev3)` {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:09:35.764994 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2ce3ade1-5816-4ac3-9752-ac5b97675b9f None admin] RBAC: Authorization granted {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:09:35.774889 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application [None req-2ce3ade1-5816-4ac3-9752-ac5b97675b9f None admin] Could not find service: volumev3.: keystone.exception.ServiceNotFound: Could not find service: volumev3. Oct 13 04:09:35.774889 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 13 04:09:35.774889 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Oct 13 04:09:35.774889 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Oct 13 04:09:35.774889 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Oct 13 04:09:35.774889 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Oct 13 04:09:35.774889 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Oct 13 04:09:35.774889 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 13 04:09:35.774889 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: volumev3. Oct 13 04:09:35.774889 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application Oct 13 04:09:35.774889 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Oct 13 04:09:35.774889 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application Oct 13 04:09:35.774889 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 13 04:09:35.774889 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 13 04:09:35.774889 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 13 04:09:35.774889 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 13 04:09:35.774889 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 13 04:09:35.775851 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 13 04:09:35.775851 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 13 04:09:35.775851 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 13 04:09:35.775851 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 13 04:09:35.775851 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 13 04:09:35.775851 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 13 04:09:35.775851 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Oct 13 04:09:35.775851 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application return self._get_service(service_id) Oct 13 04:09:35.775851 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Oct 13 04:09:35.775851 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Oct 13 04:09:35.775851 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 13 04:09:35.775851 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 13 04:09:35.775851 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 13 04:09:35.775851 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 13 04:09:35.775851 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 13 04:09:35.775851 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application return self.get_or_create( Oct 13 04:09:35.776686 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 13 04:09:35.776686 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application with Lock( Oct 13 04:09:35.776686 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 13 04:09:35.776686 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application return self._enter() Oct 13 04:09:35.776686 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 13 04:09:35.776686 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 13 04:09:35.776686 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 13 04:09:35.776686 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application return self.creator() Oct 13 04:09:35.776686 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 13 04:09:35.776686 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application created_value = creator( Oct 13 04:09:35.776686 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Oct 13 04:09:35.776686 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 13 04:09:35.776686 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: volumev3. Oct 13 04:09:35.776686 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application Oct 13 04:09:35.784154 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 349/721] 10.222.0.27 () {56 vars in 1114 bytes} [Thu Oct 13 04:09:35 2022] GET /identity/v3/services/volumev3 => generated 89 bytes in 173 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 13 04:09:35.785231 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2ce3ade1-5816-4ac3-9752-ac5b97675b9f None admin] Authenticating user token {{(pid=81408) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:09:35.822694 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-da2d10db-bf10-40a2-9474-659834ff7ec6 None None] Validating token access rules against request {{(pid=81409) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:09:35.867863 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e5bac951-5822-4e0a-859c-4ad396e328b4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81409) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:09:35.868474 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e5bac951-5822-4e0a-859c-4ad396e328b4 None admin] REQUEST_METHOD: `GET` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:09:35.868753 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e5bac951-5822-4e0a-859c-4ad396e328b4 None admin] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:09:35.869025 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e5bac951-5822-4e0a-859c-4ad396e328b4 None admin] PATH_INFO: `/v3/services` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:09:35.870118 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e5bac951-5822-4e0a-859c-4ad396e328b4 None admin] RBAC: Adding query filter params (name=compute) {{(pid=81409) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 13 04:09:35.870448 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e5bac951-5822-4e0a-859c-4ad396e328b4 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:09:35.872235 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e5bac951-5822-4e0a-859c-4ad396e328b4 None admin] RBAC: Authorization granted {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:09:35.882279 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 373/722] 10.222.0.27 () {56 vars in 1112 bytes} [Thu Oct 13 04:09:35 2022] GET /identity/v3/services?name=compute => generated 124 bytes in 192 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 13 04:09:35.885843 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e5bac951-5822-4e0a-859c-4ad396e328b4 None admin] Authenticating user token {{(pid=81409) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:09:35.887363 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG dbcounter [-] [81408] Writing DB stats keystone:SELECT=100 {{(pid=81408) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:09:35.898972 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2ce3ade1-5816-4ac3-9752-ac5b97675b9f None admin] Validating token access rules against request {{(pid=81408) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:09:35.906042 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG dbcounter [-] [81409] Writing DB stats keystone:SELECT=100 {{(pid=81409) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:09:35.938053 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4a269d9a-3ee5-44d2-87fa-dcbf32d5c0dc None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81408) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:09:35.939010 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a269d9a-3ee5-44d2-87fa-dcbf32d5c0dc None admin] REQUEST_METHOD: `GET` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:09:35.939633 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a269d9a-3ee5-44d2-87fa-dcbf32d5c0dc None admin] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:09:35.940108 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a269d9a-3ee5-44d2-87fa-dcbf32d5c0dc None admin] PATH_INFO: `/v3/services` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:09:35.940926 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4a269d9a-3ee5-44d2-87fa-dcbf32d5c0dc None admin] RBAC: Adding query filter params (name=volumev3) {{(pid=81408) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 13 04:09:35.941514 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4a269d9a-3ee5-44d2-87fa-dcbf32d5c0dc None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:09:35.943207 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4a269d9a-3ee5-44d2-87fa-dcbf32d5c0dc None admin] RBAC: Authorization granted {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:09:35.953989 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 350/723] 10.222.0.27 () {56 vars in 1114 bytes} [Thu Oct 13 04:09:35 2022] GET /identity/v3/services?name=volumev3 => generated 125 bytes in 164 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 13 04:09:35.955457 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4a269d9a-3ee5-44d2-87fa-dcbf32d5c0dc None admin] Authenticating user token {{(pid=81408) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:09:36.006671 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e5bac951-5822-4e0a-859c-4ad396e328b4 None admin] Validating token access rules against request {{(pid=81409) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:09:36.043715 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5da0d68c-da5b-4e75-8bfd-999af38f6d98 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81409) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:09:36.044318 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5da0d68c-da5b-4e75-8bfd-999af38f6d98 None admin] REQUEST_METHOD: `GET` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:09:36.044590 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5da0d68c-da5b-4e75-8bfd-999af38f6d98 None admin] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:09:36.044855 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5da0d68c-da5b-4e75-8bfd-999af38f6d98 None admin] PATH_INFO: `/v3/services` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:09:36.045479 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5da0d68c-da5b-4e75-8bfd-999af38f6d98 None admin] RBAC: Adding query filter params (type=compute) {{(pid=81409) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 13 04:09:36.045773 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5da0d68c-da5b-4e75-8bfd-999af38f6d98 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:09:36.047182 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5da0d68c-da5b-4e75-8bfd-999af38f6d98 None admin] RBAC: Authorization granted {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:09:36.062233 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 374/724] 10.222.0.27 () {56 vars in 1112 bytes} [Thu Oct 13 04:09:35 2022] GET /identity/v3/services?type=compute => generated 124 bytes in 167 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 13 04:09:36.062233 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5da0d68c-da5b-4e75-8bfd-999af38f6d98 None admin] Authenticating user token {{(pid=81409) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:09:36.075770 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4a269d9a-3ee5-44d2-87fa-dcbf32d5c0dc None admin] Validating token access rules against request {{(pid=81408) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:09:36.119981 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b0c1c9dc-d642-40ab-8b5a-e677b7defb69 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81408) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:09:36.120609 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0c1c9dc-d642-40ab-8b5a-e677b7defb69 None admin] REQUEST_METHOD: `GET` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:09:36.120887 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0c1c9dc-d642-40ab-8b5a-e677b7defb69 None admin] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:09:36.121158 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0c1c9dc-d642-40ab-8b5a-e677b7defb69 None admin] PATH_INFO: `/v3/services` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:09:36.121803 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b0c1c9dc-d642-40ab-8b5a-e677b7defb69 None admin] RBAC: Adding query filter params (type=volumev3) {{(pid=81408) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 13 04:09:36.122106 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b0c1c9dc-d642-40ab-8b5a-e677b7defb69 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:09:36.123688 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b0c1c9dc-d642-40ab-8b5a-e677b7defb69 None admin] RBAC: Authorization granted {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:09:36.129658 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 351/725] 10.222.0.27 () {56 vars in 1114 bytes} [Thu Oct 13 04:09:35 2022] GET /identity/v3/services?type=volumev3 => generated 362 bytes in 175 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 13 04:09:36.135275 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b0c1c9dc-d642-40ab-8b5a-e677b7defb69 None admin] Authenticating user token {{(pid=81408) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:09:36.157177 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG dbcounter [-] [81408] Writing DB stats keystone:SELECT=100 {{(pid=81408) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:09:36.194820 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG dbcounter [-] [81409] Writing DB stats keystone:SELECT=100 {{(pid=81409) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:09:36.212109 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5da0d68c-da5b-4e75-8bfd-999af38f6d98 None admin] Validating token access rules against request {{(pid=81409) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:09:36.253895 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b0c1c9dc-d642-40ab-8b5a-e677b7defb69 None admin] Validating token access rules against request {{(pid=81408) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:09:36.263119 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eb0ddee6-792c-4c49-893b-440f0ee46034 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81409) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:09:36.264607 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb0ddee6-792c-4c49-893b-440f0ee46034 None admin] REQUEST_METHOD: `GET` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:09:36.264881 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb0ddee6-792c-4c49-893b-440f0ee46034 None admin] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:09:36.265149 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb0ddee6-792c-4c49-893b-440f0ee46034 None admin] PATH_INFO: `/v3/services/image` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:09:36.266022 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eb0ddee6-792c-4c49-893b-440f0ee46034 None admin] RBAC: Authorizing `identity:get_service(service_id=image)` {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:09:36.267672 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eb0ddee6-792c-4c49-893b-440f0ee46034 None admin] RBAC: Authorization granted {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:09:36.280497 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application [None req-eb0ddee6-792c-4c49-893b-440f0ee46034 None admin] Could not find service: image.: keystone.exception.ServiceNotFound: Could not find service: image. Oct 13 04:09:36.280497 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 13 04:09:36.280497 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Oct 13 04:09:36.280497 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Oct 13 04:09:36.280497 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Oct 13 04:09:36.280497 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Oct 13 04:09:36.280497 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Oct 13 04:09:36.280497 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 13 04:09:36.280497 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: image. Oct 13 04:09:36.280497 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application Oct 13 04:09:36.280497 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Oct 13 04:09:36.280497 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application Oct 13 04:09:36.280497 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 13 04:09:36.280497 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 13 04:09:36.280497 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 13 04:09:36.280497 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 13 04:09:36.280497 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 13 04:09:36.281597 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 13 04:09:36.281597 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 13 04:09:36.281597 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 13 04:09:36.281597 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 13 04:09:36.281597 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 13 04:09:36.281597 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 13 04:09:36.281597 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Oct 13 04:09:36.281597 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application return self._get_service(service_id) Oct 13 04:09:36.281597 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Oct 13 04:09:36.281597 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Oct 13 04:09:36.281597 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 13 04:09:36.281597 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 13 04:09:36.281597 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 13 04:09:36.281597 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 13 04:09:36.281597 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 13 04:09:36.281597 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application return self.get_or_create( Oct 13 04:09:36.282905 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 13 04:09:36.282905 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application with Lock( Oct 13 04:09:36.282905 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 13 04:09:36.282905 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application return self._enter() Oct 13 04:09:36.282905 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 13 04:09:36.282905 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 13 04:09:36.282905 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 13 04:09:36.282905 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application return self.creator() Oct 13 04:09:36.282905 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 13 04:09:36.282905 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application created_value = creator( Oct 13 04:09:36.282905 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Oct 13 04:09:36.282905 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 13 04:09:36.282905 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: image. Oct 13 04:09:36.282905 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application Oct 13 04:09:36.282905 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 375/726] 10.222.0.27 () {56 vars in 1105 bytes} [Thu Oct 13 04:09:36 2022] GET /identity/v3/services/image => generated 86 bytes in 230 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 13 04:09:36.287924 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eb0ddee6-792c-4c49-893b-440f0ee46034 None admin] Authenticating user token {{(pid=81409) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:09:36.294785 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-65784856-3776-4d7a-9616-4381c8375faf None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81408) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:09:36.295530 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65784856-3776-4d7a-9616-4381c8375faf None admin] REQUEST_METHOD: `POST` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:09:36.295898 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65784856-3776-4d7a-9616-4381c8375faf None admin] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:09:36.296260 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65784856-3776-4d7a-9616-4381c8375faf None admin] PATH_INFO: `/v3/endpoints` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:09:36.297028 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-65784856-3776-4d7a-9616-4381c8375faf None admin] RBAC: Authorizing `identity:create_endpoint()` {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:09:36.298516 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-65784856-3776-4d7a-9616-4381c8375faf None admin] RBAC: Authorization granted {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:09:36.326293 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 352/727] 10.222.0.27 () {60 vars in 1144 bytes} [Thu Oct 13 04:09:36 2022] POST /identity/v3/endpoints => generated 346 bytes in 188 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 13 04:09:36.423504 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eb0ddee6-792c-4c49-893b-440f0ee46034 None admin] Validating token access rules against request {{(pid=81409) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:09:36.468582 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-23481799-f012-47cf-83fd-551ff20d13d4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81409) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:09:36.469115 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23481799-f012-47cf-83fd-551ff20d13d4 None admin] REQUEST_METHOD: `GET` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:09:36.469387 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23481799-f012-47cf-83fd-551ff20d13d4 None admin] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:09:36.469651 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23481799-f012-47cf-83fd-551ff20d13d4 None admin] PATH_INFO: `/v3/services` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:09:36.470304 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-23481799-f012-47cf-83fd-551ff20d13d4 None admin] RBAC: Adding query filter params (name=image) {{(pid=81409) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 13 04:09:36.470603 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-23481799-f012-47cf-83fd-551ff20d13d4 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:09:36.471979 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-23481799-f012-47cf-83fd-551ff20d13d4 None admin] RBAC: Authorization granted {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:09:36.474223 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG dbcounter [-] [81409] Writing DB stats keystone:SELECT=100 {{(pid=81409) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:09:36.479205 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 376/728] 10.222.0.27 () {56 vars in 1108 bytes} [Thu Oct 13 04:09:36 2022] GET /identity/v3/services?name=image => generated 122 bytes in 191 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 13 04:09:36.485499 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-65784856-3776-4d7a-9616-4381c8375faf None admin] Authenticating user token {{(pid=81408) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:09:36.577258 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG dbcounter [-] [81408] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81408) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:09:36.606779 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-65784856-3776-4d7a-9616-4381c8375faf None admin] Validating token access rules against request {{(pid=81408) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:09:36.631258 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d77445a9-4c9b-488c-a251-bf1170a2ef9e None None] REQUEST_METHOD: `GET` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:09:36.631573 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d77445a9-4c9b-488c-a251-bf1170a2ef9e None None] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:09:36.631786 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d77445a9-4c9b-488c-a251-bf1170a2ef9e None None] PATH_INFO: `/` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:09:36.633775 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 377/729] 10.222.0.27 () {54 vars in 927 bytes} [Thu Oct 13 04:09:36 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Oct 13 04:09:36.640010 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b67b0736-1e34-4d1e-9d76-11f361ef13fa None None] REQUEST_METHOD: `POST` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:09:36.640258 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b67b0736-1e34-4d1e-9d76-11f361ef13fa None None] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:09:36.640504 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b67b0736-1e34-4d1e-9d76-11f361ef13fa None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:09:36.648052 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8d618ca2-14bb-4abe-ba59-37695563904c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81408) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:09:36.648662 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d618ca2-14bb-4abe-ba59-37695563904c None admin] REQUEST_METHOD: `GET` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:09:36.649120 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d618ca2-14bb-4abe-ba59-37695563904c None admin] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:09:36.649523 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d618ca2-14bb-4abe-ba59-37695563904c None admin] PATH_INFO: `/v3/services` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:09:36.650447 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8d618ca2-14bb-4abe-ba59-37695563904c None admin] RBAC: Adding query filter params (type=image) {{(pid=81408) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 13 04:09:36.650748 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8d618ca2-14bb-4abe-ba59-37695563904c None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:09:36.652171 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8d618ca2-14bb-4abe-ba59-37695563904c None admin] RBAC: Authorization granted {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:09:36.662384 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 353/730] 10.222.0.27 () {56 vars in 1108 bytes} [Thu Oct 13 04:09:36 2022] GET /identity/v3/services?type=image => generated 122 bytes in 173 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 13 04:09:36.663681 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.auth.core [None req-b67b0736-1e34-4d1e-9d76-11f361ef13fa None None] MFA Rules not processed for user `0e2d979563d6421995451a0e86bf3022`. Rule list: `[]` (Enabled: `True`). {{(pid=81409) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 13 04:09:36.665694 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.fernet_utils [None req-b67b0736-1e34-4d1e-9d76-11f361ef13fa None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81409) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:09:36.741781 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 378/731] 10.222.0.27 () {58 vars in 1024 bytes} [Thu Oct 13 04:09:36 2022] POST /identity/v3/auth/tokens => generated 1474 bytes in 102 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Oct 13 04:09:37.017966 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8d618ca2-14bb-4abe-ba59-37695563904c None admin] Authenticating user token {{(pid=81408) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:09:37.124202 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8d618ca2-14bb-4abe-ba59-37695563904c None admin] Validating token access rules against request {{(pid=81408) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:09:37.164035 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e772ee1b-dce4-45e1-b2b3-30d6ddc482cf None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81408) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:09:37.164826 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e772ee1b-dce4-45e1-b2b3-30d6ddc482cf None admin] REQUEST_METHOD: `GET` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:09:37.165319 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e772ee1b-dce4-45e1-b2b3-30d6ddc482cf None admin] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:09:37.165713 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e772ee1b-dce4-45e1-b2b3-30d6ddc482cf None admin] PATH_INFO: `/v3/services/object-store` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:09:37.166357 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e772ee1b-dce4-45e1-b2b3-30d6ddc482cf None admin] RBAC: Authorizing `identity:get_service(service_id=object-store)` {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:09:37.167853 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e772ee1b-dce4-45e1-b2b3-30d6ddc482cf None admin] RBAC: Authorization granted {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:09:37.176870 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application [None req-e772ee1b-dce4-45e1-b2b3-30d6ddc482cf None admin] Could not find service: object-store.: keystone.exception.ServiceNotFound: Could not find service: object-store. Oct 13 04:09:37.176870 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 13 04:09:37.176870 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Oct 13 04:09:37.176870 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Oct 13 04:09:37.176870 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Oct 13 04:09:37.176870 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Oct 13 04:09:37.176870 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Oct 13 04:09:37.176870 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 13 04:09:37.176870 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: object-store. Oct 13 04:09:37.176870 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application Oct 13 04:09:37.176870 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Oct 13 04:09:37.176870 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application Oct 13 04:09:37.176870 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 13 04:09:37.176870 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 13 04:09:37.176870 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 13 04:09:37.176870 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 13 04:09:37.176870 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 13 04:09:37.177876 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 13 04:09:37.177876 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 13 04:09:37.177876 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 13 04:09:37.177876 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 13 04:09:37.177876 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 13 04:09:37.177876 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 13 04:09:37.177876 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Oct 13 04:09:37.177876 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application return self._get_service(service_id) Oct 13 04:09:37.177876 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Oct 13 04:09:37.177876 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Oct 13 04:09:37.177876 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 13 04:09:37.177876 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 13 04:09:37.177876 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 13 04:09:37.177876 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 13 04:09:37.177876 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 13 04:09:37.177876 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application return self.get_or_create( Oct 13 04:09:37.179302 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 13 04:09:37.179302 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application with Lock( Oct 13 04:09:37.179302 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 13 04:09:37.179302 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application return self._enter() Oct 13 04:09:37.179302 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 13 04:09:37.179302 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 13 04:09:37.179302 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 13 04:09:37.179302 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application return self.creator() Oct 13 04:09:37.179302 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 13 04:09:37.179302 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application created_value = creator( Oct 13 04:09:37.179302 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Oct 13 04:09:37.179302 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 13 04:09:37.179302 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: object-store. Oct 13 04:09:37.179302 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application Oct 13 04:09:37.179302 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 354/732] 10.222.0.27 () {56 vars in 1126 bytes} [Thu Oct 13 04:09:37 2022] GET /identity/v3/services/object-store => generated 93 bytes in 161 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 13 04:09:37.187620 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b67b0736-1e34-4d1e-9d76-11f361ef13fa None None] Authenticating user token {{(pid=81409) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:09:37.288376 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b67b0736-1e34-4d1e-9d76-11f361ef13fa None None] Validating token access rules against request {{(pid=81409) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:09:37.322184 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG dbcounter [-] [81409] Writing DB stats keystone:SELECT=100 {{(pid=81409) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:09:37.328496 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e7e811ef-30aa-4fb7-af02-a6ee37542671 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81409) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:09:37.329109 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7e811ef-30aa-4fb7-af02-a6ee37542671 None admin] REQUEST_METHOD: `GET` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:09:37.329379 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7e811ef-30aa-4fb7-af02-a6ee37542671 None admin] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:09:37.329642 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7e811ef-30aa-4fb7-af02-a6ee37542671 None admin] PATH_INFO: `/v3/services` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:09:37.330297 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e7e811ef-30aa-4fb7-af02-a6ee37542671 None admin] RBAC: Adding query filter params (name=object-store) {{(pid=81409) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 13 04:09:37.330596 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e7e811ef-30aa-4fb7-af02-a6ee37542671 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:09:37.331947 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e7e811ef-30aa-4fb7-af02-a6ee37542671 None admin] RBAC: Authorization granted {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:09:37.337802 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 379/733] 10.222.0.27 () {56 vars in 1122 bytes} [Thu Oct 13 04:09:37 2022] GET /identity/v3/services?name=object-store => generated 129 bytes in 151 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 13 04:09:37.343281 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e772ee1b-dce4-45e1-b2b3-30d6ddc482cf None admin] Authenticating user token {{(pid=81408) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:09:37.347486 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG dbcounter [-] [81408] Writing DB stats keystone:SELECT=100 {{(pid=81408) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:09:37.466416 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e772ee1b-dce4-45e1-b2b3-30d6ddc482cf None admin] Validating token access rules against request {{(pid=81408) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:09:37.502382 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b0d8c7a2-50f2-42d3-bd34-3e25c28bae94 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81408) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:09:37.503306 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0d8c7a2-50f2-42d3-bd34-3e25c28bae94 None admin] REQUEST_METHOD: `GET` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:09:37.503804 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0d8c7a2-50f2-42d3-bd34-3e25c28bae94 None admin] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:09:37.504299 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0d8c7a2-50f2-42d3-bd34-3e25c28bae94 None admin] PATH_INFO: `/v3/services` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:09:37.505135 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b0d8c7a2-50f2-42d3-bd34-3e25c28bae94 None admin] RBAC: Adding query filter params (type=object-store) {{(pid=81408) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 13 04:09:37.505641 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b0d8c7a2-50f2-42d3-bd34-3e25c28bae94 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:09:37.507304 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b0d8c7a2-50f2-42d3-bd34-3e25c28bae94 None admin] RBAC: Authorization granted {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:09:37.513670 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 355/734] 10.222.0.27 () {56 vars in 1122 bytes} [Thu Oct 13 04:09:37 2022] GET /identity/v3/services?type=object-store => generated 356 bytes in 171 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 13 04:09:37.523927 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e7e811ef-30aa-4fb7-af02-a6ee37542671 None admin] Authenticating user token {{(pid=81409) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:09:37.624001 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e7e811ef-30aa-4fb7-af02-a6ee37542671 None admin] Validating token access rules against request {{(pid=81409) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:09:37.658459 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d55e0c4a-20ea-4025-8eb5-7f9a2747c183 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81409) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:09:37.659133 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d55e0c4a-20ea-4025-8eb5-7f9a2747c183 None admin] REQUEST_METHOD: `GET` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:09:37.659407 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d55e0c4a-20ea-4025-8eb5-7f9a2747c183 None admin] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:09:37.659669 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d55e0c4a-20ea-4025-8eb5-7f9a2747c183 None admin] PATH_INFO: `/v3/endpoints` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:09:37.660366 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d55e0c4a-20ea-4025-8eb5-7f9a2747c183 None admin] RBAC: Adding query filter params (interface=admin, service_id=b505f599e7d44052a99c9638280cdad3, region_id=RegionOne) {{(pid=81409) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 13 04:09:37.660665 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d55e0c4a-20ea-4025-8eb5-7f9a2747c183 None admin] RBAC: Authorizing `identity:list_endpoints()` {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:09:37.662012 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d55e0c4a-20ea-4025-8eb5-7f9a2747c183 None admin] RBAC: Authorization granted {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:09:37.667453 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 380/735] 10.222.0.27 () {56 vars in 1249 bytes} [Thu Oct 13 04:09:37 2022] GET /identity/v3/endpoints?service_id=b505f599e7d44052a99c9638280cdad3&interface=admin®ion_id=RegionOne => generated 193 bytes in 149 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 13 04:09:37.672926 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b0d8c7a2-50f2-42d3-bd34-3e25c28bae94 None admin] Authenticating user token {{(pid=81408) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:09:37.742068 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG dbcounter [-] [81408] Writing DB stats keystone:SELECT=100 {{(pid=81408) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:09:37.785371 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b0d8c7a2-50f2-42d3-bd34-3e25c28bae94 None admin] Validating token access rules against request {{(pid=81408) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:09:37.821612 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff98a388-d7ad-41d6-92fd-5a15972a77d1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81408) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:09:37.822250 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff98a388-d7ad-41d6-92fd-5a15972a77d1 None admin] REQUEST_METHOD: `GET` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:09:37.822585 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff98a388-d7ad-41d6-92fd-5a15972a77d1 None admin] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:09:37.822857 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff98a388-d7ad-41d6-92fd-5a15972a77d1 None admin] PATH_INFO: `/v3/services` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:09:37.823464 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ff98a388-d7ad-41d6-92fd-5a15972a77d1 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:09:37.824904 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ff98a388-d7ad-41d6-92fd-5a15972a77d1 None admin] RBAC: Authorization granted {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:09:37.831678 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 356/736] 10.222.0.27 () {56 vars in 1087 bytes} [Thu Oct 13 04:09:37 2022] GET /identity/v3/services => generated 1497 bytes in 159 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 13 04:09:38.347989 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e54cb229-c6bb-4a5e-91d5-5c6e26b5625a None None] REQUEST_METHOD: `GET` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:09:38.348542 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e54cb229-c6bb-4a5e-91d5-5c6e26b5625a None None] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:09:38.348820 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e54cb229-c6bb-4a5e-91d5-5c6e26b5625a None None] PATH_INFO: `/` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:09:38.350717 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 381/737] 10.222.0.27 () {54 vars in 927 bytes} [Thu Oct 13 04:09:38 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 13 04:09:38.357392 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5334f5ac-060e-48c5-a794-6c859a20fd87 None None] REQUEST_METHOD: `POST` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:09:38.358309 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5334f5ac-060e-48c5-a794-6c859a20fd87 None None] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:09:38.358783 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5334f5ac-060e-48c5-a794-6c859a20fd87 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:09:38.377537 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.auth.core [None req-5334f5ac-060e-48c5-a794-6c859a20fd87 None None] MFA Rules not processed for user `0e2d979563d6421995451a0e86bf3022`. Rule list: `[]` (Enabled: `True`). {{(pid=81408) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 13 04:09:38.379074 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.fernet_utils [None req-5334f5ac-060e-48c5-a794-6c859a20fd87 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81408) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:09:38.469610 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 357/738] 10.222.0.27 () {58 vars in 1024 bytes} [Thu Oct 13 04:09:38 2022] POST /identity/v3/auth/tokens => generated 1474 bytes in 107 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Oct 13 04:09:38.679500 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ef3efa6-29c3-44dc-864d-9e8b291b2920 None None] REQUEST_METHOD: `GET` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:09:38.679688 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ef3efa6-29c3-44dc-864d-9e8b291b2920 None None] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:09:38.680506 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ef3efa6-29c3-44dc-864d-9e8b291b2920 None None] PATH_INFO: `/` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:09:38.682014 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 382/739] 10.222.0.27 () {54 vars in 927 bytes} [Thu Oct 13 04:09:38 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 13 04:09:38.687358 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3efd6767-5c31-4ea5-93a3-0ca64ce99a4c None None] REQUEST_METHOD: `POST` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:09:38.687856 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3efd6767-5c31-4ea5-93a3-0ca64ce99a4c None None] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:09:38.688339 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3efd6767-5c31-4ea5-93a3-0ca64ce99a4c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:09:38.707235 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.auth.core [None req-3efd6767-5c31-4ea5-93a3-0ca64ce99a4c None None] MFA Rules not processed for user `0e2d979563d6421995451a0e86bf3022`. Rule list: `[]` (Enabled: `True`). {{(pid=81408) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 13 04:09:38.708731 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.fernet_utils [None req-3efd6767-5c31-4ea5-93a3-0ca64ce99a4c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81408) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:09:38.757085 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3ef3efa6-29c3-44dc-864d-9e8b291b2920 None None] Authenticating user token {{(pid=81409) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:09:38.776145 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG dbcounter [-] [81408] Writing DB stats keystone:SELECT=100 {{(pid=81408) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:09:38.790247 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 358/740] 10.222.0.27 () {58 vars in 1024 bytes} [Thu Oct 13 04:09:38 2022] POST /identity/v3/auth/tokens => generated 1474 bytes in 103 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Oct 13 04:09:38.835098 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG dbcounter [-] [81409] Writing DB stats keystone:SELECT=100 {{(pid=81409) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:09:38.878942 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3ef3efa6-29c3-44dc-864d-9e8b291b2920 None None] Validating token access rules against request {{(pid=81409) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:09:38.897317 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e6793e6-cde6-4db2-a118-715a01257e11 None None] REQUEST_METHOD: `GET` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:09:38.897604 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e6793e6-cde6-4db2-a118-715a01257e11 None None] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:09:38.897860 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e6793e6-cde6-4db2-a118-715a01257e11 None None] PATH_INFO: `/` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:09:38.899858 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 359/741] 10.222.0.27 () {54 vars in 927 bytes} [Thu Oct 13 04:09:38 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Oct 13 04:09:38.904702 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3eff1261-3a5a-490f-bcb6-e919e9d38b71 None None] REQUEST_METHOD: `POST` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:09:38.904977 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3eff1261-3a5a-490f-bcb6-e919e9d38b71 None None] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:09:38.905249 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3eff1261-3a5a-490f-bcb6-e919e9d38b71 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:09:38.914118 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2ada1454-7320-4866-8db0-1d5f8252d5b0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81409) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:09:38.915293 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ada1454-7320-4866-8db0-1d5f8252d5b0 None admin] REQUEST_METHOD: `POST` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:09:38.915568 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ada1454-7320-4866-8db0-1d5f8252d5b0 None admin] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:09:38.915829 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ada1454-7320-4866-8db0-1d5f8252d5b0 None admin] PATH_INFO: `/v3/services` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:09:38.916484 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2ada1454-7320-4866-8db0-1d5f8252d5b0 None admin] RBAC: Authorizing `identity:create_service()` {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:09:38.917891 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2ada1454-7320-4866-8db0-1d5f8252d5b0 None admin] RBAC: Authorization granted {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:09:38.924784 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.auth.core [None req-3eff1261-3a5a-490f-bcb6-e919e9d38b71 None None] MFA Rules not processed for user `0e2d979563d6421995451a0e86bf3022`. Rule list: `[]` (Enabled: `True`). {{(pid=81408) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 13 04:09:38.925976 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.fernet_utils [None req-3eff1261-3a5a-490f-bcb6-e919e9d38b71 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81408) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:09:38.934515 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 383/742] 10.222.0.27 () {60 vars in 1141 bytes} [Thu Oct 13 04:09:38 2022] POST /identity/v3/services => generated 242 bytes in 178 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 13 04:09:39.020610 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 360/743] 10.222.0.27 () {58 vars in 1024 bytes} [Thu Oct 13 04:09:38 2022] POST /identity/v3/auth/tokens => generated 1570 bytes in 116 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Oct 13 04:09:39.075288 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2ada1454-7320-4866-8db0-1d5f8252d5b0 None admin] Authenticating user token {{(pid=81409) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:09:39.183939 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2ada1454-7320-4866-8db0-1d5f8252d5b0 None admin] Validating token access rules against request {{(pid=81409) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:09:39.201632 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG dbcounter [-] [81409] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81409) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:09:39.220321 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b92a1136-e809-464c-aead-162982a32357 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81409) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:09:39.220973 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b92a1136-e809-464c-aead-162982a32357 None admin] REQUEST_METHOD: `POST` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:09:39.221248 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b92a1136-e809-464c-aead-162982a32357 None admin] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:09:39.221515 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b92a1136-e809-464c-aead-162982a32357 None admin] PATH_INFO: `/v3/projects` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:09:39.222153 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b92a1136-e809-464c-aead-162982a32357 None admin] RBAC: Authorizing `identity:create_project()` {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:09:39.223686 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b92a1136-e809-464c-aead-162982a32357 None admin] RBAC: Authorization granted {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:09:39.226566 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: WARNING keystone.server.flask.common [None req-b92a1136-e809-464c-aead-162982a32357 None admin] Deprecated: Not specifying a domain during a create user, group or project call, and relying on falling back to the default domain, is deprecated as of Liberty. There is no plan to remove this compatibility, however, future API versions may remove this, so please specify the domain explicitly or use a domain-scoped token. Oct 13 04:09:39.247718 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.sql.core [None req-b92a1136-e809-464c-aead-162982a32357 None admin] Conflict project: (pymysql.err.IntegrityError) (1062, "Duplicate entry 'default-service' for key 'project.ixu_project_name_domain_id'") Oct 13 04:09:39.247718 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [SQL: INSERT INTO project (id, name, domain_id, description, enabled, extra, parent_id, is_domain) VALUES (%(id)s, %(name)s, %(domain_id)s, %(description)s, %(enabled)s, %(extra)s, %(parent_id)s, %(is_domain)s)] Oct 13 04:09:39.247718 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [parameters: {'id': '1326ca1882084ea4b040fa5324c12f67', 'name': 'service', 'domain_id': 'default', 'description': '', 'enabled': 1, 'extra': '{}', 'parent_id': 'default', 'is_domain': 0}] Oct 13 04:09:39.247718 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: (Background on this error at: https://sqlalche.me/e/14/gkpj) {{(pid=81409) wrapper /opt/stack/keystone/keystone/common/sql/core.py:533}} Oct 13 04:09:39.255032 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application [None req-b92a1136-e809-464c-aead-162982a32357 None admin] Conflict occurred attempting to store project - it is not permitted to have two projects with either the same name or same id in the same domain: name is service, project id 1326ca1882084ea4b040fa5324c12f67.: keystone.exception.Conflict: Conflict occurred attempting to store project - it is not permitted to have two projects with either the same name or same id in the same domain: name is service, project id 1326ca1882084ea4b040fa5324c12f67. Oct 13 04:09:39.255032 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 13 04:09:39.255032 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1900, in _execute_context Oct 13 04:09:39.255032 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application self.dialect.do_execute( Oct 13 04:09:39.255032 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/default.py", line 732, in do_execute Oct 13 04:09:39.255032 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application cursor.execute(statement, parameters) Oct 13 04:09:39.255032 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/cursors.py", line 148, in execute Oct 13 04:09:39.255032 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application result = self._query(query) Oct 13 04:09:39.255032 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/cursors.py", line 310, in _query Oct 13 04:09:39.255032 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application conn.query(q) Oct 13 04:09:39.255032 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 548, in query Oct 13 04:09:39.255032 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application self._affected_rows = self._read_query_result(unbuffered=unbuffered) Oct 13 04:09:39.255032 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 775, in _read_query_result Oct 13 04:09:39.255847 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application result.read() Oct 13 04:09:39.255847 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 1156, in read Oct 13 04:09:39.255847 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application first_packet = self.connection._read_packet() Oct 13 04:09:39.255847 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 725, in _read_packet Oct 13 04:09:39.255847 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application packet.raise_for_error() Oct 13 04:09:39.255847 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/protocol.py", line 221, in raise_for_error Oct 13 04:09:39.255847 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application err.raise_mysql_exception(self._data) Oct 13 04:09:39.255847 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/err.py", line 143, in raise_mysql_exception Oct 13 04:09:39.255847 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application raise errorclass(errno, errval) Oct 13 04:09:39.255847 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application pymysql.err.IntegrityError: (1062, "Duplicate entry 'default-service' for key 'project.ixu_project_name_domain_id'") Oct 13 04:09:39.255847 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application Oct 13 04:09:39.255847 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application The above exception was the direct cause of the following exception: Oct 13 04:09:39.255847 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application Oct 13 04:09:39.255847 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 13 04:09:39.255847 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/sql/core.py", line 528, in wrapper Oct 13 04:09:39.255847 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application return method(*args, **kwargs) Oct 13 04:09:39.255847 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 241, in create_project Oct 13 04:09:39.255847 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application return project_ref.to_dict() Oct 13 04:09:39.256770 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ Oct 13 04:09:39.256770 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application next(self.gen) Oct 13 04:09:39.256770 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 1060, in _transaction_scope Oct 13 04:09:39.256770 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application yield resource Oct 13 04:09:39.256770 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ Oct 13 04:09:39.256770 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application next(self.gen) Oct 13 04:09:39.256770 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 662, in _session Oct 13 04:09:39.256770 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application self.session.rollback() Oct 13 04:09:39.256770 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_utils/excutils.py", line 227, in __exit__ Oct 13 04:09:39.256770 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application self.force_reraise() Oct 13 04:09:39.256770 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_utils/excutils.py", line 200, in force_reraise Oct 13 04:09:39.256770 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application raise self.value Oct 13 04:09:39.256770 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 659, in _session Oct 13 04:09:39.256770 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application self._end_session_transaction(self.session) Oct 13 04:09:39.256770 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 687, in _end_session_transaction Oct 13 04:09:39.256770 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application session.commit() Oct 13 04:09:39.256770 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 1451, in commit Oct 13 04:09:39.257647 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application self._transaction.commit(_to_root=self.future) Oct 13 04:09:39.257647 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 829, in commit Oct 13 04:09:39.257647 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application self._prepare_impl() Oct 13 04:09:39.257647 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 808, in _prepare_impl Oct 13 04:09:39.257647 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application self.session.flush() Oct 13 04:09:39.257647 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 3383, in flush Oct 13 04:09:39.257647 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application self._flush(objects) Oct 13 04:09:39.257647 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 3523, in _flush Oct 13 04:09:39.257647 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application transaction.rollback(_capture_exception=True) Oct 13 04:09:39.257647 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/util/langhelpers.py", line 70, in __exit__ Oct 13 04:09:39.257647 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application compat.raise_( Oct 13 04:09:39.257647 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/util/compat.py", line 208, in raise_ Oct 13 04:09:39.257647 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application raise exception Oct 13 04:09:39.257647 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 3483, in _flush Oct 13 04:09:39.257647 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application flush_context.execute() Oct 13 04:09:39.257647 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/unitofwork.py", line 456, in execute Oct 13 04:09:39.257647 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application rec.execute(self) Oct 13 04:09:39.259082 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/unitofwork.py", line 630, in execute Oct 13 04:09:39.259082 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application util.preloaded.orm_persistence.save_obj( Oct 13 04:09:39.259082 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/persistence.py", line 245, in save_obj Oct 13 04:09:39.259082 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application _emit_insert_statements( Oct 13 04:09:39.259082 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/persistence.py", line 1097, in _emit_insert_statements Oct 13 04:09:39.259082 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application c = connection._execute_20( Oct 13 04:09:39.259082 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1705, in _execute_20 Oct 13 04:09:39.259082 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application return meth(self, args_10style, kwargs_10style, execution_options) Oct 13 04:09:39.259082 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/sql/elements.py", line 333, in _execute_on_connection Oct 13 04:09:39.259082 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application return connection._execute_clauseelement( Oct 13 04:09:39.259082 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1572, in _execute_clauseelement Oct 13 04:09:39.259082 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application ret = self._execute_context( Oct 13 04:09:39.259082 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1943, in _execute_context Oct 13 04:09:39.259082 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application self._handle_dbapi_exception( Oct 13 04:09:39.259082 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 2122, in _handle_dbapi_exception Oct 13 04:09:39.259872 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application util.raise_(newraise, with_traceback=exc_info[2], from_=e) Oct 13 04:09:39.259872 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/util/compat.py", line 208, in raise_ Oct 13 04:09:39.259872 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application raise exception Oct 13 04:09:39.259872 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1900, in _execute_context Oct 13 04:09:39.259872 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application self.dialect.do_execute( Oct 13 04:09:39.259872 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/default.py", line 732, in do_execute Oct 13 04:09:39.259872 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application cursor.execute(statement, parameters) Oct 13 04:09:39.259872 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/cursors.py", line 148, in execute Oct 13 04:09:39.259872 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application result = self._query(query) Oct 13 04:09:39.259872 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/cursors.py", line 310, in _query Oct 13 04:09:39.259872 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application conn.query(q) Oct 13 04:09:39.259872 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 548, in query Oct 13 04:09:39.259872 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application self._affected_rows = self._read_query_result(unbuffered=unbuffered) Oct 13 04:09:39.259872 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 775, in _read_query_result Oct 13 04:09:39.259872 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application result.read() Oct 13 04:09:39.259872 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 1156, in read Oct 13 04:09:39.259872 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application first_packet = self.connection._read_packet() Oct 13 04:09:39.260731 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 725, in _read_packet Oct 13 04:09:39.260731 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application packet.raise_for_error() Oct 13 04:09:39.260731 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/protocol.py", line 221, in raise_for_error Oct 13 04:09:39.260731 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application err.raise_mysql_exception(self._data) Oct 13 04:09:39.260731 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/err.py", line 143, in raise_mysql_exception Oct 13 04:09:39.260731 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application raise errorclass(errno, errval) Oct 13 04:09:39.260731 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application oslo_db.exception.DBDuplicateEntry: (pymysql.err.IntegrityError) (1062, "Duplicate entry 'default-service' for key 'project.ixu_project_name_domain_id'") Oct 13 04:09:39.260731 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application [SQL: INSERT INTO project (id, name, domain_id, description, enabled, extra, parent_id, is_domain) VALUES (%(id)s, %(name)s, %(domain_id)s, %(description)s, %(enabled)s, %(extra)s, %(parent_id)s, %(is_domain)s)] Oct 13 04:09:39.260731 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application [parameters: {'id': '1326ca1882084ea4b040fa5324c12f67', 'name': 'service', 'domain_id': 'default', 'description': '', 'enabled': 1, 'extra': '{}', 'parent_id': 'default', 'is_domain': 0}] Oct 13 04:09:39.260731 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application (Background on this error at: https://sqlalche.me/e/14/gkpj) Oct 13 04:09:39.260731 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application Oct 13 04:09:39.260731 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Oct 13 04:09:39.260731 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application Oct 13 04:09:39.260731 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 13 04:09:39.260731 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 216, in create_project Oct 13 04:09:39.261514 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application ret = self.driver.create_project(project_id, project) Oct 13 04:09:39.261514 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/sql/core.py", line 563, in wrapper Oct 13 04:09:39.261514 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application raise exception.Conflict(type=conflict_type, Oct 13 04:09:39.261514 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application keystone.exception.Conflict: Conflict occurred attempting to store project - Duplicate entry found with name service at domain ID default. Oct 13 04:09:39.261514 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application Oct 13 04:09:39.261514 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Oct 13 04:09:39.261514 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application Oct 13 04:09:39.261514 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 13 04:09:39.261514 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 13 04:09:39.261514 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 13 04:09:39.261514 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 13 04:09:39.261514 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 13 04:09:39.261514 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 13 04:09:39.261514 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 13 04:09:39.261514 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 13 04:09:39.261514 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 13 04:09:39.261514 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 13 04:09:39.266428 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 13 04:09:39.266428 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 176, in post Oct 13 04:09:39.266428 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application ref = PROVIDERS.resource_api.create_project( Oct 13 04:09:39.266428 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 13 04:09:39.266428 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 13 04:09:39.266428 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 218, in create_project Oct 13 04:09:39.266428 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application raise exception.Conflict( Oct 13 04:09:39.266428 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application keystone.exception.Conflict: Conflict occurred attempting to store project - it is not permitted to have two projects with either the same name or same id in the same domain: name is service, project id 1326ca1882084ea4b040fa5324c12f67. Oct 13 04:09:39.266428 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application Oct 13 04:09:39.266428 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 384/744] 10.222.0.27 () {60 vars in 1140 bytes} [Thu Oct 13 04:09:39 2022] POST /identity/v3/projects => generated 262 bytes in 182 msecs (HTTP/1.1 409) 5 headers in 183 bytes (1 switches on core 0) Oct 13 04:09:39.267681 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3eff1261-3a5a-490f-bcb6-e919e9d38b71 None None] Authenticating user token {{(pid=81408) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:09:39.308951 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b92a1136-e809-464c-aead-162982a32357 None admin] Authenticating user token {{(pid=81409) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:09:39.389262 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3eff1261-3a5a-490f-bcb6-e919e9d38b71 None None] Validating token access rules against request {{(pid=81408) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:09:39.418831 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG dbcounter [-] [81408] Writing DB stats keystone:SELECT=100 {{(pid=81408) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:09:39.424038 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b92a1136-e809-464c-aead-162982a32357 None admin] Validating token access rules against request {{(pid=81409) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:09:39.431081 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-751f432a-9307-486c-afe7-a2b37fe26416 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81408) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:09:39.431715 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-751f432a-9307-486c-afe7-a2b37fe26416 None admin] REQUEST_METHOD: `GET` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:09:39.431989 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-751f432a-9307-486c-afe7-a2b37fe26416 None admin] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:09:39.432265 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-751f432a-9307-486c-afe7-a2b37fe26416 None admin] PATH_INFO: `/v3/projects/service` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:09:39.439609 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-751f432a-9307-486c-afe7-a2b37fe26416 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:09:39.440995 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-751f432a-9307-486c-afe7-a2b37fe26416 None admin] RBAC: Authorization granted {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:09:39.454342 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application [None req-751f432a-9307-486c-afe7-a2b37fe26416 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Oct 13 04:09:39.454342 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 13 04:09:39.454342 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 13 04:09:39.454342 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 13 04:09:39.454342 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 13 04:09:39.454342 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 13 04:09:39.454342 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 13 04:09:39.454342 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 13 04:09:39.454342 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 13 04:09:39.454342 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 13 04:09:39.454342 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 13 04:09:39.454342 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 13 04:09:39.454342 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Oct 13 04:09:39.454342 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application return self._get_project(project_id) Oct 13 04:09:39.454342 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Oct 13 04:09:39.455240 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Oct 13 04:09:39.455240 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 13 04:09:39.455240 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 13 04:09:39.455240 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 13 04:09:39.455240 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 13 04:09:39.455240 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 13 04:09:39.455240 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application return self.get_or_create( Oct 13 04:09:39.455240 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 13 04:09:39.455240 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application with Lock( Oct 13 04:09:39.455240 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 13 04:09:39.455240 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application return self._enter() Oct 13 04:09:39.455240 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 13 04:09:39.455240 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 13 04:09:39.455240 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 13 04:09:39.455240 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application return self.creator() Oct 13 04:09:39.455240 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 13 04:09:39.455240 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application created_value = creator( Oct 13 04:09:39.456118 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Oct 13 04:09:39.456118 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Oct 13 04:09:39.456118 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Oct 13 04:09:39.456118 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Oct 13 04:09:39.456118 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Oct 13 04:09:39.456118 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Oct 13 04:09:39.456118 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Oct 13 04:09:39.456118 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application Oct 13 04:09:39.460179 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 361/745] 10.222.0.27 () {56 vars in 1111 bytes} [Thu Oct 13 04:09:39 2022] GET /identity/v3/projects/service => generated 88 bytes in 190 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 13 04:09:39.461713 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-751f432a-9307-486c-afe7-a2b37fe26416 None admin] Authenticating user token {{(pid=81408) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:09:39.468502 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5293de80-8ed6-4053-b9e4-986c4bdbdf26 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81409) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:09:39.469477 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5293de80-8ed6-4053-b9e4-986c4bdbdf26 None admin] REQUEST_METHOD: `POST` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:09:39.469958 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5293de80-8ed6-4053-b9e4-986c4bdbdf26 None admin] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:09:39.470473 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5293de80-8ed6-4053-b9e4-986c4bdbdf26 None admin] PATH_INFO: `/v3/services` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:09:39.471784 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5293de80-8ed6-4053-b9e4-986c4bdbdf26 None admin] RBAC: Authorizing `identity:create_service()` {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:09:39.473947 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5293de80-8ed6-4053-b9e4-986c4bdbdf26 None admin] RBAC: Authorization granted {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:09:39.500298 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 385/746] 10.222.0.27 () {60 vars in 1141 bytes} [Thu Oct 13 04:09:39 2022] POST /identity/v3/services => generated 242 bytes in 192 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 13 04:09:39.587794 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-751f432a-9307-486c-afe7-a2b37fe26416 None admin] Validating token access rules against request {{(pid=81408) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:09:39.635522 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f044f16-5d80-41eb-a748-dbb0e3833533 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81408) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:09:39.636143 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f044f16-5d80-41eb-a748-dbb0e3833533 None admin] REQUEST_METHOD: `GET` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:09:39.636419 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f044f16-5d80-41eb-a748-dbb0e3833533 None admin] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:09:39.636686 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f044f16-5d80-41eb-a748-dbb0e3833533 None admin] PATH_INFO: `/v3/projects` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:09:39.637358 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5f044f16-5d80-41eb-a748-dbb0e3833533 None admin] RBAC: Adding query filter params (name=service) {{(pid=81408) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 13 04:09:39.637660 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5f044f16-5d80-41eb-a748-dbb0e3833533 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:09:39.639160 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5f044f16-5d80-41eb-a748-dbb0e3833533 None admin] RBAC: Authorization granted {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:09:39.659475 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 362/747] 10.222.0.27 () {56 vars in 1112 bytes} [Thu Oct 13 04:09:39 2022] GET /identity/v3/projects?name=service => generated 411 bytes in 199 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 13 04:09:40.126668 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab7c8d3f-9485-4aca-baf0-c0e11ee8a3ca None None] REQUEST_METHOD: `GET` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:09:40.127218 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab7c8d3f-9485-4aca-baf0-c0e11ee8a3ca None None] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:09:40.127629 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab7c8d3f-9485-4aca-baf0-c0e11ee8a3ca None None] PATH_INFO: `/` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:09:40.129328 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 386/748] 10.222.0.27 () {54 vars in 927 bytes} [Thu Oct 13 04:09:40 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 13 04:09:40.135605 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3abfe477-09cd-4858-a10e-5e905b8c7e8a None None] REQUEST_METHOD: `POST` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:09:40.136051 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3abfe477-09cd-4858-a10e-5e905b8c7e8a None None] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:09:40.136487 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3abfe477-09cd-4858-a10e-5e905b8c7e8a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:09:40.155412 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.auth.core [None req-3abfe477-09cd-4858-a10e-5e905b8c7e8a None None] MFA Rules not processed for user `0e2d979563d6421995451a0e86bf3022`. Rule list: `[]` (Enabled: `True`). {{(pid=81408) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 13 04:09:40.156801 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.fernet_utils [None req-3abfe477-09cd-4858-a10e-5e905b8c7e8a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81408) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:09:40.185458 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG dbcounter [-] [81408] Writing DB stats keystone:SELECT=100 {{(pid=81408) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:09:40.231948 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 363/749] 10.222.0.27 () {58 vars in 1024 bytes} [Thu Oct 13 04:09:40 2022] POST /identity/v3/auth/tokens => generated 1666 bytes in 97 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Oct 13 04:09:40.506015 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ab7c8d3f-9485-4aca-baf0-c0e11ee8a3ca None None] Authenticating user token {{(pid=81409) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:09:40.544081 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG dbcounter [-] [81409] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=81409) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:09:40.618929 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ab7c8d3f-9485-4aca-baf0-c0e11ee8a3ca None None] Validating token access rules against request {{(pid=81409) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:09:40.657468 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9119d88a-f11e-47e7-b1a4-554c0554fbcd None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81409) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:09:40.658387 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9119d88a-f11e-47e7-b1a4-554c0554fbcd None admin] REQUEST_METHOD: `GET` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:09:40.658918 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9119d88a-f11e-47e7-b1a4-554c0554fbcd None admin] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:09:40.659336 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9119d88a-f11e-47e7-b1a4-554c0554fbcd None admin] PATH_INFO: `/v3/services/object-store` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:09:40.660065 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9119d88a-f11e-47e7-b1a4-554c0554fbcd None admin] RBAC: Authorizing `identity:get_service(service_id=object-store)` {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:09:40.661617 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9119d88a-f11e-47e7-b1a4-554c0554fbcd None admin] RBAC: Authorization granted {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:09:40.671054 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application [None req-9119d88a-f11e-47e7-b1a4-554c0554fbcd None admin] Could not find service: object-store.: keystone.exception.ServiceNotFound: Could not find service: object-store. Oct 13 04:09:40.671054 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 13 04:09:40.671054 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Oct 13 04:09:40.671054 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Oct 13 04:09:40.671054 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Oct 13 04:09:40.671054 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Oct 13 04:09:40.671054 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Oct 13 04:09:40.671054 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 13 04:09:40.671054 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: object-store. Oct 13 04:09:40.671054 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application Oct 13 04:09:40.671054 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Oct 13 04:09:40.671054 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application Oct 13 04:09:40.671054 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 13 04:09:40.671054 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 13 04:09:40.671054 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 13 04:09:40.671054 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 13 04:09:40.671054 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 13 04:09:40.672254 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 13 04:09:40.672254 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 13 04:09:40.672254 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 13 04:09:40.672254 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 13 04:09:40.672254 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 13 04:09:40.672254 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 13 04:09:40.672254 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Oct 13 04:09:40.672254 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application return self._get_service(service_id) Oct 13 04:09:40.672254 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Oct 13 04:09:40.672254 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Oct 13 04:09:40.672254 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 13 04:09:40.672254 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 13 04:09:40.672254 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 13 04:09:40.672254 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 13 04:09:40.672254 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 13 04:09:40.672254 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application return self.get_or_create( Oct 13 04:09:40.673316 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 13 04:09:40.673316 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application with Lock( Oct 13 04:09:40.673316 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 13 04:09:40.673316 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application return self._enter() Oct 13 04:09:40.673316 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 13 04:09:40.673316 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 13 04:09:40.673316 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 13 04:09:40.673316 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application return self.creator() Oct 13 04:09:40.673316 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 13 04:09:40.673316 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application created_value = creator( Oct 13 04:09:40.673316 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Oct 13 04:09:40.673316 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 13 04:09:40.673316 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: object-store. Oct 13 04:09:40.673316 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application Oct 13 04:09:40.675552 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 387/750] 10.222.0.27 () {56 vars in 1126 bytes} [Thu Oct 13 04:09:40 2022] GET /identity/v3/services/object-store => generated 93 bytes in 170 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 13 04:09:40.683743 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3abfe477-09cd-4858-a10e-5e905b8c7e8a None None] Authenticating user token {{(pid=81408) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:09:40.786302 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3abfe477-09cd-4858-a10e-5e905b8c7e8a None None] Validating token access rules against request {{(pid=81408) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:09:40.822738 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9ac102d5-b7b8-4004-b855-bd218c8d0bbc None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81408) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:09:40.823349 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ac102d5-b7b8-4004-b855-bd218c8d0bbc None admin] REQUEST_METHOD: `GET` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:09:40.823683 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ac102d5-b7b8-4004-b855-bd218c8d0bbc None admin] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:09:40.823962 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ac102d5-b7b8-4004-b855-bd218c8d0bbc None admin] PATH_INFO: `/v3/services` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:09:40.824596 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9ac102d5-b7b8-4004-b855-bd218c8d0bbc None admin] RBAC: Adding query filter params (name=object-store) {{(pid=81408) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 13 04:09:40.824895 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9ac102d5-b7b8-4004-b855-bd218c8d0bbc None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:09:40.826381 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9ac102d5-b7b8-4004-b855-bd218c8d0bbc None admin] RBAC: Authorization granted {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:09:40.831785 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 364/751] 10.222.0.27 () {56 vars in 1122 bytes} [Thu Oct 13 04:09:40 2022] GET /identity/v3/services?name=object-store => generated 129 bytes in 150 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 13 04:09:40.838053 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9119d88a-f11e-47e7-b1a4-554c0554fbcd None admin] Authenticating user token {{(pid=81409) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:09:40.939144 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG dbcounter [-] [81409] Writing DB stats keystone:SELECT=100 {{(pid=81409) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:09:40.944206 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9119d88a-f11e-47e7-b1a4-554c0554fbcd None admin] Validating token access rules against request {{(pid=81409) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:09:40.980451 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-db531267-75f2-4796-b8fb-6d30859f1440 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81409) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:09:40.981053 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db531267-75f2-4796-b8fb-6d30859f1440 None admin] REQUEST_METHOD: `GET` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:09:40.981325 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db531267-75f2-4796-b8fb-6d30859f1440 None admin] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:09:40.981592 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db531267-75f2-4796-b8fb-6d30859f1440 None admin] PATH_INFO: `/v3/services` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:09:40.982229 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-db531267-75f2-4796-b8fb-6d30859f1440 None admin] RBAC: Adding query filter params (type=object-store) {{(pid=81409) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 13 04:09:40.982545 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-db531267-75f2-4796-b8fb-6d30859f1440 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:09:40.983901 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-db531267-75f2-4796-b8fb-6d30859f1440 None admin] RBAC: Authorization granted {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:09:40.989283 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 388/752] 10.222.0.27 () {56 vars in 1122 bytes} [Thu Oct 13 04:09:40 2022] GET /identity/v3/services?type=object-store => generated 356 bytes in 152 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 13 04:09:40.995651 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9ac102d5-b7b8-4004-b855-bd218c8d0bbc None admin] Authenticating user token {{(pid=81408) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:09:41.025083 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG dbcounter [-] [81408] Writing DB stats keystone:SELECT=100 {{(pid=81408) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:09:41.110181 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9ac102d5-b7b8-4004-b855-bd218c8d0bbc None admin] Validating token access rules against request {{(pid=81408) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:09:41.149587 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-056ddd06-b78b-45e0-8462-5d42024b1b97 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81408) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:09:41.150542 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-056ddd06-b78b-45e0-8462-5d42024b1b97 None admin] REQUEST_METHOD: `POST` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:09:41.151004 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-056ddd06-b78b-45e0-8462-5d42024b1b97 None admin] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:09:41.151484 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-056ddd06-b78b-45e0-8462-5d42024b1b97 None admin] PATH_INFO: `/v3/endpoints` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:09:41.152338 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-056ddd06-b78b-45e0-8462-5d42024b1b97 None admin] RBAC: Authorizing `identity:create_endpoint()` {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:09:41.153891 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-056ddd06-b78b-45e0-8462-5d42024b1b97 None admin] RBAC: Authorization granted {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:09:41.172310 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 365/753] 10.222.0.27 () {60 vars in 1144 bytes} [Thu Oct 13 04:09:40 2022] POST /identity/v3/endpoints => generated 325 bytes in 177 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 13 04:09:41.224670 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df2c1f1a-cbbc-46ff-9210-d4f457de3470 None None] REQUEST_METHOD: `GET` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:09:41.224865 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df2c1f1a-cbbc-46ff-9210-d4f457de3470 None None] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:09:41.225115 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df2c1f1a-cbbc-46ff-9210-d4f457de3470 None None] PATH_INFO: `/` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:09:41.227542 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 389/754] 10.222.0.27 () {54 vars in 927 bytes} [Thu Oct 13 04:09:41 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 13 04:09:41.233193 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92b65ecb-f943-48c9-bcf9-4d0c0df7124f None None] REQUEST_METHOD: `POST` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:09:41.233479 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92b65ecb-f943-48c9-bcf9-4d0c0df7124f None None] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:09:41.233754 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92b65ecb-f943-48c9-bcf9-4d0c0df7124f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:09:41.253909 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.auth.core [None req-92b65ecb-f943-48c9-bcf9-4d0c0df7124f None None] MFA Rules not processed for user `0e2d979563d6421995451a0e86bf3022`. Rule list: `[]` (Enabled: `True`). {{(pid=81408) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 13 04:09:41.255193 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.fernet_utils [None req-92b65ecb-f943-48c9-bcf9-4d0c0df7124f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81408) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:09:41.343082 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 366/755] 10.222.0.27 () {58 vars in 1024 bytes} [Thu Oct 13 04:09:41 2022] POST /identity/v3/auth/tokens => generated 1813 bytes in 111 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Oct 13 04:09:41.626593 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-df2c1f1a-cbbc-46ff-9210-d4f457de3470 None None] Authenticating user token {{(pid=81409) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:09:41.733691 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6bc5e877-0a1d-4e8d-8cf9-4a86fd5078c0 None None] REQUEST_METHOD: `GET` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:09:41.734280 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6bc5e877-0a1d-4e8d-8cf9-4a86fd5078c0 None None] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:09:41.734280 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6bc5e877-0a1d-4e8d-8cf9-4a86fd5078c0 None None] PATH_INFO: `/` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:09:41.735828 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 367/756] 10.222.0.27 () {54 vars in 927 bytes} [Thu Oct 13 04:09:41 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 13 04:09:41.738914 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-df2c1f1a-cbbc-46ff-9210-d4f457de3470 None None] Validating token access rules against request {{(pid=81409) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:09:41.741165 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf99c7d4-09a8-4d5c-a5ac-bde08cf47443 None None] REQUEST_METHOD: `POST` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:09:41.741664 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf99c7d4-09a8-4d5c-a5ac-bde08cf47443 None None] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:09:41.742003 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf99c7d4-09a8-4d5c-a5ac-bde08cf47443 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:09:41.762871 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.auth.core [None req-bf99c7d4-09a8-4d5c-a5ac-bde08cf47443 None None] MFA Rules not processed for user `0e2d979563d6421995451a0e86bf3022`. Rule list: `[]` (Enabled: `True`). {{(pid=81408) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 13 04:09:41.764411 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.fernet_utils [None req-bf99c7d4-09a8-4d5c-a5ac-bde08cf47443 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81408) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:09:41.772885 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG dbcounter [-] [81408] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81408) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:09:41.784583 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b57b3705-f5d1-4548-974f-a02d46f0a331 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81409) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:09:41.785540 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b57b3705-f5d1-4548-974f-a02d46f0a331 None admin] REQUEST_METHOD: `GET` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:09:41.785821 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b57b3705-f5d1-4548-974f-a02d46f0a331 None admin] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:09:41.786094 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b57b3705-f5d1-4548-974f-a02d46f0a331 None admin] PATH_INFO: `/v3/services/compute` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:09:41.787030 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b57b3705-f5d1-4548-974f-a02d46f0a331 None admin] RBAC: Authorizing `identity:get_service(service_id=compute)` {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:09:41.788417 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b57b3705-f5d1-4548-974f-a02d46f0a331 None admin] RBAC: Authorization granted {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:09:41.801036 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application [None req-b57b3705-f5d1-4548-974f-a02d46f0a331 None admin] Could not find service: compute.: keystone.exception.ServiceNotFound: Could not find service: compute. Oct 13 04:09:41.801036 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 13 04:09:41.801036 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Oct 13 04:09:41.801036 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Oct 13 04:09:41.801036 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Oct 13 04:09:41.801036 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Oct 13 04:09:41.801036 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Oct 13 04:09:41.801036 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 13 04:09:41.801036 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: compute. Oct 13 04:09:41.801036 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application Oct 13 04:09:41.801036 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Oct 13 04:09:41.801036 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application Oct 13 04:09:41.801036 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 13 04:09:41.801036 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 13 04:09:41.801036 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 13 04:09:41.801036 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 13 04:09:41.801036 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 13 04:09:41.802050 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 13 04:09:41.802050 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 13 04:09:41.802050 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 13 04:09:41.802050 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 13 04:09:41.802050 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 13 04:09:41.802050 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 13 04:09:41.802050 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Oct 13 04:09:41.802050 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application return self._get_service(service_id) Oct 13 04:09:41.802050 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Oct 13 04:09:41.802050 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Oct 13 04:09:41.802050 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 13 04:09:41.802050 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 13 04:09:41.802050 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 13 04:09:41.802050 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 13 04:09:41.802050 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 13 04:09:41.802050 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application return self.get_or_create( Oct 13 04:09:41.803393 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 13 04:09:41.803393 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application with Lock( Oct 13 04:09:41.803393 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 13 04:09:41.803393 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application return self._enter() Oct 13 04:09:41.803393 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 13 04:09:41.803393 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 13 04:09:41.803393 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 13 04:09:41.803393 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application return self.creator() Oct 13 04:09:41.803393 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 13 04:09:41.803393 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application created_value = creator( Oct 13 04:09:41.803393 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Oct 13 04:09:41.803393 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 13 04:09:41.803393 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: compute. Oct 13 04:09:41.803393 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application Oct 13 04:09:41.803393 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 390/757] 10.222.0.27 () {56 vars in 1111 bytes} [Thu Oct 13 04:09:41 2022] GET /identity/v3/services/compute => generated 88 bytes in 176 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 13 04:09:41.808545 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b57b3705-f5d1-4548-974f-a02d46f0a331 None admin] Authenticating user token {{(pid=81409) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:09:41.844791 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG dbcounter [-] [81409] Writing DB stats keystone:SELECT=100 {{(pid=81409) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:09:41.845211 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 368/758] 10.222.0.27 () {58 vars in 1024 bytes} [Thu Oct 13 04:09:41 2022] POST /identity/v3/auth/tokens => generated 1813 bytes in 105 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Oct 13 04:09:41.924370 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b57b3705-f5d1-4548-974f-a02d46f0a331 None admin] Validating token access rules against request {{(pid=81409) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:09:41.962342 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4084d09e-b67a-4b05-9830-176df7bfe681 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81409) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:09:41.963175 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4084d09e-b67a-4b05-9830-176df7bfe681 None admin] REQUEST_METHOD: `GET` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:09:41.963455 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4084d09e-b67a-4b05-9830-176df7bfe681 None admin] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:09:41.963726 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4084d09e-b67a-4b05-9830-176df7bfe681 None admin] PATH_INFO: `/v3/services` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:09:41.964582 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4084d09e-b67a-4b05-9830-176df7bfe681 None admin] RBAC: Adding query filter params (name=compute) {{(pid=81409) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 13 04:09:41.964889 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4084d09e-b67a-4b05-9830-176df7bfe681 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:09:41.966334 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4084d09e-b67a-4b05-9830-176df7bfe681 None admin] RBAC: Authorization granted {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:09:41.972718 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 391/759] 10.222.0.27 () {56 vars in 1112 bytes} [Thu Oct 13 04:09:41 2022] GET /identity/v3/services?name=compute => generated 124 bytes in 164 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 13 04:09:41.978973 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bf99c7d4-09a8-4d5c-a5ac-bde08cf47443 None None] Authenticating user token {{(pid=81408) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:09:42.082710 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bf99c7d4-09a8-4d5c-a5ac-bde08cf47443 None None] Validating token access rules against request {{(pid=81408) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:09:42.109462 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b267b040-e6e6-4d49-ad6b-6fe557399841 None None] REQUEST_METHOD: `GET` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:09:42.109744 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b267b040-e6e6-4d49-ad6b-6fe557399841 None None] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:09:42.110021 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b267b040-e6e6-4d49-ad6b-6fe557399841 None None] PATH_INFO: `/` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:09:42.111654 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 392/760] 10.222.0.27 () {54 vars in 927 bytes} [Thu Oct 13 04:09:42 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Oct 13 04:09:42.117495 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53c352dd-67ff-444b-8f41-d23ba3e0e707 None None] REQUEST_METHOD: `POST` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:09:42.118298 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53c352dd-67ff-444b-8f41-d23ba3e0e707 None None] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:09:42.118592 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53c352dd-67ff-444b-8f41-d23ba3e0e707 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:09:42.125760 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0daecd9c-278a-4a38-8fef-e3964fec16f9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81408) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:09:42.126704 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0daecd9c-278a-4a38-8fef-e3964fec16f9 None admin] REQUEST_METHOD: `GET` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:09:42.127159 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0daecd9c-278a-4a38-8fef-e3964fec16f9 None admin] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:09:42.127624 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0daecd9c-278a-4a38-8fef-e3964fec16f9 None admin] PATH_INFO: `/v3/services` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:09:42.128457 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0daecd9c-278a-4a38-8fef-e3964fec16f9 None admin] RBAC: Adding query filter params (type=compute) {{(pid=81408) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 13 04:09:42.128932 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0daecd9c-278a-4a38-8fef-e3964fec16f9 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:09:42.130516 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0daecd9c-278a-4a38-8fef-e3964fec16f9 None admin] RBAC: Authorization granted {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:09:42.136187 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 369/761] 10.222.0.27 () {56 vars in 1112 bytes} [Thu Oct 13 04:09:41 2022] GET /identity/v3/services?type=compute => generated 352 bytes in 158 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 13 04:09:42.141013 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.auth.core [None req-53c352dd-67ff-444b-8f41-d23ba3e0e707 None None] MFA Rules not processed for user `0e2d979563d6421995451a0e86bf3022`. Rule list: `[]` (Enabled: `True`). {{(pid=81409) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 13 04:09:42.141013 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.fernet_utils [None req-53c352dd-67ff-444b-8f41-d23ba3e0e707 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81409) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:09:42.141935 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0daecd9c-278a-4a38-8fef-e3964fec16f9 None admin] Authenticating user token {{(pid=81408) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:09:42.152250 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG dbcounter [-] [81408] Writing DB stats keystone:SELECT=100 {{(pid=81408) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:09:42.216177 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 393/762] 10.222.0.27 () {58 vars in 1024 bytes} [Thu Oct 13 04:09:42 2022] POST /identity/v3/auth/tokens => generated 1813 bytes in 99 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Oct 13 04:09:42.216979 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-53c352dd-67ff-444b-8f41-d23ba3e0e707 None None] Authenticating user token {{(pid=81409) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:09:42.241125 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG dbcounter [-] [81409] Writing DB stats keystone:SELECT=100 {{(pid=81409) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:09:42.268905 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0daecd9c-278a-4a38-8fef-e3964fec16f9 None admin] Validating token access rules against request {{(pid=81408) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:09:42.311998 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-267d3af4-7cbf-4600-b967-89530055a823 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81408) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:09:42.312621 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-267d3af4-7cbf-4600-b967-89530055a823 None admin] REQUEST_METHOD: `GET` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:09:42.312912 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-267d3af4-7cbf-4600-b967-89530055a823 None admin] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:09:42.313199 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-267d3af4-7cbf-4600-b967-89530055a823 None admin] PATH_INFO: `/v3/services/image` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:09:42.313734 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-267d3af4-7cbf-4600-b967-89530055a823 None admin] RBAC: Authorizing `identity:get_service(service_id=image)` {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:09:42.315120 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-267d3af4-7cbf-4600-b967-89530055a823 None admin] RBAC: Authorization granted {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:09:42.324593 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application [None req-267d3af4-7cbf-4600-b967-89530055a823 None admin] Could not find service: image.: keystone.exception.ServiceNotFound: Could not find service: image. Oct 13 04:09:42.324593 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 13 04:09:42.324593 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Oct 13 04:09:42.324593 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Oct 13 04:09:42.324593 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Oct 13 04:09:42.324593 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Oct 13 04:09:42.324593 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Oct 13 04:09:42.324593 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 13 04:09:42.324593 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: image. Oct 13 04:09:42.324593 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application Oct 13 04:09:42.324593 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Oct 13 04:09:42.324593 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application Oct 13 04:09:42.324593 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 13 04:09:42.324593 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 13 04:09:42.324593 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 13 04:09:42.324593 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 13 04:09:42.324593 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 13 04:09:42.325687 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 13 04:09:42.325687 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 13 04:09:42.325687 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 13 04:09:42.325687 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 13 04:09:42.325687 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 13 04:09:42.325687 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 13 04:09:42.325687 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Oct 13 04:09:42.325687 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application return self._get_service(service_id) Oct 13 04:09:42.325687 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Oct 13 04:09:42.325687 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Oct 13 04:09:42.325687 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 13 04:09:42.325687 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 13 04:09:42.325687 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 13 04:09:42.325687 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 13 04:09:42.325687 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 13 04:09:42.325687 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application return self.get_or_create( Oct 13 04:09:42.327013 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 13 04:09:42.327013 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application with Lock( Oct 13 04:09:42.327013 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 13 04:09:42.327013 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application return self._enter() Oct 13 04:09:42.327013 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 13 04:09:42.327013 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 13 04:09:42.327013 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 13 04:09:42.327013 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application return self.creator() Oct 13 04:09:42.327013 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 13 04:09:42.327013 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application created_value = creator( Oct 13 04:09:42.327013 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Oct 13 04:09:42.327013 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 13 04:09:42.327013 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: image. Oct 13 04:09:42.327013 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application Oct 13 04:09:42.330670 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 370/763] 10.222.0.27 () {56 vars in 1105 bytes} [Thu Oct 13 04:09:42 2022] GET /identity/v3/services/image => generated 86 bytes in 189 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 13 04:09:42.331489 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-267d3af4-7cbf-4600-b967-89530055a823 None admin] Authenticating user token {{(pid=81408) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:09:42.338444 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-53c352dd-67ff-444b-8f41-d23ba3e0e707 None None] Validating token access rules against request {{(pid=81409) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:09:42.385078 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-71d4d3d4-f647-4b88-af3c-a2f38b34525f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81409) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:09:42.386328 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71d4d3d4-f647-4b88-af3c-a2f38b34525f None admin] REQUEST_METHOD: `GET` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:09:42.387196 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71d4d3d4-f647-4b88-af3c-a2f38b34525f None admin] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:09:42.387673 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71d4d3d4-f647-4b88-af3c-a2f38b34525f None admin] PATH_INFO: `/v3/endpoints` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:09:42.388589 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-71d4d3d4-f647-4b88-af3c-a2f38b34525f None admin] RBAC: Adding query filter params (interface=public, service_id=a315f953fd934a398af6a5b69af0dd6a, region_id=RegionOne) {{(pid=81409) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 13 04:09:42.389112 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-71d4d3d4-f647-4b88-af3c-a2f38b34525f None admin] RBAC: Authorizing `identity:list_endpoints()` {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:09:42.391580 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-71d4d3d4-f647-4b88-af3c-a2f38b34525f None admin] RBAC: Authorization granted {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:09:42.404738 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 394/764] 10.222.0.27 () {56 vars in 1251 bytes} [Thu Oct 13 04:09:42 2022] GET /identity/v3/endpoints?service_id=a315f953fd934a398af6a5b69af0dd6a&interface=public®ion_id=RegionOne => generated 194 bytes in 184 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 13 04:09:42.413179 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-71d4d3d4-f647-4b88-af3c-a2f38b34525f None admin] Authenticating user token {{(pid=81409) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:09:42.417273 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG dbcounter [-] [81408] Writing DB stats keystone:SELECT=100 {{(pid=81408) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:09:42.462714 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-267d3af4-7cbf-4600-b967-89530055a823 None admin] Validating token access rules against request {{(pid=81408) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:09:42.508645 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG dbcounter [-] [81409] Writing DB stats keystone:SELECT=100 {{(pid=81409) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:09:42.513321 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b277d4f2-cbf2-4b25-871d-5ad581d9f108 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81408) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:09:42.514063 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b277d4f2-cbf2-4b25-871d-5ad581d9f108 None admin] REQUEST_METHOD: `GET` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:09:42.514645 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b277d4f2-cbf2-4b25-871d-5ad581d9f108 None admin] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:09:42.514924 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b277d4f2-cbf2-4b25-871d-5ad581d9f108 None admin] PATH_INFO: `/v3/services` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:09:42.515715 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b277d4f2-cbf2-4b25-871d-5ad581d9f108 None admin] RBAC: Adding query filter params (name=image) {{(pid=81408) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 13 04:09:42.516031 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b277d4f2-cbf2-4b25-871d-5ad581d9f108 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:09:42.518037 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b277d4f2-cbf2-4b25-871d-5ad581d9f108 None admin] RBAC: Authorization granted {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:09:42.529053 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 371/765] 10.222.0.27 () {56 vars in 1108 bytes} [Thu Oct 13 04:09:42 2022] GET /identity/v3/services?name=image => generated 122 bytes in 193 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 13 04:09:42.529214 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-71d4d3d4-f647-4b88-af3c-a2f38b34525f None admin] Validating token access rules against request {{(pid=81409) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:09:42.529323 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b277d4f2-cbf2-4b25-871d-5ad581d9f108 None admin] Authenticating user token {{(pid=81408) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:09:42.561373 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4ffcbb59-fc6d-41b6-a44b-38dd2e45db65 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81409) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:09:42.561984 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ffcbb59-fc6d-41b6-a44b-38dd2e45db65 None admin] REQUEST_METHOD: `GET` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:09:42.562291 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ffcbb59-fc6d-41b6-a44b-38dd2e45db65 None admin] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:09:42.562594 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ffcbb59-fc6d-41b6-a44b-38dd2e45db65 None admin] PATH_INFO: `/v3/services` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:09:42.563207 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4ffcbb59-fc6d-41b6-a44b-38dd2e45db65 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:09:42.564659 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4ffcbb59-fc6d-41b6-a44b-38dd2e45db65 None admin] RBAC: Authorization granted {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:09:42.571179 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 395/766] 10.222.0.27 () {56 vars in 1087 bytes} [Thu Oct 13 04:09:42 2022] GET /identity/v3/services => generated 1957 bytes in 165 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 13 04:09:42.571961 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4ffcbb59-fc6d-41b6-a44b-38dd2e45db65 None admin] Authenticating user token {{(pid=81409) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:09:42.667289 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b277d4f2-cbf2-4b25-871d-5ad581d9f108 None admin] Validating token access rules against request {{(pid=81408) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:09:42.709718 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4ffcbb59-fc6d-41b6-a44b-38dd2e45db65 None admin] Validating token access rules against request {{(pid=81409) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:09:42.718241 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-26bed5a2-af06-47be-b5ea-bae5e46b3c74 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81408) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:09:42.719871 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26bed5a2-af06-47be-b5ea-bae5e46b3c74 None admin] REQUEST_METHOD: `POST` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:09:42.720329 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26bed5a2-af06-47be-b5ea-bae5e46b3c74 None admin] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:09:42.720798 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26bed5a2-af06-47be-b5ea-bae5e46b3c74 None admin] PATH_INFO: `/v3/users` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:09:42.722106 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-26bed5a2-af06-47be-b5ea-bae5e46b3c74 None admin] RBAC: Authorizing `identity:create_user()` {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:09:42.724589 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-26bed5a2-af06-47be-b5ea-bae5e46b3c74 None admin] RBAC: Authorization granted {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:09:42.729734 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: WARNING keystone.server.flask.common [None req-26bed5a2-af06-47be-b5ea-bae5e46b3c74 None admin] Deprecated: Not specifying a domain during a create user, group or project call, and relying on falling back to the default domain, is deprecated as of Liberty. There is no plan to remove this compatibility, however, future API versions may remove this, so please specify the domain explicitly or use a domain-scoped token. Oct 13 04:09:42.738847 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG dbcounter [-] [81408] Writing DB stats keystone:SELECT=100 {{(pid=81408) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:09:42.746286 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.sql.core [None req-26bed5a2-af06-47be-b5ea-bae5e46b3c74 None admin] Conflict user: (pymysql.err.IntegrityError) (1062, "Duplicate entry 'default-cinder' for key 'local_user.domain_id'") Oct 13 04:09:42.746286 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [SQL: INSERT INTO local_user (user_id, domain_id, name, failed_auth_count, failed_auth_at) VALUES (%(user_id)s, %(domain_id)s, %(name)s, %(failed_auth_count)s, %(failed_auth_at)s)] Oct 13 04:09:42.746286 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [parameters: {'user_id': 'b3dbc481d315433d9de3d8fc289e09f4', 'domain_id': 'default', 'name': 'cinder', 'failed_auth_count': 0, 'failed_auth_at': None}] Oct 13 04:09:42.746286 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: (Background on this error at: https://sqlalche.me/e/14/gkpj) {{(pid=81408) wrapper /opt/stack/keystone/keystone/common/sql/core.py:533}} Oct 13 04:09:42.757543 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application [None req-26bed5a2-af06-47be-b5ea-bae5e46b3c74 None admin] Conflict occurred attempting to store user - Duplicate entry found with name cinder at domain ID default.: keystone.exception.Conflict: Conflict occurred attempting to store user - Duplicate entry found with name cinder at domain ID default. Oct 13 04:09:42.757543 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 13 04:09:42.757543 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1900, in _execute_context Oct 13 04:09:42.757543 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application self.dialect.do_execute( Oct 13 04:09:42.757543 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/default.py", line 732, in do_execute Oct 13 04:09:42.757543 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application cursor.execute(statement, parameters) Oct 13 04:09:42.757543 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/cursors.py", line 148, in execute Oct 13 04:09:42.757543 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application result = self._query(query) Oct 13 04:09:42.757543 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/cursors.py", line 310, in _query Oct 13 04:09:42.757543 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application conn.query(q) Oct 13 04:09:42.757543 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 548, in query Oct 13 04:09:42.757543 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application self._affected_rows = self._read_query_result(unbuffered=unbuffered) Oct 13 04:09:42.757543 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 775, in _read_query_result Oct 13 04:09:42.757543 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application result.read() Oct 13 04:09:42.757543 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 1156, in read Oct 13 04:09:42.759132 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application first_packet = self.connection._read_packet() Oct 13 04:09:42.759132 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 725, in _read_packet Oct 13 04:09:42.759132 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application packet.raise_for_error() Oct 13 04:09:42.759132 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/protocol.py", line 221, in raise_for_error Oct 13 04:09:42.759132 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application err.raise_mysql_exception(self._data) Oct 13 04:09:42.759132 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/err.py", line 143, in raise_mysql_exception Oct 13 04:09:42.759132 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application raise errorclass(errno, errval) Oct 13 04:09:42.759132 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application pymysql.err.IntegrityError: (1062, "Duplicate entry 'default-cinder' for key 'local_user.domain_id'") Oct 13 04:09:42.759132 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application Oct 13 04:09:42.759132 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application The above exception was the direct cause of the following exception: Oct 13 04:09:42.759132 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application Oct 13 04:09:42.759132 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 13 04:09:42.759132 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/sql/core.py", line 528, in wrapper Oct 13 04:09:42.759132 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application return method(*args, **kwargs) Oct 13 04:09:42.759132 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 141, in create_user Oct 13 04:09:42.759132 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application return base.filter_user(user_ref.to_dict()) Oct 13 04:09:42.759132 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ Oct 13 04:09:42.759132 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application next(self.gen) Oct 13 04:09:42.760073 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 1060, in _transaction_scope Oct 13 04:09:42.760073 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application yield resource Oct 13 04:09:42.760073 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ Oct 13 04:09:42.760073 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application next(self.gen) Oct 13 04:09:42.760073 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 662, in _session Oct 13 04:09:42.760073 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application self.session.rollback() Oct 13 04:09:42.760073 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_utils/excutils.py", line 227, in __exit__ Oct 13 04:09:42.760073 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application self.force_reraise() Oct 13 04:09:42.760073 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_utils/excutils.py", line 200, in force_reraise Oct 13 04:09:42.760073 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application raise self.value Oct 13 04:09:42.760073 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 659, in _session Oct 13 04:09:42.760073 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application self._end_session_transaction(self.session) Oct 13 04:09:42.760073 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 687, in _end_session_transaction Oct 13 04:09:42.760073 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application session.commit() Oct 13 04:09:42.760073 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 1451, in commit Oct 13 04:09:42.760073 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application self._transaction.commit(_to_root=self.future) Oct 13 04:09:42.760073 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 829, in commit Oct 13 04:09:42.760965 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application self._prepare_impl() Oct 13 04:09:42.760965 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 808, in _prepare_impl Oct 13 04:09:42.760965 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application self.session.flush() Oct 13 04:09:42.760965 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 3383, in flush Oct 13 04:09:42.760965 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application self._flush(objects) Oct 13 04:09:42.760965 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 3523, in _flush Oct 13 04:09:42.760965 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application transaction.rollback(_capture_exception=True) Oct 13 04:09:42.760965 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/util/langhelpers.py", line 70, in __exit__ Oct 13 04:09:42.760965 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application compat.raise_( Oct 13 04:09:42.760965 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/util/compat.py", line 208, in raise_ Oct 13 04:09:42.760965 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application raise exception Oct 13 04:09:42.760965 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 3483, in _flush Oct 13 04:09:42.760965 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application flush_context.execute() Oct 13 04:09:42.760965 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/unitofwork.py", line 456, in execute Oct 13 04:09:42.760965 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application rec.execute(self) Oct 13 04:09:42.760965 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/unitofwork.py", line 630, in execute Oct 13 04:09:42.760965 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application util.preloaded.orm_persistence.save_obj( Oct 13 04:09:42.761837 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/persistence.py", line 245, in save_obj Oct 13 04:09:42.761837 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application _emit_insert_statements( Oct 13 04:09:42.761837 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/persistence.py", line 1238, in _emit_insert_statements Oct 13 04:09:42.761837 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application result = connection._execute_20( Oct 13 04:09:42.761837 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1705, in _execute_20 Oct 13 04:09:42.761837 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application return meth(self, args_10style, kwargs_10style, execution_options) Oct 13 04:09:42.761837 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/sql/elements.py", line 333, in _execute_on_connection Oct 13 04:09:42.761837 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application return connection._execute_clauseelement( Oct 13 04:09:42.761837 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1572, in _execute_clauseelement Oct 13 04:09:42.761837 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application ret = self._execute_context( Oct 13 04:09:42.761837 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1943, in _execute_context Oct 13 04:09:42.761837 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application self._handle_dbapi_exception( Oct 13 04:09:42.761837 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 2122, in _handle_dbapi_exception Oct 13 04:09:42.761837 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application util.raise_(newraise, with_traceback=exc_info[2], from_=e) Oct 13 04:09:42.761837 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/util/compat.py", line 208, in raise_ Oct 13 04:09:42.761837 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application raise exception Oct 13 04:09:42.763232 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1900, in _execute_context Oct 13 04:09:42.763232 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application self.dialect.do_execute( Oct 13 04:09:42.763232 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/default.py", line 732, in do_execute Oct 13 04:09:42.763232 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application cursor.execute(statement, parameters) Oct 13 04:09:42.763232 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/cursors.py", line 148, in execute Oct 13 04:09:42.763232 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application result = self._query(query) Oct 13 04:09:42.763232 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/cursors.py", line 310, in _query Oct 13 04:09:42.763232 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application conn.query(q) Oct 13 04:09:42.763232 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 548, in query Oct 13 04:09:42.763232 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application self._affected_rows = self._read_query_result(unbuffered=unbuffered) Oct 13 04:09:42.763232 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 775, in _read_query_result Oct 13 04:09:42.763232 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application result.read() Oct 13 04:09:42.763232 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 1156, in read Oct 13 04:09:42.763232 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application first_packet = self.connection._read_packet() Oct 13 04:09:42.763232 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 725, in _read_packet Oct 13 04:09:42.763232 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application packet.raise_for_error() Oct 13 04:09:42.765899 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/protocol.py", line 221, in raise_for_error Oct 13 04:09:42.765899 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application err.raise_mysql_exception(self._data) Oct 13 04:09:42.765899 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/err.py", line 143, in raise_mysql_exception Oct 13 04:09:42.765899 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application raise errorclass(errno, errval) Oct 13 04:09:42.765899 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application oslo_db.exception.DBDuplicateEntry: (pymysql.err.IntegrityError) (1062, "Duplicate entry 'default-cinder' for key 'local_user.domain_id'") Oct 13 04:09:42.765899 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application [SQL: INSERT INTO local_user (user_id, domain_id, name, failed_auth_count, failed_auth_at) VALUES (%(user_id)s, %(domain_id)s, %(name)s, %(failed_auth_count)s, %(failed_auth_at)s)] Oct 13 04:09:42.765899 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application [parameters: {'user_id': 'b3dbc481d315433d9de3d8fc289e09f4', 'domain_id': 'default', 'name': 'cinder', 'failed_auth_count': 0, 'failed_auth_at': None}] Oct 13 04:09:42.765899 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application (Background on this error at: https://sqlalche.me/e/14/gkpj) Oct 13 04:09:42.765899 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application Oct 13 04:09:42.765899 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Oct 13 04:09:42.765899 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application Oct 13 04:09:42.765899 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 13 04:09:42.765899 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 13 04:09:42.765899 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 13 04:09:42.765899 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 13 04:09:42.768833 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 13 04:09:42.768833 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 13 04:09:42.768833 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 13 04:09:42.768833 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 13 04:09:42.768833 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 13 04:09:42.768833 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 13 04:09:42.768833 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 13 04:09:42.768833 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 244, in post Oct 13 04:09:42.768833 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.create_user( Oct 13 04:09:42.768833 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 13 04:09:42.768833 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 13 04:09:42.768833 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Oct 13 04:09:42.768833 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 13 04:09:42.768833 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Oct 13 04:09:42.768833 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 13 04:09:42.768833 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1000, in create_user Oct 13 04:09:42.768833 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application ref = self._create_user_with_federated_objects(user, driver) Oct 13 04:09:42.771043 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 966, in _create_user_with_federated_objects Oct 13 04:09:42.771043 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application user = driver.create_user(user['id'], user) Oct 13 04:09:42.771043 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/sql/core.py", line 563, in wrapper Oct 13 04:09:42.771043 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application raise exception.Conflict(type=conflict_type, Oct 13 04:09:42.771043 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application keystone.exception.Conflict: Conflict occurred attempting to store user - Duplicate entry found with name cinder at domain ID default. Oct 13 04:09:42.771043 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application Oct 13 04:09:42.771043 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 372/767] 10.222.0.27 () {60 vars in 1131 bytes} [Thu Oct 13 04:09:42 2022] POST /identity/v3/users => generated 160 bytes in 234 msecs (HTTP/1.1 409) 5 headers in 183 bytes (1 switches on core 0) Oct 13 04:09:42.771963 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a91b7101-8aaf-4bc5-8705-4dfde09ba4bb None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81409) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:09:42.771963 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a91b7101-8aaf-4bc5-8705-4dfde09ba4bb None admin] REQUEST_METHOD: `GET` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:09:42.771963 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a91b7101-8aaf-4bc5-8705-4dfde09ba4bb None admin] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:09:42.771963 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a91b7101-8aaf-4bc5-8705-4dfde09ba4bb None admin] PATH_INFO: `/v3/services` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:09:42.772556 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a91b7101-8aaf-4bc5-8705-4dfde09ba4bb None admin] RBAC: Adding query filter params (type=image) {{(pid=81409) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 13 04:09:42.772556 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a91b7101-8aaf-4bc5-8705-4dfde09ba4bb None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:09:42.772556 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a91b7101-8aaf-4bc5-8705-4dfde09ba4bb None admin] RBAC: Authorization granted {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:09:42.776947 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-26bed5a2-af06-47be-b5ea-bae5e46b3c74 None admin] Authenticating user token {{(pid=81408) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:09:42.777621 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 396/768] 10.222.0.27 () {56 vars in 1108 bytes} [Thu Oct 13 04:09:42 2022] GET /identity/v3/services?type=image => generated 350 bytes in 206 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 13 04:09:42.783445 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a91b7101-8aaf-4bc5-8705-4dfde09ba4bb None admin] Authenticating user token {{(pid=81409) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:09:42.801525 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG dbcounter [-] [81409] Writing DB stats keystone:SELECT=100 {{(pid=81409) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:09:42.904061 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a91b7101-8aaf-4bc5-8705-4dfde09ba4bb None admin] Validating token access rules against request {{(pid=81409) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:09:42.905181 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-26bed5a2-af06-47be-b5ea-bae5e46b3c74 None admin] Validating token access rules against request {{(pid=81408) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:09:42.946126 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7eb7b203-6606-4f14-b8b0-4aae7d40c712 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81408) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:09:42.946848 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4d91879f-2931-4c3f-8aeb-449c245ebc1b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81409) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:09:42.947394 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7eb7b203-6606-4f14-b8b0-4aae7d40c712 None admin] REQUEST_METHOD: `GET` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:09:42.947503 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d91879f-2931-4c3f-8aeb-449c245ebc1b None admin] REQUEST_METHOD: `GET` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:09:42.947856 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d91879f-2931-4c3f-8aeb-449c245ebc1b None admin] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:09:42.948126 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7eb7b203-6606-4f14-b8b0-4aae7d40c712 None admin] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:09:42.948226 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d91879f-2931-4c3f-8aeb-449c245ebc1b None admin] PATH_INFO: `/v3/endpoints` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:09:42.948667 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7eb7b203-6606-4f14-b8b0-4aae7d40c712 None admin] PATH_INFO: `/v3/users/cinder` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:09:42.949010 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4d91879f-2931-4c3f-8aeb-449c245ebc1b None admin] RBAC: Adding query filter params (interface=public, service_id=bf9a562793324ba2b5cb8f4dc4f4b6e2, region_id=RegionOne) {{(pid=81409) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 13 04:09:42.949380 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4d91879f-2931-4c3f-8aeb-449c245ebc1b None admin] RBAC: Authorizing `identity:list_endpoints()` {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:09:42.950897 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4d91879f-2931-4c3f-8aeb-449c245ebc1b None admin] RBAC: Authorization granted {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:09:42.958252 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7eb7b203-6606-4f14-b8b0-4aae7d40c712 None admin] RBAC: Authorizing `identity:get_user(user_id=cinder)` {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:09:42.958526 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 397/769] 10.222.0.27 () {56 vars in 1251 bytes} [Thu Oct 13 04:09:42 2022] GET /identity/v3/endpoints?service_id=bf9a562793324ba2b5cb8f4dc4f4b6e2&interface=public®ion_id=RegionOne => generated 194 bytes in 175 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 13 04:09:42.963466 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4d91879f-2931-4c3f-8aeb-449c245ebc1b None admin] Authenticating user token {{(pid=81409) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:09:42.964385 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7eb7b203-6606-4f14-b8b0-4aae7d40c712 None admin] RBAC: Authorization granted {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:09:42.977460 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application [None req-7eb7b203-6606-4f14-b8b0-4aae7d40c712 None admin] Could not find user: cinder.: keystone.exception.UserNotFound: Could not find user: cinder. Oct 13 04:09:42.977460 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 13 04:09:42.977460 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 13 04:09:42.977460 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 13 04:09:42.977460 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 13 04:09:42.977460 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 13 04:09:42.977460 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 13 04:09:42.977460 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 13 04:09:42.977460 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 13 04:09:42.977460 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 13 04:09:42.977460 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 13 04:09:42.977460 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 13 04:09:42.977460 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Oct 13 04:09:42.977460 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application return self._get_user(user_id) Oct 13 04:09:42.977460 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Oct 13 04:09:42.977460 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Oct 13 04:09:42.978619 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 13 04:09:42.978619 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 13 04:09:42.978619 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Oct 13 04:09:42.978619 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 13 04:09:42.978619 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Oct 13 04:09:42.978619 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 13 04:09:42.978619 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 13 04:09:42.978619 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 13 04:09:42.978619 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 13 04:09:42.978619 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application return self.get_or_create( Oct 13 04:09:42.978619 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 13 04:09:42.978619 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application with Lock( Oct 13 04:09:42.978619 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 13 04:09:42.978619 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application return self._enter() Oct 13 04:09:42.978619 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 13 04:09:42.978619 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 13 04:09:42.978619 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 13 04:09:42.978619 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application return self.creator() Oct 13 04:09:42.979564 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 13 04:09:42.979564 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application created_value = creator( Oct 13 04:09:42.979564 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Oct 13 04:09:42.979564 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Oct 13 04:09:42.979564 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Oct 13 04:09:42.979564 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Oct 13 04:09:42.979564 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Oct 13 04:09:42.979564 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Oct 13 04:09:42.979564 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: cinder. Oct 13 04:09:42.979564 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application Oct 13 04:09:42.979564 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 373/770] 10.222.0.27 () {56 vars in 1099 bytes} [Thu Oct 13 04:09:42 2022] GET /identity/v3/users/cinder => generated 84 bytes in 203 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Oct 13 04:09:42.984384 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7eb7b203-6606-4f14-b8b0-4aae7d40c712 None admin] Authenticating user token {{(pid=81408) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:09:43.062740 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG dbcounter [-] [81408] Writing DB stats keystone:INSERT=2,keystone:SELECT=98 {{(pid=81408) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:09:43.070395 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG dbcounter [-] [81409] Writing DB stats keystone:SELECT=100 {{(pid=81409) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:09:43.100768 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4d91879f-2931-4c3f-8aeb-449c245ebc1b None admin] Validating token access rules against request {{(pid=81409) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:09:43.126054 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7eb7b203-6606-4f14-b8b0-4aae7d40c712 None admin] Validating token access rules against request {{(pid=81408) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:09:43.145042 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6a703272-1ac4-45cf-84cd-c2469049fc0e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81409) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:09:43.145685 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a703272-1ac4-45cf-84cd-c2469049fc0e None admin] REQUEST_METHOD: `GET` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:09:43.145964 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a703272-1ac4-45cf-84cd-c2469049fc0e None admin] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:09:43.146254 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a703272-1ac4-45cf-84cd-c2469049fc0e None admin] PATH_INFO: `/v3/services` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:09:43.146868 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6a703272-1ac4-45cf-84cd-c2469049fc0e None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:09:43.148998 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6a703272-1ac4-45cf-84cd-c2469049fc0e None admin] RBAC: Authorization granted {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:09:43.158375 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 398/771] 10.222.0.27 () {56 vars in 1087 bytes} [Thu Oct 13 04:09:42 2022] GET /identity/v3/services => generated 1957 bytes in 195 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Oct 13 04:09:43.172372 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cebff417-2873-4946-b018-810b1aaac225 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81408) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:09:43.173222 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cebff417-2873-4946-b018-810b1aaac225 None admin] REQUEST_METHOD: `GET` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:09:43.173633 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cebff417-2873-4946-b018-810b1aaac225 None admin] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:09:43.173999 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cebff417-2873-4946-b018-810b1aaac225 None admin] PATH_INFO: `/v3/users` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:09:43.174865 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cebff417-2873-4946-b018-810b1aaac225 None admin] RBAC: Adding query filter params (name=cinder) {{(pid=81408) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 13 04:09:43.175268 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cebff417-2873-4946-b018-810b1aaac225 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:09:43.176845 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cebff417-2873-4946-b018-810b1aaac225 None admin] RBAC: Authorization granted {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:09:43.225425 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 374/772] 10.222.0.27 () {56 vars in 1101 bytes} [Thu Oct 13 04:09:42 2022] GET /identity/v3/users?name=cinder => generated 354 bytes in 241 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 13 04:09:43.422516 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf063c0b-ceb6-4ee6-90b6-53c2121270c4 None None] REQUEST_METHOD: `GET` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:09:43.422924 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf063c0b-ceb6-4ee6-90b6-53c2121270c4 None None] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:09:43.423309 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf063c0b-ceb6-4ee6-90b6-53c2121270c4 None None] PATH_INFO: `/` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:09:43.424914 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 399/773] 10.222.0.27 () {54 vars in 927 bytes} [Thu Oct 13 04:09:43 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 13 04:09:43.430453 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f952852-30e9-4692-9825-8994222b73e8 None None] REQUEST_METHOD: `POST` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:09:43.430738 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f952852-30e9-4692-9825-8994222b73e8 None None] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:09:43.431005 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f952852-30e9-4692-9825-8994222b73e8 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:09:43.451790 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.auth.core [None req-9f952852-30e9-4692-9825-8994222b73e8 None None] MFA Rules not processed for user `0e2d979563d6421995451a0e86bf3022`. Rule list: `[]` (Enabled: `True`). {{(pid=81408) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 13 04:09:43.453481 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.fernet_utils [None req-9f952852-30e9-4692-9825-8994222b73e8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81408) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:09:43.527638 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 375/774] 10.222.0.27 () {58 vars in 1024 bytes} [Thu Oct 13 04:09:43 2022] POST /identity/v3/auth/tokens => generated 1813 bytes in 98 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Oct 13 04:09:43.797447 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cf063c0b-ceb6-4ee6-90b6-53c2121270c4 None None] Authenticating user token {{(pid=81409) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:09:43.903235 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cf063c0b-ceb6-4ee6-90b6-53c2121270c4 None None] Validating token access rules against request {{(pid=81409) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:09:43.937886 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e179a32f-52fc-4147-b210-4692c0369dc4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81409) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:09:43.938534 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e179a32f-52fc-4147-b210-4692c0369dc4 None admin] REQUEST_METHOD: `GET` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:09:43.938807 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e179a32f-52fc-4147-b210-4692c0369dc4 None admin] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:09:43.939071 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e179a32f-52fc-4147-b210-4692c0369dc4 None admin] PATH_INFO: `/v3/domains/default` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:09:43.940174 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e179a32f-52fc-4147-b210-4692c0369dc4 None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:09:43.941592 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e179a32f-52fc-4147-b210-4692c0369dc4 None admin] RBAC: Authorization granted {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:09:43.946291 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 400/775] 10.222.0.27 () {56 vars in 1108 bytes} [Thu Oct 13 04:09:43 2022] GET /identity/v3/domains/default => generated 199 bytes in 146 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 13 04:09:43.949740 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9f952852-30e9-4692-9825-8994222b73e8 None None] Authenticating user token {{(pid=81408) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:09:43.995536 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG dbcounter [-] [81408] Writing DB stats keystone:SELECT=100 {{(pid=81408) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:09:44.051815 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9f952852-30e9-4692-9825-8994222b73e8 None None] Validating token access rules against request {{(pid=81408) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:09:44.091581 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fbc82375-4197-42e1-8359-1f5e91ae1564 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81408) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:09:44.092197 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fbc82375-4197-42e1-8359-1f5e91ae1564 None admin] REQUEST_METHOD: `POST` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:09:44.092473 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fbc82375-4197-42e1-8359-1f5e91ae1564 None admin] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:09:44.092741 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fbc82375-4197-42e1-8359-1f5e91ae1564 None admin] PATH_INFO: `/v3/projects` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:09:44.093413 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fbc82375-4197-42e1-8359-1f5e91ae1564 None admin] RBAC: Authorizing `identity:create_project()` {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:09:44.094923 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fbc82375-4197-42e1-8359-1f5e91ae1564 None admin] RBAC: Authorization granted {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:09:44.130238 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 376/776] 10.222.0.27 () {60 vars in 1141 bytes} [Thu Oct 13 04:09:43 2022] POST /identity/v3/projects => generated 311 bytes in 173 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 13 04:09:44.907789 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d995ffab-71de-4159-8332-3461fab98138 None None] REQUEST_METHOD: `GET` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:09:44.908353 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d995ffab-71de-4159-8332-3461fab98138 None None] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:09:44.908693 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d995ffab-71de-4159-8332-3461fab98138 None None] PATH_INFO: `/` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:09:44.911116 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 401/777] 10.222.0.27 () {54 vars in 927 bytes} [Thu Oct 13 04:09:44 2022] GET /identity => generated 270 bytes in 5 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 13 04:09:44.917405 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e1b7513-91e7-4bcf-81f3-3e17f1ff5099 None None] REQUEST_METHOD: `POST` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:09:44.917697 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e1b7513-91e7-4bcf-81f3-3e17f1ff5099 None None] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:09:44.917973 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e1b7513-91e7-4bcf-81f3-3e17f1ff5099 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:09:44.937730 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.auth.core [None req-2e1b7513-91e7-4bcf-81f3-3e17f1ff5099 None None] MFA Rules not processed for user `0e2d979563d6421995451a0e86bf3022`. Rule list: `[]` (Enabled: `True`). {{(pid=81408) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 13 04:09:44.938982 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.fernet_utils [None req-2e1b7513-91e7-4bcf-81f3-3e17f1ff5099 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81408) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:09:45.017517 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 377/778] 10.222.0.27 () {58 vars in 1024 bytes} [Thu Oct 13 04:09:44 2022] POST /identity/v3/auth/tokens => generated 1813 bytes in 101 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Oct 13 04:09:45.288884 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d995ffab-71de-4159-8332-3461fab98138 None None] Authenticating user token {{(pid=81409) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:09:45.303566 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG dbcounter [-] [81409] Writing DB stats keystone:SELECT=100 {{(pid=81409) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:09:45.391339 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23f47c09-1daa-42ab-ba97-c5211cb515fb None None] REQUEST_METHOD: `GET` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:09:45.391679 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23f47c09-1daa-42ab-ba97-c5211cb515fb None None] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:09:45.391893 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23f47c09-1daa-42ab-ba97-c5211cb515fb None None] PATH_INFO: `/` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:09:45.393447 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 378/779] 10.222.0.27 () {54 vars in 927 bytes} [Thu Oct 13 04:09:45 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Oct 13 04:09:45.398475 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ba49a5c-7fab-4c8f-97c5-d6f4c725d965 None None] REQUEST_METHOD: `POST` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:09:45.398796 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ba49a5c-7fab-4c8f-97c5-d6f4c725d965 None None] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:09:45.399077 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ba49a5c-7fab-4c8f-97c5-d6f4c725d965 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:09:45.419803 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d995ffab-71de-4159-8332-3461fab98138 None None] Validating token access rules against request {{(pid=81409) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:09:45.424050 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.auth.core [None req-7ba49a5c-7fab-4c8f-97c5-d6f4c725d965 None None] MFA Rules not processed for user `0e2d979563d6421995451a0e86bf3022`. Rule list: `[]` (Enabled: `True`). {{(pid=81408) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 13 04:09:45.425255 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.fernet_utils [None req-7ba49a5c-7fab-4c8f-97c5-d6f4c725d965 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81408) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:09:45.447841 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG dbcounter [-] [81408] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81408) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:09:45.464006 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-72f4403e-0ba2-41a8-84cf-652d9d658360 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81409) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:09:45.464990 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72f4403e-0ba2-41a8-84cf-652d9d658360 None admin] REQUEST_METHOD: `GET` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:09:45.465443 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72f4403e-0ba2-41a8-84cf-652d9d658360 None admin] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:09:45.465924 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72f4403e-0ba2-41a8-84cf-652d9d658360 None admin] PATH_INFO: `/v3/services/compute` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:09:45.466743 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-72f4403e-0ba2-41a8-84cf-652d9d658360 None admin] RBAC: Authorizing `identity:get_service(service_id=compute)` {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:09:45.470804 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-72f4403e-0ba2-41a8-84cf-652d9d658360 None admin] RBAC: Authorization granted {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:09:45.482087 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application [None req-72f4403e-0ba2-41a8-84cf-652d9d658360 None admin] Could not find service: compute.: keystone.exception.ServiceNotFound: Could not find service: compute. Oct 13 04:09:45.482087 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 13 04:09:45.482087 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Oct 13 04:09:45.482087 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Oct 13 04:09:45.482087 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Oct 13 04:09:45.482087 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Oct 13 04:09:45.482087 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Oct 13 04:09:45.482087 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 13 04:09:45.482087 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: compute. Oct 13 04:09:45.482087 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application Oct 13 04:09:45.482087 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Oct 13 04:09:45.482087 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application Oct 13 04:09:45.482087 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 13 04:09:45.482087 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 13 04:09:45.482087 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 13 04:09:45.482087 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 13 04:09:45.482087 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 13 04:09:45.483155 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 13 04:09:45.483155 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 13 04:09:45.483155 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 13 04:09:45.483155 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 13 04:09:45.483155 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 13 04:09:45.483155 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 13 04:09:45.483155 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Oct 13 04:09:45.483155 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application return self._get_service(service_id) Oct 13 04:09:45.483155 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Oct 13 04:09:45.483155 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Oct 13 04:09:45.483155 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 13 04:09:45.483155 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 13 04:09:45.483155 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 13 04:09:45.483155 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 13 04:09:45.483155 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 13 04:09:45.483155 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application return self.get_or_create( Oct 13 04:09:45.483993 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 13 04:09:45.483993 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application with Lock( Oct 13 04:09:45.483993 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 13 04:09:45.483993 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application return self._enter() Oct 13 04:09:45.483993 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 13 04:09:45.483993 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 13 04:09:45.483993 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 13 04:09:45.483993 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application return self.creator() Oct 13 04:09:45.483993 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 13 04:09:45.483993 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application created_value = creator( Oct 13 04:09:45.483993 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Oct 13 04:09:45.483993 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 13 04:09:45.483993 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: compute. Oct 13 04:09:45.483993 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application Oct 13 04:09:45.483993 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 402/780] 10.222.0.27 () {56 vars in 1111 bytes} [Thu Oct 13 04:09:45 2022] GET /identity/v3/services/compute => generated 88 bytes in 195 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 13 04:09:45.489402 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-72f4403e-0ba2-41a8-84cf-652d9d658360 None admin] Authenticating user token {{(pid=81409) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:09:45.506300 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 379/781] 10.222.0.27 () {58 vars in 1024 bytes} [Thu Oct 13 04:09:45 2022] POST /identity/v3/auth/tokens => generated 1813 bytes in 109 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Oct 13 04:09:45.578848 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG dbcounter [-] [81409] Writing DB stats keystone:SELECT=100 {{(pid=81409) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:09:45.615101 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-72f4403e-0ba2-41a8-84cf-652d9d658360 None admin] Validating token access rules against request {{(pid=81409) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:09:45.649399 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-964e1dbf-e67d-4017-854f-4a3cda791c5b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81409) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:09:45.650000 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-964e1dbf-e67d-4017-854f-4a3cda791c5b None admin] REQUEST_METHOD: `GET` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:09:45.650303 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-964e1dbf-e67d-4017-854f-4a3cda791c5b None admin] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:09:45.650576 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-964e1dbf-e67d-4017-854f-4a3cda791c5b None admin] PATH_INFO: `/v3/services` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:09:45.651203 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-964e1dbf-e67d-4017-854f-4a3cda791c5b None admin] RBAC: Adding query filter params (name=compute) {{(pid=81409) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 13 04:09:45.651505 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-964e1dbf-e67d-4017-854f-4a3cda791c5b None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:09:45.652859 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-964e1dbf-e67d-4017-854f-4a3cda791c5b None admin] RBAC: Authorization granted {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:09:45.662801 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 403/782] 10.222.0.27 () {56 vars in 1112 bytes} [Thu Oct 13 04:09:45 2022] GET /identity/v3/services?name=compute => generated 124 bytes in 169 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 13 04:09:45.663918 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7ba49a5c-7fab-4c8f-97c5-d6f4c725d965 None None] Authenticating user token {{(pid=81408) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:09:45.762611 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7ba49a5c-7fab-4c8f-97c5-d6f4c725d965 None None] Validating token access rules against request {{(pid=81408) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:09:45.778221 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-964e1dbf-e67d-4017-854f-4a3cda791c5b None admin] Authenticating user token {{(pid=81409) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:09:45.799027 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d2bd82c1-1017-4e03-8359-e63549a3784e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81408) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:09:45.799722 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2bd82c1-1017-4e03-8359-e63549a3784e None admin] REQUEST_METHOD: `GET` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:09:45.799920 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2bd82c1-1017-4e03-8359-e63549a3784e None admin] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:09:45.800174 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2bd82c1-1017-4e03-8359-e63549a3784e None admin] PATH_INFO: `/v3/services` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:09:45.800797 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d2bd82c1-1017-4e03-8359-e63549a3784e None admin] RBAC: Adding query filter params (type=compute) {{(pid=81408) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 13 04:09:45.801096 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d2bd82c1-1017-4e03-8359-e63549a3784e None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:09:45.802488 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d2bd82c1-1017-4e03-8359-e63549a3784e None admin] RBAC: Authorization granted {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:09:45.808421 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 380/783] 10.222.0.27 () {56 vars in 1112 bytes} [Thu Oct 13 04:09:45 2022] GET /identity/v3/services?type=compute => generated 352 bytes in 145 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 13 04:09:45.814505 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d2bd82c1-1017-4e03-8359-e63549a3784e None admin] Authenticating user token {{(pid=81408) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:09:45.835924 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG dbcounter [-] [81408] Writing DB stats keystone:SELECT=100 {{(pid=81408) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:09:45.896238 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-964e1dbf-e67d-4017-854f-4a3cda791c5b None admin] Validating token access rules against request {{(pid=81409) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:09:45.921301 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d2bd82c1-1017-4e03-8359-e63549a3784e None admin] Validating token access rules against request {{(pid=81408) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:09:45.935088 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3514f267-866e-4ae0-a070-746473c33420 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81409) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:09:45.935722 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3514f267-866e-4ae0-a070-746473c33420 None admin] REQUEST_METHOD: `GET` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:09:45.935999 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3514f267-866e-4ae0-a070-746473c33420 None admin] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:09:45.936264 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3514f267-866e-4ae0-a070-746473c33420 None admin] PATH_INFO: `/v3/services/image` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:09:45.936838 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3514f267-866e-4ae0-a070-746473c33420 None admin] RBAC: Authorizing `identity:get_service(service_id=image)` {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:09:45.938197 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3514f267-866e-4ae0-a070-746473c33420 None admin] RBAC: Authorization granted {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:09:45.940850 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG dbcounter [-] [81409] Writing DB stats keystone:SELECT=100 {{(pid=81409) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:09:45.948464 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application [None req-3514f267-866e-4ae0-a070-746473c33420 None admin] Could not find service: image.: keystone.exception.ServiceNotFound: Could not find service: image. Oct 13 04:09:45.948464 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 13 04:09:45.948464 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Oct 13 04:09:45.948464 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Oct 13 04:09:45.948464 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Oct 13 04:09:45.948464 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Oct 13 04:09:45.948464 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Oct 13 04:09:45.948464 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 13 04:09:45.948464 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: image. Oct 13 04:09:45.948464 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application Oct 13 04:09:45.948464 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Oct 13 04:09:45.948464 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application Oct 13 04:09:45.948464 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 13 04:09:45.948464 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 13 04:09:45.948464 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 13 04:09:45.948464 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 13 04:09:45.948464 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 13 04:09:45.949653 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 13 04:09:45.949653 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 13 04:09:45.949653 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 13 04:09:45.949653 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 13 04:09:45.949653 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 13 04:09:45.949653 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 13 04:09:45.949653 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Oct 13 04:09:45.949653 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application return self._get_service(service_id) Oct 13 04:09:45.949653 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Oct 13 04:09:45.949653 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Oct 13 04:09:45.949653 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 13 04:09:45.949653 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 13 04:09:45.949653 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 13 04:09:45.949653 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 13 04:09:45.949653 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 13 04:09:45.949653 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application return self.get_or_create( Oct 13 04:09:45.950691 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 13 04:09:45.950691 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application with Lock( Oct 13 04:09:45.950691 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 13 04:09:45.950691 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application return self._enter() Oct 13 04:09:45.950691 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 13 04:09:45.950691 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 13 04:09:45.950691 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 13 04:09:45.950691 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application return self.creator() Oct 13 04:09:45.950691 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 13 04:09:45.950691 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application created_value = creator( Oct 13 04:09:45.950691 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Oct 13 04:09:45.950691 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 13 04:09:45.950691 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: image. Oct 13 04:09:45.950691 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application Oct 13 04:09:45.954948 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 404/784] 10.222.0.27 () {56 vars in 1105 bytes} [Thu Oct 13 04:09:45 2022] GET /identity/v3/services/image => generated 86 bytes in 177 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Oct 13 04:09:45.955644 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3514f267-866e-4ae0-a070-746473c33420 None admin] Authenticating user token {{(pid=81409) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:09:45.967685 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1b03c21f-6904-4a11-ab84-37b87d731bd4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81408) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:09:45.968938 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b03c21f-6904-4a11-ab84-37b87d731bd4 None admin] REQUEST_METHOD: `POST` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:09:45.969454 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b03c21f-6904-4a11-ab84-37b87d731bd4 None admin] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:09:45.969889 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b03c21f-6904-4a11-ab84-37b87d731bd4 None admin] PATH_INFO: `/v3/endpoints` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:09:45.970901 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1b03c21f-6904-4a11-ab84-37b87d731bd4 None admin] RBAC: Authorizing `identity:create_endpoint()` {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:09:45.973073 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1b03c21f-6904-4a11-ab84-37b87d731bd4 None admin] RBAC: Authorization granted {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:09:46.000506 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 381/785] 10.222.0.27 () {60 vars in 1144 bytes} [Thu Oct 13 04:09:45 2022] POST /identity/v3/endpoints => generated 334 bytes in 186 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 13 04:09:46.063917 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3514f267-866e-4ae0-a070-746473c33420 None admin] Validating token access rules against request {{(pid=81409) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:09:46.110836 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-95e3b315-afe0-4df8-adde-950645e43a5e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81409) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:09:46.111348 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95e3b315-afe0-4df8-adde-950645e43a5e None admin] REQUEST_METHOD: `GET` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:09:46.111622 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95e3b315-afe0-4df8-adde-950645e43a5e None admin] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:09:46.111887 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95e3b315-afe0-4df8-adde-950645e43a5e None admin] PATH_INFO: `/v3/services` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:09:46.112544 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-95e3b315-afe0-4df8-adde-950645e43a5e None admin] RBAC: Adding query filter params (name=image) {{(pid=81409) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 13 04:09:46.112847 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-95e3b315-afe0-4df8-adde-950645e43a5e None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:09:46.114241 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-95e3b315-afe0-4df8-adde-950645e43a5e None admin] RBAC: Authorization granted {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:09:46.124731 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 405/786] 10.222.0.27 () {56 vars in 1108 bytes} [Thu Oct 13 04:09:45 2022] GET /identity/v3/services?name=image => generated 122 bytes in 165 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 13 04:09:46.126669 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1b03c21f-6904-4a11-ab84-37b87d731bd4 None admin] Authenticating user token {{(pid=81408) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:09:46.209552 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG dbcounter [-] [81408] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81408) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:09:46.237729 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1b03c21f-6904-4a11-ab84-37b87d731bd4 None admin] Validating token access rules against request {{(pid=81408) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:09:46.276907 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-14488e4d-cee9-4044-acd0-eccef2ac916f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81408) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:09:46.277515 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14488e4d-cee9-4044-acd0-eccef2ac916f None admin] REQUEST_METHOD: `GET` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:09:46.277785 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14488e4d-cee9-4044-acd0-eccef2ac916f None admin] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:09:46.278049 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14488e4d-cee9-4044-acd0-eccef2ac916f None admin] PATH_INFO: `/v3/services` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:09:46.278707 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-14488e4d-cee9-4044-acd0-eccef2ac916f None admin] RBAC: Adding query filter params (type=image) {{(pid=81408) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 13 04:09:46.279006 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-14488e4d-cee9-4044-acd0-eccef2ac916f None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:09:46.280364 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-14488e4d-cee9-4044-acd0-eccef2ac916f None admin] RBAC: Authorization granted {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:09:46.285875 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 382/787] 10.222.0.27 () {56 vars in 1108 bytes} [Thu Oct 13 04:09:46 2022] GET /identity/v3/services?type=image => generated 350 bytes in 160 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 13 04:09:46.291837 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-95e3b315-afe0-4df8-adde-950645e43a5e None admin] Authenticating user token {{(pid=81409) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:09:46.352688 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d589f85-0222-4410-b93e-275608794656 None None] REQUEST_METHOD: `GET` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:09:46.353015 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d589f85-0222-4410-b93e-275608794656 None None] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:09:46.353240 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d589f85-0222-4410-b93e-275608794656 None None] PATH_INFO: `/` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:09:46.354772 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 383/788] 10.222.0.27 () {54 vars in 927 bytes} [Thu Oct 13 04:09:46 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Oct 13 04:09:46.356381 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG dbcounter [-] [81409] Writing DB stats keystone:SELECT=100 {{(pid=81409) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:09:46.359930 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd945338-5cb8-4400-8c8c-2ff3cae2c184 None None] REQUEST_METHOD: `POST` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:09:46.360214 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd945338-5cb8-4400-8c8c-2ff3cae2c184 None None] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:09:46.360485 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd945338-5cb8-4400-8c8c-2ff3cae2c184 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:09:46.382324 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.auth.core [None req-fd945338-5cb8-4400-8c8c-2ff3cae2c184 None None] MFA Rules not processed for user `0e2d979563d6421995451a0e86bf3022`. Rule list: `[]` (Enabled: `True`). {{(pid=81408) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 13 04:09:46.383939 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.fernet_utils [None req-fd945338-5cb8-4400-8c8c-2ff3cae2c184 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81408) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:09:46.414849 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-95e3b315-afe0-4df8-adde-950645e43a5e None admin] Validating token access rules against request {{(pid=81409) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:09:46.451210 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a3933594-0e2a-4fde-9dab-8532e86c2e36 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81409) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:09:46.451815 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3933594-0e2a-4fde-9dab-8532e86c2e36 None admin] REQUEST_METHOD: `POST` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:09:46.452097 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3933594-0e2a-4fde-9dab-8532e86c2e36 None admin] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:09:46.452357 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3933594-0e2a-4fde-9dab-8532e86c2e36 None admin] PATH_INFO: `/v3/endpoints` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:09:46.453026 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a3933594-0e2a-4fde-9dab-8532e86c2e36 None admin] RBAC: Authorizing `identity:create_endpoint()` {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:09:46.454417 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a3933594-0e2a-4fde-9dab-8532e86c2e36 None admin] RBAC: Authorization granted {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:09:46.456996 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 384/789] 10.222.0.27 () {58 vars in 1024 bytes} [Thu Oct 13 04:09:46 2022] POST /identity/v3/auth/tokens => generated 1969 bytes in 98 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Oct 13 04:09:46.475182 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 406/790] 10.222.0.27 () {60 vars in 1144 bytes} [Thu Oct 13 04:09:46 2022] POST /identity/v3/endpoints => generated 327 bytes in 184 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 13 04:09:46.729269 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fd945338-5cb8-4400-8c8c-2ff3cae2c184 None None] Authenticating user token {{(pid=81408) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:09:46.802874 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG dbcounter [-] [81408] Writing DB stats keystone:SELECT=100 {{(pid=81408) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:09:46.835938 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fd945338-5cb8-4400-8c8c-2ff3cae2c184 None None] Validating token access rules against request {{(pid=81408) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:09:46.878656 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d52aa66d-f358-48da-996c-14d09f84b9c0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81408) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:09:46.879289 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d52aa66d-f358-48da-996c-14d09f84b9c0 None admin] REQUEST_METHOD: `GET` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:09:46.879564 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d52aa66d-f358-48da-996c-14d09f84b9c0 None admin] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:09:46.879835 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d52aa66d-f358-48da-996c-14d09f84b9c0 None admin] PATH_INFO: `/v3/domains/default` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:09:46.880941 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d52aa66d-f358-48da-996c-14d09f84b9c0 None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:09:46.882477 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d52aa66d-f358-48da-996c-14d09f84b9c0 None admin] RBAC: Authorization granted {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:09:46.886238 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 385/791] 10.222.0.27 () {56 vars in 1108 bytes} [Thu Oct 13 04:09:46 2022] GET /identity/v3/domains/default => generated 199 bytes in 156 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 13 04:09:46.889801 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a3933594-0e2a-4fde-9dab-8532e86c2e36 None admin] Authenticating user token {{(pid=81409) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:09:46.997682 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a3933594-0e2a-4fde-9dab-8532e86c2e36 None admin] Validating token access rules against request {{(pid=81409) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:09:47.015792 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG dbcounter [-] [81409] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81409) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:09:47.035628 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e4466567-2ef6-4863-9c1e-b6072ec492af None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81409) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:09:47.036710 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4466567-2ef6-4863-9c1e-b6072ec492af None admin] REQUEST_METHOD: `POST` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:09:47.036981 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4466567-2ef6-4863-9c1e-b6072ec492af None admin] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:09:47.037245 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4466567-2ef6-4863-9c1e-b6072ec492af None admin] PATH_INFO: `/v3/users` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:09:47.037893 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e4466567-2ef6-4863-9c1e-b6072ec492af None admin] RBAC: Authorizing `identity:create_user()` {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:09:47.039891 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e4466567-2ef6-4863-9c1e-b6072ec492af None admin] RBAC: Authorization granted {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:09:47.070234 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 407/792] 10.222.0.27 () {60 vars in 1132 bytes} [Thu Oct 13 04:09:46 2022] POST /identity/v3/users => generated 285 bytes in 174 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 13 04:09:48.256427 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-870c88ca-1ae3-4fdb-8031-7cb601fa9a9c None None] REQUEST_METHOD: `GET` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:09:48.256994 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-870c88ca-1ae3-4fdb-8031-7cb601fa9a9c None None] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:09:48.256994 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-870c88ca-1ae3-4fdb-8031-7cb601fa9a9c None None] PATH_INFO: `/` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:09:48.258679 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 386/793] 10.222.0.27 () {54 vars in 927 bytes} [Thu Oct 13 04:09:48 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 13 04:09:48.263515 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4ad0941-4d1d-40f8-a357-9409f9fe2289 None None] REQUEST_METHOD: `POST` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:09:48.263923 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4ad0941-4d1d-40f8-a357-9409f9fe2289 None None] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:09:48.264092 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4ad0941-4d1d-40f8-a357-9409f9fe2289 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:09:48.283295 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.auth.core [None req-e4ad0941-4d1d-40f8-a357-9409f9fe2289 None None] MFA Rules not processed for user `0e2d979563d6421995451a0e86bf3022`. Rule list: `[]` (Enabled: `True`). {{(pid=81409) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 13 04:09:48.284484 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.fernet_utils [None req-e4ad0941-4d1d-40f8-a357-9409f9fe2289 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81409) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:09:48.351894 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 408/794] 10.222.0.27 () {58 vars in 1024 bytes} [Thu Oct 13 04:09:48 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 89 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Oct 13 04:09:48.618182 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-870c88ca-1ae3-4fdb-8031-7cb601fa9a9c None None] Authenticating user token {{(pid=81408) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:09:48.662125 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03b718fb-e2d4-4d72-bd1c-7d76c601fb1a None None] REQUEST_METHOD: `GET` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:09:48.662814 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03b718fb-e2d4-4d72-bd1c-7d76c601fb1a None None] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:09:48.663094 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03b718fb-e2d4-4d72-bd1c-7d76c601fb1a None None] PATH_INFO: `/` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:09:48.665543 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 409/795] 10.222.0.27 () {54 vars in 927 bytes} [Thu Oct 13 04:09:48 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 13 04:09:48.670879 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64531490-a244-4ade-aec5-328ec1d07f76 None None] REQUEST_METHOD: `POST` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:09:48.671163 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64531490-a244-4ade-aec5-328ec1d07f76 None None] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:09:48.671428 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64531490-a244-4ade-aec5-328ec1d07f76 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:09:48.689903 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.auth.core [None req-64531490-a244-4ade-aec5-328ec1d07f76 None None] MFA Rules not processed for user `0e2d979563d6421995451a0e86bf3022`. Rule list: `[]` (Enabled: `True`). {{(pid=81409) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 13 04:09:48.691131 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.fernet_utils [None req-64531490-a244-4ade-aec5-328ec1d07f76 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81409) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:09:48.724870 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-870c88ca-1ae3-4fdb-8031-7cb601fa9a9c None None] Validating token access rules against request {{(pid=81408) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:09:48.759842 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 410/796] 10.222.0.27 () {58 vars in 1024 bytes} [Thu Oct 13 04:09:48 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 90 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Oct 13 04:09:48.762085 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9ea5b0bf-679c-4f3f-9841-89f437c2e149 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81408) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:09:48.762738 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ea5b0bf-679c-4f3f-9841-89f437c2e149 None admin] REQUEST_METHOD: `GET` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:09:48.763011 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ea5b0bf-679c-4f3f-9841-89f437c2e149 None admin] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:09:48.763276 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ea5b0bf-679c-4f3f-9841-89f437c2e149 None admin] PATH_INFO: `/v3/roles/ResellerAdmin` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:09:48.765428 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG dbcounter [-] [81408] Writing DB stats keystone:SELECT=100 {{(pid=81408) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:09:48.775298 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9ea5b0bf-679c-4f3f-9841-89f437c2e149 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 13 04:09:48.780040 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application [None req-9ea5b0bf-679c-4f3f-9841-89f437c2e149 None admin] Could not find role: ResellerAdmin.: keystone.exception.RoleNotFound: Could not find role: ResellerAdmin. Oct 13 04:09:48.780040 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 13 04:09:48.780040 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 13 04:09:48.780040 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 13 04:09:48.780040 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 13 04:09:48.780040 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 13 04:09:48.780040 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 13 04:09:48.780040 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 13 04:09:48.780040 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 13 04:09:48.780040 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 13 04:09:48.780040 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 13 04:09:48.780040 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 13 04:09:48.780040 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 13 04:09:48.780040 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 13 04:09:48.780040 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 13 04:09:48.780040 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 13 04:09:48.780940 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 13 04:09:48.780940 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 13 04:09:48.780940 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 13 04:09:48.780940 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 13 04:09:48.780940 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 13 04:09:48.780940 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 13 04:09:48.780940 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 13 04:09:48.780940 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 13 04:09:48.780940 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 13 04:09:48.780940 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application return self.get_or_create( Oct 13 04:09:48.780940 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 13 04:09:48.780940 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application with Lock( Oct 13 04:09:48.780940 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 13 04:09:48.780940 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application return self._enter() Oct 13 04:09:48.780940 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 13 04:09:48.780940 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 13 04:09:48.781939 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 13 04:09:48.781939 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application return self.creator() Oct 13 04:09:48.781939 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 13 04:09:48.781939 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application created_value = creator( Oct 13 04:09:48.781939 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 13 04:09:48.781939 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 13 04:09:48.781939 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 13 04:09:48.781939 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 13 04:09:48.781939 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 13 04:09:48.781939 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 13 04:09:48.781939 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: ResellerAdmin. Oct 13 04:09:48.781939 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application Oct 13 04:09:48.781939 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 387/797] 10.222.0.27 () {56 vars in 1120 bytes} [Thu Oct 13 04:09:48 2022] GET /identity/v3/roles/ResellerAdmin => generated 91 bytes in 164 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 13 04:09:48.787100 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-64531490-a244-4ade-aec5-328ec1d07f76 None None] Authenticating user token {{(pid=81409) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:09:48.820003 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG dbcounter [-] [81409] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81409) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:09:48.893823 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-64531490-a244-4ade-aec5-328ec1d07f76 None None] Validating token access rules against request {{(pid=81409) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:09:48.927594 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c0d78110-8307-402b-8538-b8bc757a6c3c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81409) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:09:48.928200 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0d78110-8307-402b-8538-b8bc757a6c3c None admin] REQUEST_METHOD: `GET` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:09:48.928476 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0d78110-8307-402b-8538-b8bc757a6c3c None admin] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:09:48.928748 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0d78110-8307-402b-8538-b8bc757a6c3c None admin] PATH_INFO: `/v3/roles` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:09:48.929405 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c0d78110-8307-402b-8538-b8bc757a6c3c None admin] RBAC: Adding query filter params (name=ResellerAdmin) {{(pid=81409) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 13 04:09:48.930228 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c0d78110-8307-402b-8538-b8bc757a6c3c None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:09:48.931599 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c0d78110-8307-402b-8538-b8bc757a6c3c None admin] RBAC: Authorization granted {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:09:48.944266 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 411/798] 10.222.0.27 () {56 vars in 1115 bytes} [Thu Oct 13 04:09:48 2022] GET /identity/v3/roles?name=ResellerAdmin => generated 338 bytes in 153 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 13 04:09:48.946618 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9ea5b0bf-679c-4f3f-9841-89f437c2e149 None admin] Authenticating user token {{(pid=81408) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:09:49.035543 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c0d78110-8307-402b-8538-b8bc757a6c3c None admin] Authenticating user token {{(pid=81409) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:09:49.050481 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9ea5b0bf-679c-4f3f-9841-89f437c2e149 None admin] Validating token access rules against request {{(pid=81408) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:09:49.090065 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3d641c99-e52d-4f46-ad42-c88351d49beb None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81408) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:09:49.090723 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d641c99-e52d-4f46-ad42-c88351d49beb None admin] REQUEST_METHOD: `GET` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:09:49.091003 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d641c99-e52d-4f46-ad42-c88351d49beb None admin] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:09:49.091313 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d641c99-e52d-4f46-ad42-c88351d49beb None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:09:49.102014 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3d641c99-e52d-4f46-ad42-c88351d49beb None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:09:49.103530 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3d641c99-e52d-4f46-ad42-c88351d49beb None admin] RBAC: Authorization granted {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:09:49.119037 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application [None req-3d641c99-e52d-4f46-ad42-c88351d49beb None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 13 04:09:49.119037 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 13 04:09:49.119037 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 13 04:09:49.119037 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 13 04:09:49.119037 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 13 04:09:49.119037 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 13 04:09:49.119037 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 13 04:09:49.119037 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 13 04:09:49.119037 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 13 04:09:49.119037 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 13 04:09:49.119037 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 13 04:09:49.119037 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 13 04:09:49.119037 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 13 04:09:49.119037 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 13 04:09:49.119037 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 13 04:09:49.119037 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 13 04:09:49.120057 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 13 04:09:49.120057 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 13 04:09:49.120057 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 13 04:09:49.120057 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 13 04:09:49.120057 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 13 04:09:49.120057 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application return self.get_or_create( Oct 13 04:09:49.120057 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 13 04:09:49.120057 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application with Lock( Oct 13 04:09:49.120057 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 13 04:09:49.120057 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application return self._enter() Oct 13 04:09:49.120057 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 13 04:09:49.120057 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 13 04:09:49.120057 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 13 04:09:49.120057 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application return self.creator() Oct 13 04:09:49.120057 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 13 04:09:49.120057 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application created_value = creator( Oct 13 04:09:49.120057 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 13 04:09:49.120963 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 13 04:09:49.120963 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 13 04:09:49.120963 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application Oct 13 04:09:49.120963 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 388/799] 10.222.0.27 () {56 vars in 1108 bytes} [Thu Oct 13 04:09:48 2022] GET /identity/v3/domains/Default => generated 87 bytes in 175 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 13 04:09:49.126170 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3d641c99-e52d-4f46-ad42-c88351d49beb None admin] Authenticating user token {{(pid=81408) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:09:49.140424 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG dbcounter [-] [81409] Writing DB stats keystone:SELECT=100 {{(pid=81409) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:09:49.149280 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c0d78110-8307-402b-8538-b8bc757a6c3c None admin] Validating token access rules against request {{(pid=81409) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:09:49.181697 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG dbcounter [-] [81408] Writing DB stats keystone:SELECT=100 {{(pid=81408) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:09:49.188367 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-20c353ba-4ee3-487c-8c55-b8181b55e542 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81409) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:09:49.189503 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20c353ba-4ee3-487c-8c55-b8181b55e542 None admin] REQUEST_METHOD: `POST` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:09:49.189778 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20c353ba-4ee3-487c-8c55-b8181b55e542 None admin] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:09:49.190044 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20c353ba-4ee3-487c-8c55-b8181b55e542 None admin] PATH_INFO: `/v3/domains` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:09:49.191008 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-20c353ba-4ee3-487c-8c55-b8181b55e542 None admin] RBAC: Authorizing `identity:create_domain()` {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:09:49.192580 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-20c353ba-4ee3-487c-8c55-b8181b55e542 None admin] RBAC: Authorization granted {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:09:49.202171 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.sql.core [None req-20c353ba-4ee3-487c-8c55-b8181b55e542 None admin] Conflict project: (pymysql.err.IntegrityError) (1062, "Duplicate entry '<>-Default' for key 'project.ixu_project_name_domain_id'") Oct 13 04:09:49.202171 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [SQL: INSERT INTO project (id, name, domain_id, description, enabled, extra, parent_id, is_domain) VALUES (%(id)s, %(name)s, %(domain_id)s, %(description)s, %(enabled)s, %(extra)s, %(parent_id)s, %(is_domain)s)] Oct 13 04:09:49.202171 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [parameters: {'id': '3f7328b315ab48d0841b555198a03378', 'name': 'Default', 'domain_id': '<>', 'description': '', 'enabled': 1, 'extra': '{}', 'parent_id': None, 'is_domain': 1}] Oct 13 04:09:49.202171 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: (Background on this error at: https://sqlalche.me/e/14/gkpj) {{(pid=81409) wrapper /opt/stack/keystone/keystone/common/sql/core.py:533}} Oct 13 04:09:49.210759 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application [None req-20c353ba-4ee3-487c-8c55-b8181b55e542 None admin] Conflict occurred attempting to store project - it is not permitted to have two projects acting as domains with the same name: Default.: keystone.exception.Conflict: Conflict occurred attempting to store project - it is not permitted to have two projects acting as domains with the same name: Default. Oct 13 04:09:49.210759 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 13 04:09:49.210759 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1900, in _execute_context Oct 13 04:09:49.210759 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application self.dialect.do_execute( Oct 13 04:09:49.210759 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/default.py", line 732, in do_execute Oct 13 04:09:49.210759 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application cursor.execute(statement, parameters) Oct 13 04:09:49.210759 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/cursors.py", line 148, in execute Oct 13 04:09:49.210759 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application result = self._query(query) Oct 13 04:09:49.210759 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/cursors.py", line 310, in _query Oct 13 04:09:49.210759 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application conn.query(q) Oct 13 04:09:49.210759 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 548, in query Oct 13 04:09:49.210759 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application self._affected_rows = self._read_query_result(unbuffered=unbuffered) Oct 13 04:09:49.210759 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 775, in _read_query_result Oct 13 04:09:49.210759 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application result.read() Oct 13 04:09:49.212135 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 1156, in read Oct 13 04:09:49.212135 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application first_packet = self.connection._read_packet() Oct 13 04:09:49.212135 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 725, in _read_packet Oct 13 04:09:49.212135 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application packet.raise_for_error() Oct 13 04:09:49.212135 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/protocol.py", line 221, in raise_for_error Oct 13 04:09:49.212135 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application err.raise_mysql_exception(self._data) Oct 13 04:09:49.212135 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/err.py", line 143, in raise_mysql_exception Oct 13 04:09:49.212135 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application raise errorclass(errno, errval) Oct 13 04:09:49.212135 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application pymysql.err.IntegrityError: (1062, "Duplicate entry '<>-Default' for key 'project.ixu_project_name_domain_id'") Oct 13 04:09:49.212135 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application Oct 13 04:09:49.212135 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application The above exception was the direct cause of the following exception: Oct 13 04:09:49.212135 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application Oct 13 04:09:49.212135 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 13 04:09:49.212135 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/sql/core.py", line 528, in wrapper Oct 13 04:09:49.212135 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application return method(*args, **kwargs) Oct 13 04:09:49.212135 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 241, in create_project Oct 13 04:09:49.212135 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application return project_ref.to_dict() Oct 13 04:09:49.212135 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ Oct 13 04:09:49.213107 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application next(self.gen) Oct 13 04:09:49.213107 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 1060, in _transaction_scope Oct 13 04:09:49.213107 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application yield resource Oct 13 04:09:49.213107 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ Oct 13 04:09:49.213107 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application next(self.gen) Oct 13 04:09:49.213107 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 662, in _session Oct 13 04:09:49.213107 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application self.session.rollback() Oct 13 04:09:49.213107 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_utils/excutils.py", line 227, in __exit__ Oct 13 04:09:49.213107 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application self.force_reraise() Oct 13 04:09:49.213107 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_utils/excutils.py", line 200, in force_reraise Oct 13 04:09:49.213107 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application raise self.value Oct 13 04:09:49.213107 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 659, in _session Oct 13 04:09:49.213107 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application self._end_session_transaction(self.session) Oct 13 04:09:49.213107 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 687, in _end_session_transaction Oct 13 04:09:49.213107 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application session.commit() Oct 13 04:09:49.213107 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 1451, in commit Oct 13 04:09:49.213107 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application self._transaction.commit(_to_root=self.future) Oct 13 04:09:49.213989 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 829, in commit Oct 13 04:09:49.213989 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application self._prepare_impl() Oct 13 04:09:49.213989 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 808, in _prepare_impl Oct 13 04:09:49.213989 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application self.session.flush() Oct 13 04:09:49.213989 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 3383, in flush Oct 13 04:09:49.213989 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application self._flush(objects) Oct 13 04:09:49.213989 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 3523, in _flush Oct 13 04:09:49.213989 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application transaction.rollback(_capture_exception=True) Oct 13 04:09:49.213989 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/util/langhelpers.py", line 70, in __exit__ Oct 13 04:09:49.213989 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application compat.raise_( Oct 13 04:09:49.213989 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/util/compat.py", line 208, in raise_ Oct 13 04:09:49.213989 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application raise exception Oct 13 04:09:49.213989 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 3483, in _flush Oct 13 04:09:49.213989 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application flush_context.execute() Oct 13 04:09:49.213989 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/unitofwork.py", line 456, in execute Oct 13 04:09:49.213989 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application rec.execute(self) Oct 13 04:09:49.213989 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/unitofwork.py", line 630, in execute Oct 13 04:09:49.215799 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application util.preloaded.orm_persistence.save_obj( Oct 13 04:09:49.215799 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/persistence.py", line 245, in save_obj Oct 13 04:09:49.215799 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application _emit_insert_statements( Oct 13 04:09:49.215799 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/persistence.py", line 1097, in _emit_insert_statements Oct 13 04:09:49.215799 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application c = connection._execute_20( Oct 13 04:09:49.215799 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1705, in _execute_20 Oct 13 04:09:49.215799 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application return meth(self, args_10style, kwargs_10style, execution_options) Oct 13 04:09:49.215799 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/sql/elements.py", line 333, in _execute_on_connection Oct 13 04:09:49.215799 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application return connection._execute_clauseelement( Oct 13 04:09:49.215799 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1572, in _execute_clauseelement Oct 13 04:09:49.215799 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application ret = self._execute_context( Oct 13 04:09:49.215799 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1943, in _execute_context Oct 13 04:09:49.215799 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application self._handle_dbapi_exception( Oct 13 04:09:49.215799 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 2122, in _handle_dbapi_exception Oct 13 04:09:49.215799 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application util.raise_(newraise, with_traceback=exc_info[2], from_=e) Oct 13 04:09:49.216755 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/util/compat.py", line 208, in raise_ Oct 13 04:09:49.216755 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application raise exception Oct 13 04:09:49.216755 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1900, in _execute_context Oct 13 04:09:49.216755 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application self.dialect.do_execute( Oct 13 04:09:49.216755 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/default.py", line 732, in do_execute Oct 13 04:09:49.216755 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application cursor.execute(statement, parameters) Oct 13 04:09:49.216755 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/cursors.py", line 148, in execute Oct 13 04:09:49.216755 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application result = self._query(query) Oct 13 04:09:49.216755 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/cursors.py", line 310, in _query Oct 13 04:09:49.216755 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application conn.query(q) Oct 13 04:09:49.216755 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 548, in query Oct 13 04:09:49.216755 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application self._affected_rows = self._read_query_result(unbuffered=unbuffered) Oct 13 04:09:49.216755 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 775, in _read_query_result Oct 13 04:09:49.216755 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application result.read() Oct 13 04:09:49.216755 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 1156, in read Oct 13 04:09:49.216755 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application first_packet = self.connection._read_packet() Oct 13 04:09:49.216755 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 725, in _read_packet Oct 13 04:09:49.218918 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application packet.raise_for_error() Oct 13 04:09:49.218918 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/protocol.py", line 221, in raise_for_error Oct 13 04:09:49.218918 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application err.raise_mysql_exception(self._data) Oct 13 04:09:49.218918 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/err.py", line 143, in raise_mysql_exception Oct 13 04:09:49.218918 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application raise errorclass(errno, errval) Oct 13 04:09:49.218918 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application oslo_db.exception.DBDuplicateEntry: (pymysql.err.IntegrityError) (1062, "Duplicate entry '<>-Default' for key 'project.ixu_project_name_domain_id'") Oct 13 04:09:49.218918 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application [SQL: INSERT INTO project (id, name, domain_id, description, enabled, extra, parent_id, is_domain) VALUES (%(id)s, %(name)s, %(domain_id)s, %(description)s, %(enabled)s, %(extra)s, %(parent_id)s, %(is_domain)s)] Oct 13 04:09:49.218918 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application [parameters: {'id': '3f7328b315ab48d0841b555198a03378', 'name': 'Default', 'domain_id': '<>', 'description': '', 'enabled': 1, 'extra': '{}', 'parent_id': None, 'is_domain': 1}] Oct 13 04:09:49.218918 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application (Background on this error at: https://sqlalche.me/e/14/gkpj) Oct 13 04:09:49.218918 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application Oct 13 04:09:49.218918 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Oct 13 04:09:49.218918 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application Oct 13 04:09:49.218918 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 13 04:09:49.218918 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 216, in create_project Oct 13 04:09:49.218918 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application ret = self.driver.create_project(project_id, project) Oct 13 04:09:49.221582 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/sql/core.py", line 563, in wrapper Oct 13 04:09:49.221582 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application raise exception.Conflict(type=conflict_type, Oct 13 04:09:49.221582 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application keystone.exception.Conflict: Conflict occurred attempting to store project - Duplicate entry found with name Default. Oct 13 04:09:49.221582 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application Oct 13 04:09:49.221582 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Oct 13 04:09:49.221582 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application Oct 13 04:09:49.221582 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 13 04:09:49.221582 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 13 04:09:49.221582 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 13 04:09:49.221582 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 13 04:09:49.221582 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 13 04:09:49.221582 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 13 04:09:49.221582 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 13 04:09:49.221582 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 13 04:09:49.221582 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 13 04:09:49.221582 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 13 04:09:49.221582 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 13 04:09:49.223746 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 128, in post Oct 13 04:09:49.223746 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application ref = PROVIDERS.resource_api.create_domain( Oct 13 04:09:49.223746 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 13 04:09:49.223746 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 13 04:09:49.223746 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 765, in create_domain Oct 13 04:09:49.223746 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application is_domain_project = self.create_project( Oct 13 04:09:49.223746 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 13 04:09:49.223746 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 13 04:09:49.223746 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 218, in create_project Oct 13 04:09:49.223746 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application raise exception.Conflict( Oct 13 04:09:49.223746 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application keystone.exception.Conflict: Conflict occurred attempting to store project - it is not permitted to have two projects acting as domains with the same name: Default. Oct 13 04:09:49.223746 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application Oct 13 04:09:49.223746 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 412/800] 10.222.0.27 () {60 vars in 1137 bytes} [Thu Oct 13 04:09:49 2022] POST /identity/v3/domains => generated 190 bytes in 178 msecs (HTTP/1.1 409) 5 headers in 183 bytes (2 switches on core 0) Oct 13 04:09:49.223746 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-20c353ba-4ee3-487c-8c55-b8181b55e542 None admin] Authenticating user token {{(pid=81409) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:09:49.247689 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3d641c99-e52d-4f46-ad42-c88351d49beb None admin] Validating token access rules against request {{(pid=81408) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:09:49.283034 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e77f3bc-89c0-4a94-8f2d-f7e90a0fb543 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81408) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:09:49.283893 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e77f3bc-89c0-4a94-8f2d-f7e90a0fb543 None admin] REQUEST_METHOD: `GET` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:09:49.284202 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e77f3bc-89c0-4a94-8f2d-f7e90a0fb543 None admin] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:09:49.284514 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e77f3bc-89c0-4a94-8f2d-f7e90a0fb543 None admin] PATH_INFO: `/v3/domains` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:09:49.285135 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1e77f3bc-89c0-4a94-8f2d-f7e90a0fb543 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81408) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 13 04:09:49.285440 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1e77f3bc-89c0-4a94-8f2d-f7e90a0fb543 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:09:49.286871 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1e77f3bc-89c0-4a94-8f2d-f7e90a0fb543 None admin] RBAC: Authorization granted {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:09:49.304305 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 389/801] 10.222.0.27 () {56 vars in 1109 bytes} [Thu Oct 13 04:09:49 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 174 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 13 04:09:49.304305 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-daa8173a-426b-4076-8fc1-9f72e3d97e3f None None] REQUEST_METHOD: `GET` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:09:49.304305 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-daa8173a-426b-4076-8fc1-9f72e3d97e3f None None] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:09:49.304305 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-daa8173a-426b-4076-8fc1-9f72e3d97e3f None None] PATH_INFO: `/` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:09:49.304305 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 390/802] 10.222.0.27 () {54 vars in 927 bytes} [Thu Oct 13 04:09:49 2022] GET /identity => generated 270 bytes in 2 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 13 04:09:49.307082 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-daa8173a-426b-4076-8fc1-9f72e3d97e3f None None] Authenticating user token {{(pid=81408) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:09:49.326910 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-20c353ba-4ee3-487c-8c55-b8181b55e542 None admin] Validating token access rules against request {{(pid=81409) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:09:49.360682 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5859b7d6-ba11-440c-bd0e-c2b7fd4fa944 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81409) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:09:49.361303 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5859b7d6-ba11-440c-bd0e-c2b7fd4fa944 None admin] REQUEST_METHOD: `GET` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:09:49.361579 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5859b7d6-ba11-440c-bd0e-c2b7fd4fa944 None admin] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:09:49.361844 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5859b7d6-ba11-440c-bd0e-c2b7fd4fa944 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:09:49.372960 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5859b7d6-ba11-440c-bd0e-c2b7fd4fa944 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:09:49.374449 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5859b7d6-ba11-440c-bd0e-c2b7fd4fa944 None admin] RBAC: Authorization granted {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:09:49.388840 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application [None req-5859b7d6-ba11-440c-bd0e-c2b7fd4fa944 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 13 04:09:49.388840 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 13 04:09:49.388840 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 13 04:09:49.388840 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 13 04:09:49.388840 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 13 04:09:49.388840 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 13 04:09:49.388840 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 13 04:09:49.388840 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 13 04:09:49.388840 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 13 04:09:49.388840 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 13 04:09:49.388840 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 13 04:09:49.388840 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 13 04:09:49.388840 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 13 04:09:49.388840 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 13 04:09:49.388840 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 13 04:09:49.388840 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 13 04:09:49.389829 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 13 04:09:49.389829 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 13 04:09:49.389829 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 13 04:09:49.389829 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 13 04:09:49.389829 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 13 04:09:49.389829 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application return self.get_or_create( Oct 13 04:09:49.389829 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 13 04:09:49.389829 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application with Lock( Oct 13 04:09:49.389829 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 13 04:09:49.389829 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application return self._enter() Oct 13 04:09:49.389829 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 13 04:09:49.389829 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 13 04:09:49.389829 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 13 04:09:49.389829 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application return self.creator() Oct 13 04:09:49.389829 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 13 04:09:49.389829 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application created_value = creator( Oct 13 04:09:49.389829 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 13 04:09:49.391156 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 13 04:09:49.391156 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 13 04:09:49.391156 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application Oct 13 04:09:49.391156 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 413/803] 10.222.0.27 () {56 vars in 1108 bytes} [Thu Oct 13 04:09:49 2022] GET /identity/v3/domains/Default => generated 87 bytes in 172 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Oct 13 04:09:49.391156 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bfb8bb32-c8ce-4fb1-8e27-1fd2a59acef6 None None] REQUEST_METHOD: `POST` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:09:49.391534 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bfb8bb32-c8ce-4fb1-8e27-1fd2a59acef6 None None] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:09:49.391725 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bfb8bb32-c8ce-4fb1-8e27-1fd2a59acef6 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:09:49.411793 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.auth.core [None req-bfb8bb32-c8ce-4fb1-8e27-1fd2a59acef6 None None] MFA Rules not processed for user `0e2d979563d6421995451a0e86bf3022`. Rule list: `[]` (Enabled: `True`). {{(pid=81409) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 13 04:09:49.411949 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-daa8173a-426b-4076-8fc1-9f72e3d97e3f None None] Validating token access rules against request {{(pid=81408) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:09:49.412929 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.fernet_utils [None req-bfb8bb32-c8ce-4fb1-8e27-1fd2a59acef6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81409) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:09:49.416973 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG dbcounter [-] [81409] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81409) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:09:49.421462 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG dbcounter [-] [81408] Writing DB stats keystone:SELECT=100 {{(pid=81408) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:09:49.451222 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-22c4b4d4-4379-4de8-a990-75526027b070 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81408) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:09:49.451858 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22c4b4d4-4379-4de8-a990-75526027b070 None admin] REQUEST_METHOD: `GET` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:09:49.452132 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22c4b4d4-4379-4de8-a990-75526027b070 None admin] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:09:49.452401 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22c4b4d4-4379-4de8-a990-75526027b070 None admin] PATH_INFO: `/v3/users/nova` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:09:49.459638 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-22c4b4d4-4379-4de8-a990-75526027b070 None admin] RBAC: Authorizing `identity:get_user(user_id=nova)` {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:09:49.461129 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-22c4b4d4-4379-4de8-a990-75526027b070 None admin] RBAC: Authorization granted {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:09:49.474561 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application [None req-22c4b4d4-4379-4de8-a990-75526027b070 None admin] Could not find user: nova.: keystone.exception.UserNotFound: Could not find user: nova. Oct 13 04:09:49.474561 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 13 04:09:49.474561 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 13 04:09:49.474561 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 13 04:09:49.474561 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 13 04:09:49.474561 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 13 04:09:49.474561 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 13 04:09:49.474561 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 13 04:09:49.474561 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 13 04:09:49.474561 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 13 04:09:49.474561 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 13 04:09:49.474561 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 13 04:09:49.474561 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Oct 13 04:09:49.474561 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application return self._get_user(user_id) Oct 13 04:09:49.474561 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Oct 13 04:09:49.474561 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Oct 13 04:09:49.475742 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 13 04:09:49.475742 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 13 04:09:49.475742 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Oct 13 04:09:49.475742 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 13 04:09:49.475742 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Oct 13 04:09:49.475742 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 13 04:09:49.475742 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 13 04:09:49.475742 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 13 04:09:49.475742 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 13 04:09:49.475742 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application return self.get_or_create( Oct 13 04:09:49.475742 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 13 04:09:49.475742 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application with Lock( Oct 13 04:09:49.475742 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 13 04:09:49.475742 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application return self._enter() Oct 13 04:09:49.475742 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 13 04:09:49.475742 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 13 04:09:49.475742 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 13 04:09:49.475742 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application return self.creator() Oct 13 04:09:49.477736 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 13 04:09:49.477736 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application created_value = creator( Oct 13 04:09:49.477736 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Oct 13 04:09:49.477736 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Oct 13 04:09:49.477736 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Oct 13 04:09:49.477736 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Oct 13 04:09:49.477736 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Oct 13 04:09:49.477736 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Oct 13 04:09:49.477736 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: nova. Oct 13 04:09:49.477736 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application Oct 13 04:09:49.478535 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 391/804] 10.222.0.27 () {56 vars in 1093 bytes} [Thu Oct 13 04:09:49 2022] GET /identity/v3/users/nova => generated 82 bytes in 172 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 13 04:09:49.479546 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-22c4b4d4-4379-4de8-a990-75526027b070 None admin] Authenticating user token {{(pid=81408) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:09:49.584611 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-22c4b4d4-4379-4de8-a990-75526027b070 None admin] Validating token access rules against request {{(pid=81408) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:09:49.598246 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 414/805] 10.222.0.27 () {58 vars in 1024 bytes} [Thu Oct 13 04:09:49 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 208 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Oct 13 04:09:49.599249 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bfb8bb32-c8ce-4fb1-8e27-1fd2a59acef6 None None] Authenticating user token {{(pid=81409) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:09:49.619000 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5cb98042-700a-4e49-8ae7-388fb0c8a17c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81408) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:09:49.619606 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5cb98042-700a-4e49-8ae7-388fb0c8a17c None admin] REQUEST_METHOD: `GET` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:09:49.619883 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5cb98042-700a-4e49-8ae7-388fb0c8a17c None admin] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:09:49.620154 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5cb98042-700a-4e49-8ae7-388fb0c8a17c None admin] PATH_INFO: `/v3/domains` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:09:49.620798 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5cb98042-700a-4e49-8ae7-388fb0c8a17c None admin] RBAC: Adding query filter params (name=Default) {{(pid=81408) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 13 04:09:49.621098 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5cb98042-700a-4e49-8ae7-388fb0c8a17c None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:09:49.622525 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5cb98042-700a-4e49-8ae7-388fb0c8a17c None admin] RBAC: Authorization granted {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:09:49.635668 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 392/806] 10.222.0.27 () {56 vars in 1109 bytes} [Thu Oct 13 04:09:49 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 156 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 13 04:09:49.726957 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bfb8bb32-c8ce-4fb1-8e27-1fd2a59acef6 None None] Validating token access rules against request {{(pid=81409) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:09:49.764469 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3aedca91-5c5e-4e32-9788-f7c043e28078 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81409) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:09:49.765231 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3aedca91-5c5e-4e32-9788-f7c043e28078 None admin] REQUEST_METHOD: `GET` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:09:49.765599 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3aedca91-5c5e-4e32-9788-f7c043e28078 None admin] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:09:49.765959 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3aedca91-5c5e-4e32-9788-f7c043e28078 None admin] PATH_INFO: `/v3/users` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:09:49.766812 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3aedca91-5c5e-4e32-9788-f7c043e28078 None admin] RBAC: Adding query filter params (domain_id=default, name=nova) {{(pid=81409) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 13 04:09:49.767208 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3aedca91-5c5e-4e32-9788-f7c043e28078 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:09:49.768771 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3aedca91-5c5e-4e32-9788-f7c043e28078 None admin] RBAC: Authorization granted {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:09:49.779441 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG dbcounter [-] [81409] Writing DB stats keystone:SELECT=100 {{(pid=81409) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:09:49.790572 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 415/807] 10.222.0.27 () {56 vars in 1133 bytes} [Thu Oct 13 04:09:49 2022] GET /identity/v3/users?domain_id=default&name=nova => generated 368 bytes in 192 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 13 04:09:49.796729 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5cb98042-700a-4e49-8ae7-388fb0c8a17c None admin] Authenticating user token {{(pid=81408) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:09:49.828486 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG dbcounter [-] [81408] Writing DB stats keystone:SELECT=100 {{(pid=81408) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:09:49.873937 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3aedca91-5c5e-4e32-9788-f7c043e28078 None admin] Authenticating user token {{(pid=81409) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:09:49.908431 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5cb98042-700a-4e49-8ae7-388fb0c8a17c None admin] Validating token access rules against request {{(pid=81408) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:09:49.946340 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-885bf037-94ee-4fff-9cec-eeeaa290d077 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81408) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:09:49.947023 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-885bf037-94ee-4fff-9cec-eeeaa290d077 None admin] REQUEST_METHOD: `GET` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:09:49.947305 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-885bf037-94ee-4fff-9cec-eeeaa290d077 None admin] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:09:49.947572 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-885bf037-94ee-4fff-9cec-eeeaa290d077 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:09:49.957725 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-885bf037-94ee-4fff-9cec-eeeaa290d077 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:09:49.959199 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-885bf037-94ee-4fff-9cec-eeeaa290d077 None admin] RBAC: Authorization granted {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:09:49.975533 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application [None req-885bf037-94ee-4fff-9cec-eeeaa290d077 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 13 04:09:49.975533 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 13 04:09:49.975533 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 13 04:09:49.975533 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 13 04:09:49.975533 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 13 04:09:49.975533 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 13 04:09:49.975533 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 13 04:09:49.975533 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 13 04:09:49.975533 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 13 04:09:49.975533 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 13 04:09:49.975533 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 13 04:09:49.975533 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 13 04:09:49.975533 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 13 04:09:49.975533 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 13 04:09:49.975533 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 13 04:09:49.975533 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 13 04:09:49.977758 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 13 04:09:49.977758 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 13 04:09:49.977758 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 13 04:09:49.977758 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 13 04:09:49.977758 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 13 04:09:49.977758 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application return self.get_or_create( Oct 13 04:09:49.977758 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 13 04:09:49.977758 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application with Lock( Oct 13 04:09:49.977758 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 13 04:09:49.977758 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application return self._enter() Oct 13 04:09:49.977758 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 13 04:09:49.977758 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 13 04:09:49.977758 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 13 04:09:49.977758 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application return self.creator() Oct 13 04:09:49.977758 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 13 04:09:49.977758 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application created_value = creator( Oct 13 04:09:49.977758 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 13 04:09:49.979084 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 13 04:09:49.979084 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 13 04:09:49.979084 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application Oct 13 04:09:49.979084 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 393/808] 10.222.0.27 () {56 vars in 1108 bytes} [Thu Oct 13 04:09:49 2022] GET /identity/v3/domains/Default => generated 87 bytes in 181 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 13 04:09:49.983504 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-885bf037-94ee-4fff-9cec-eeeaa290d077 None admin] Authenticating user token {{(pid=81408) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:09:49.989820 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3aedca91-5c5e-4e32-9788-f7c043e28078 None admin] Validating token access rules against request {{(pid=81409) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:09:50.027264 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-91dac882-c9fe-4c76-9dea-b7ddb296b77e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81409) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:09:50.027882 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91dac882-c9fe-4c76-9dea-b7ddb296b77e None admin] REQUEST_METHOD: `GET` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:09:50.028156 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91dac882-c9fe-4c76-9dea-b7ddb296b77e None admin] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:09:50.028425 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91dac882-c9fe-4c76-9dea-b7ddb296b77e None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:09:50.038506 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-91dac882-c9fe-4c76-9dea-b7ddb296b77e None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 13 04:09:50.043448 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application [None req-91dac882-c9fe-4c76-9dea-b7ddb296b77e None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Oct 13 04:09:50.043448 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 13 04:09:50.043448 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 13 04:09:50.043448 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 13 04:09:50.043448 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 13 04:09:50.043448 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 13 04:09:50.043448 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 13 04:09:50.043448 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 13 04:09:50.043448 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 13 04:09:50.043448 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 13 04:09:50.043448 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 13 04:09:50.043448 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 13 04:09:50.043448 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 13 04:09:50.043448 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 13 04:09:50.043448 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 13 04:09:50.043448 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 13 04:09:50.044613 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 13 04:09:50.044613 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 13 04:09:50.044613 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 13 04:09:50.044613 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 13 04:09:50.044613 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 13 04:09:50.044613 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 13 04:09:50.044613 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 13 04:09:50.044613 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 13 04:09:50.044613 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 13 04:09:50.044613 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application return self.get_or_create( Oct 13 04:09:50.044613 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 13 04:09:50.044613 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application with Lock( Oct 13 04:09:50.044613 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 13 04:09:50.044613 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application return self._enter() Oct 13 04:09:50.044613 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 13 04:09:50.044613 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 13 04:09:50.045641 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 13 04:09:50.045641 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application return self.creator() Oct 13 04:09:50.045641 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 13 04:09:50.045641 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application created_value = creator( Oct 13 04:09:50.045641 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 13 04:09:50.045641 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 13 04:09:50.045641 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 13 04:09:50.045641 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 13 04:09:50.045641 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 13 04:09:50.045641 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 13 04:09:50.045641 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Oct 13 04:09:50.045641 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application Oct 13 04:09:50.045641 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 416/809] 10.222.0.27 () {56 vars in 1096 bytes} [Thu Oct 13 04:09:49 2022] GET /identity/v3/roles/admin => generated 83 bytes in 171 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Oct 13 04:09:50.050949 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-91dac882-c9fe-4c76-9dea-b7ddb296b77e None admin] Authenticating user token {{(pid=81409) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:09:50.064882 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG dbcounter [-] [81408] Writing DB stats keystone:SELECT=100 {{(pid=81408) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:09:50.088798 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-885bf037-94ee-4fff-9cec-eeeaa290d077 None admin] Validating token access rules against request {{(pid=81408) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:09:50.108108 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG dbcounter [-] [81409] Writing DB stats keystone:SELECT=100 {{(pid=81409) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:09:50.122495 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c3c32718-b210-45ee-ab01-b6fa096c1627 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81408) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:09:50.123117 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c3c32718-b210-45ee-ab01-b6fa096c1627 None admin] REQUEST_METHOD: `GET` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:09:50.123392 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c3c32718-b210-45ee-ab01-b6fa096c1627 None admin] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:09:50.123658 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c3c32718-b210-45ee-ab01-b6fa096c1627 None admin] PATH_INFO: `/v3/domains` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:09:50.124291 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c3c32718-b210-45ee-ab01-b6fa096c1627 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81408) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 13 04:09:50.124587 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c3c32718-b210-45ee-ab01-b6fa096c1627 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:09:50.125956 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c3c32718-b210-45ee-ab01-b6fa096c1627 None admin] RBAC: Authorization granted {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:09:50.139827 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 394/810] 10.222.0.27 () {56 vars in 1109 bytes} [Thu Oct 13 04:09:49 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 157 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 13 04:09:50.145311 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c3c32718-b210-45ee-ab01-b6fa096c1627 None admin] Authenticating user token {{(pid=81408) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:09:50.151762 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-91dac882-c9fe-4c76-9dea-b7ddb296b77e None admin] Validating token access rules against request {{(pid=81409) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:09:50.184457 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-578ca0b6-9aa5-4f5b-9a84-253716edb282 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81409) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:09:50.185053 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-578ca0b6-9aa5-4f5b-9a84-253716edb282 None admin] REQUEST_METHOD: `GET` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:09:50.185328 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-578ca0b6-9aa5-4f5b-9a84-253716edb282 None admin] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:09:50.185598 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-578ca0b6-9aa5-4f5b-9a84-253716edb282 None admin] PATH_INFO: `/v3/roles` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:09:50.186281 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-578ca0b6-9aa5-4f5b-9a84-253716edb282 None admin] RBAC: Adding query filter params (name=admin) {{(pid=81409) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 13 04:09:50.186581 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-578ca0b6-9aa5-4f5b-9a84-253716edb282 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:09:50.187934 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-578ca0b6-9aa5-4f5b-9a84-253716edb282 None admin] RBAC: Authorization granted {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:09:50.196384 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 417/811] 10.222.0.27 () {56 vars in 1099 bytes} [Thu Oct 13 04:09:50 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 146 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 13 04:09:50.202050 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-578ca0b6-9aa5-4f5b-9a84-253716edb282 None admin] Authenticating user token {{(pid=81409) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:09:50.245097 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c3c32718-b210-45ee-ab01-b6fa096c1627 None admin] Validating token access rules against request {{(pid=81408) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:09:50.278917 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8eae3c73-1802-4fc6-8be2-2848efcc0a4f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81408) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:09:50.279540 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8eae3c73-1802-4fc6-8be2-2848efcc0a4f None admin] REQUEST_METHOD: `GET` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:09:50.279822 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8eae3c73-1802-4fc6-8be2-2848efcc0a4f None admin] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:09:50.280094 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8eae3c73-1802-4fc6-8be2-2848efcc0a4f None admin] PATH_INFO: `/v3/projects/service` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:09:50.286548 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8eae3c73-1802-4fc6-8be2-2848efcc0a4f None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:09:50.287999 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8eae3c73-1802-4fc6-8be2-2848efcc0a4f None admin] RBAC: Authorization granted {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:09:50.290375 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG dbcounter [-] [81408] Writing DB stats keystone:SELECT=100 {{(pid=81408) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:09:50.301004 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-578ca0b6-9aa5-4f5b-9a84-253716edb282 None admin] Validating token access rules against request {{(pid=81409) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:09:50.301635 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application [None req-8eae3c73-1802-4fc6-8be2-2848efcc0a4f None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Oct 13 04:09:50.301635 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 13 04:09:50.301635 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 13 04:09:50.301635 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 13 04:09:50.301635 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 13 04:09:50.301635 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 13 04:09:50.301635 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 13 04:09:50.301635 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 13 04:09:50.301635 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 13 04:09:50.301635 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 13 04:09:50.301635 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 13 04:09:50.301635 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 13 04:09:50.301635 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Oct 13 04:09:50.301635 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application return self._get_project(project_id) Oct 13 04:09:50.301635 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Oct 13 04:09:50.302626 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Oct 13 04:09:50.302626 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 13 04:09:50.302626 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 13 04:09:50.302626 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 13 04:09:50.302626 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 13 04:09:50.302626 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 13 04:09:50.302626 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application return self.get_or_create( Oct 13 04:09:50.302626 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 13 04:09:50.302626 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application with Lock( Oct 13 04:09:50.302626 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 13 04:09:50.302626 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application return self._enter() Oct 13 04:09:50.302626 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 13 04:09:50.302626 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 13 04:09:50.302626 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 13 04:09:50.302626 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application return self.creator() Oct 13 04:09:50.302626 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 13 04:09:50.302626 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application created_value = creator( Oct 13 04:09:50.303525 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Oct 13 04:09:50.303525 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Oct 13 04:09:50.303525 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Oct 13 04:09:50.303525 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Oct 13 04:09:50.303525 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Oct 13 04:09:50.303525 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Oct 13 04:09:50.303525 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Oct 13 04:09:50.303525 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application Oct 13 04:09:50.303525 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 395/812] 10.222.0.27 () {56 vars in 1111 bytes} [Thu Oct 13 04:09:50 2022] GET /identity/v3/projects/service => generated 88 bytes in 158 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Oct 13 04:09:50.309166 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8eae3c73-1802-4fc6-8be2-2848efcc0a4f None admin] Authenticating user token {{(pid=81408) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:09:50.321597 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG dbcounter [-] [81409] Writing DB stats keystone:SELECT=100 {{(pid=81409) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:09:50.335296 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-80ed50db-6fae-4131-9155-57af593fe5e0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81409) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:09:50.335960 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80ed50db-6fae-4131-9155-57af593fe5e0 None admin] REQUEST_METHOD: `GET` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:09:50.336238 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80ed50db-6fae-4131-9155-57af593fe5e0 None admin] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:09:50.336506 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80ed50db-6fae-4131-9155-57af593fe5e0 None admin] PATH_INFO: `/v3/users/778ef39a805140eab48c78a122f83abc` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:09:50.350267 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-80ed50db-6fae-4131-9155-57af593fe5e0 None admin] RBAC: Authorizing `identity:get_user(user_id=778ef39a805140eab48c78a122f83abc)` {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:09:50.352530 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-80ed50db-6fae-4131-9155-57af593fe5e0 None admin] RBAC: Authorization granted {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:09:50.354175 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 418/813] 10.222.0.27 () {56 vars in 1177 bytes} [Thu Oct 13 04:09:50 2022] GET /identity/v3/users/778ef39a805140eab48c78a122f83abc => generated 285 bytes in 152 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 13 04:09:50.359883 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-80ed50db-6fae-4131-9155-57af593fe5e0 None admin] Authenticating user token {{(pid=81409) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:09:50.407980 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8eae3c73-1802-4fc6-8be2-2848efcc0a4f None admin] Validating token access rules against request {{(pid=81408) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:09:50.442210 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-91754ca1-88a1-4824-a012-3857e781ca29 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81408) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:09:50.442833 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91754ca1-88a1-4824-a012-3857e781ca29 None admin] REQUEST_METHOD: `GET` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:09:50.443107 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91754ca1-88a1-4824-a012-3857e781ca29 None admin] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:09:50.443374 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91754ca1-88a1-4824-a012-3857e781ca29 None admin] PATH_INFO: `/v3/projects` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:09:50.444056 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-91754ca1-88a1-4824-a012-3857e781ca29 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81408) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 13 04:09:50.444358 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-91754ca1-88a1-4824-a012-3857e781ca29 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:09:50.445846 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-91754ca1-88a1-4824-a012-3857e781ca29 None admin] RBAC: Authorization granted {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:09:50.457792 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 396/814] 10.222.0.27 () {56 vars in 1148 bytes} [Thu Oct 13 04:09:50 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 149 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 13 04:09:50.462001 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-80ed50db-6fae-4131-9155-57af593fe5e0 None admin] Validating token access rules against request {{(pid=81409) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:09:50.464443 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-91754ca1-88a1-4824-a012-3857e781ca29 None admin] Authenticating user token {{(pid=81408) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:09:50.494855 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-58a87dcf-5a26-48ca-994f-b310613dee9a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81409) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:09:50.495536 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58a87dcf-5a26-48ca-994f-b310613dee9a None admin] REQUEST_METHOD: `GET` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:09:50.495747 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58a87dcf-5a26-48ca-994f-b310613dee9a None admin] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:09:50.496011 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58a87dcf-5a26-48ca-994f-b310613dee9a None admin] PATH_INFO: `/v3/projects/78b3aa8e26634401b668ae1402864d8b` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:09:50.497115 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-58a87dcf-5a26-48ca-994f-b310613dee9a None admin] RBAC: Authorizing `identity:get_project(project_id=78b3aa8e26634401b668ae1402864d8b)` {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:09:50.498549 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-58a87dcf-5a26-48ca-994f-b310613dee9a None admin] RBAC: Authorization granted {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:09:50.500308 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 419/815] 10.222.0.27 () {56 vars in 1186 bytes} [Thu Oct 13 04:09:50 2022] GET /identity/v3/projects/78b3aa8e26634401b668ae1402864d8b => generated 311 bytes in 141 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 13 04:09:50.505921 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-58a87dcf-5a26-48ca-994f-b310613dee9a None admin] Authenticating user token {{(pid=81409) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:09:50.522166 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG dbcounter [-] [81408] Writing DB stats keystone:SELECT=100 {{(pid=81408) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:09:50.552625 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG dbcounter [-] [81409] Writing DB stats keystone:SELECT=100 {{(pid=81409) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:09:50.564971 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-91754ca1-88a1-4824-a012-3857e781ca29 None admin] Validating token access rules against request {{(pid=81408) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:09:50.598509 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b4c07c1c-6a53-4743-98f1-f0f347a841d4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81408) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:09:50.599159 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4c07c1c-6a53-4743-98f1-f0f347a841d4 None admin] REQUEST_METHOD: `PUT` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:09:50.599436 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4c07c1c-6a53-4743-98f1-f0f347a841d4 None admin] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:09:50.599706 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4c07c1c-6a53-4743-98f1-f0f347a841d4 None admin] PATH_INFO: `/v3/projects/dcec0431fb6849499d6aec35a234daa6/users/cb740ed7f7684b079e45f887be83f795/roles/0b094166c73d4276a8e8eec82da7f5ff` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:09:50.602482 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b4c07c1c-6a53-4743-98f1-f0f347a841d4 None admin] RBAC: Authorizing `identity:create_grant(project_id=dcec0431fb6849499d6aec35a234daa6, user_id=cb740ed7f7684b079e45f887be83f795, role_id=0b094166c73d4276a8e8eec82da7f5ff)` {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:09:50.603917 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b4c07c1c-6a53-4743-98f1-f0f347a841d4 None admin] RBAC: Authorization granted {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:09:50.605893 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-58a87dcf-5a26-48ca-994f-b310613dee9a None admin] Validating token access rules against request {{(pid=81409) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:09:50.616376 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 397/816] 10.222.0.27 () {58 vars in 1439 bytes} [Thu Oct 13 04:09:50 2022] PUT /identity/v3/projects/dcec0431fb6849499d6aec35a234daa6/users/cb740ed7f7684b079e45f887be83f795/roles/0b094166c73d4276a8e8eec82da7f5ff => generated 0 bytes in 152 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 13 04:09:50.645316 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5685b646-2e7b-4247-b299-44487bcc39c6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81409) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:09:50.646000 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5685b646-2e7b-4247-b299-44487bcc39c6 None admin] REQUEST_METHOD: `PUT` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:09:50.646313 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5685b646-2e7b-4247-b299-44487bcc39c6 None admin] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:09:50.646586 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5685b646-2e7b-4247-b299-44487bcc39c6 None admin] PATH_INFO: `/v3/projects/78b3aa8e26634401b668ae1402864d8b/users/778ef39a805140eab48c78a122f83abc/roles/ef55fbd94e1040be83695754075a7ed5` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:09:50.649469 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5685b646-2e7b-4247-b299-44487bcc39c6 None admin] RBAC: Authorizing `identity:create_grant(project_id=78b3aa8e26634401b668ae1402864d8b, user_id=778ef39a805140eab48c78a122f83abc, role_id=ef55fbd94e1040be83695754075a7ed5)` {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:09:50.651178 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5685b646-2e7b-4247-b299-44487bcc39c6 None admin] RBAC: Authorization granted {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:09:50.665523 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 420/817] 10.222.0.27 () {58 vars in 1439 bytes} [Thu Oct 13 04:09:50 2022] PUT /identity/v3/projects/78b3aa8e26634401b668ae1402864d8b/users/778ef39a805140eab48c78a122f83abc/roles/ef55fbd94e1040be83695754075a7ed5 => generated 0 bytes in 160 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 13 04:09:52.755885 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69d4f04f-1116-4a71-b4ee-52ea8e25e6bb None None] REQUEST_METHOD: `GET` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:09:52.756411 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69d4f04f-1116-4a71-b4ee-52ea8e25e6bb None None] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:09:52.756411 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69d4f04f-1116-4a71-b4ee-52ea8e25e6bb None None] PATH_INFO: `/` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:09:52.757852 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 398/818] 10.222.0.27 () {54 vars in 927 bytes} [Thu Oct 13 04:09:52 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Oct 13 04:09:52.763950 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1d4d87d-b757-46d9-b636-10873a9a2698 None None] REQUEST_METHOD: `POST` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:09:52.764408 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1d4d87d-b757-46d9-b636-10873a9a2698 None None] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:09:52.764699 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1d4d87d-b757-46d9-b636-10873a9a2698 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:09:52.782514 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.auth.core [None req-a1d4d87d-b757-46d9-b636-10873a9a2698 None None] MFA Rules not processed for user `0e2d979563d6421995451a0e86bf3022`. Rule list: `[]` (Enabled: `True`). {{(pid=81409) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 13 04:09:52.783699 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.fernet_utils [None req-a1d4d87d-b757-46d9-b636-10873a9a2698 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81409) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:09:52.794552 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9907f3bc-7c37-42a1-a83e-0fe359f0799a None None] REQUEST_METHOD: `GET` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:09:52.794924 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9907f3bc-7c37-42a1-a83e-0fe359f0799a None None] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:09:52.795259 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9907f3bc-7c37-42a1-a83e-0fe359f0799a None None] PATH_INFO: `/` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:09:52.796646 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 399/819] 10.222.0.27 () {54 vars in 927 bytes} [Thu Oct 13 04:09:52 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Oct 13 04:09:52.801570 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-73766e47-89ba-4459-800b-025dd0ccb84a None None] REQUEST_METHOD: `POST` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:09:52.801847 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-73766e47-89ba-4459-800b-025dd0ccb84a None None] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:09:52.802114 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-73766e47-89ba-4459-800b-025dd0ccb84a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:09:52.821000 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.auth.core [None req-73766e47-89ba-4459-800b-025dd0ccb84a None None] MFA Rules not processed for user `0e2d979563d6421995451a0e86bf3022`. Rule list: `[]` (Enabled: `True`). {{(pid=81408) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 13 04:09:52.822551 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.fernet_utils [None req-73766e47-89ba-4459-800b-025dd0ccb84a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81408) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:09:52.849516 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 421/820] 10.222.0.27 () {58 vars in 1024 bytes} [Thu Oct 13 04:09:52 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 86 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Oct 13 04:09:52.887942 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 400/821] 10.222.0.27 () {58 vars in 1024 bytes} [Thu Oct 13 04:09:52 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 87 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Oct 13 04:09:53.122118 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a1d4d87d-b757-46d9-b636-10873a9a2698 None None] Authenticating user token {{(pid=81409) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:09:53.155392 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG dbcounter [-] [81409] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81409) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:09:53.167623 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-73766e47-89ba-4459-800b-025dd0ccb84a None None] Authenticating user token {{(pid=81408) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:09:53.215645 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG dbcounter [-] [81408] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81408) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:09:53.222318 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a1d4d87d-b757-46d9-b636-10873a9a2698 None None] Validating token access rules against request {{(pid=81409) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:09:53.258338 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-36eaaf13-c369-4d8c-97b1-b57310519b1a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81409) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:09:53.258990 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36eaaf13-c369-4d8c-97b1-b57310519b1a None admin] REQUEST_METHOD: `GET` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:09:53.259351 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36eaaf13-c369-4d8c-97b1-b57310519b1a None admin] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:09:53.259622 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36eaaf13-c369-4d8c-97b1-b57310519b1a None admin] PATH_INFO: `/v3/roles/ResellerAdmin` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:09:53.269398 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-36eaaf13-c369-4d8c-97b1-b57310519b1a None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 13 04:09:53.270004 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-73766e47-89ba-4459-800b-025dd0ccb84a None None] Validating token access rules against request {{(pid=81408) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:09:53.274932 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application [None req-36eaaf13-c369-4d8c-97b1-b57310519b1a None admin] Could not find role: ResellerAdmin.: keystone.exception.RoleNotFound: Could not find role: ResellerAdmin. Oct 13 04:09:53.274932 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 13 04:09:53.274932 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 13 04:09:53.274932 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 13 04:09:53.274932 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 13 04:09:53.274932 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 13 04:09:53.274932 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 13 04:09:53.274932 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 13 04:09:53.274932 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 13 04:09:53.274932 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 13 04:09:53.274932 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 13 04:09:53.274932 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 13 04:09:53.274932 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 13 04:09:53.274932 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 13 04:09:53.274932 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 13 04:09:53.274932 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 13 04:09:53.275871 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 13 04:09:53.275871 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 13 04:09:53.275871 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 13 04:09:53.275871 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 13 04:09:53.275871 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 13 04:09:53.275871 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 13 04:09:53.275871 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 13 04:09:53.275871 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 13 04:09:53.275871 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 13 04:09:53.275871 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application return self.get_or_create( Oct 13 04:09:53.275871 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 13 04:09:53.275871 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application with Lock( Oct 13 04:09:53.275871 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 13 04:09:53.275871 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application return self._enter() Oct 13 04:09:53.275871 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 13 04:09:53.275871 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 13 04:09:53.276716 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 13 04:09:53.276716 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application return self.creator() Oct 13 04:09:53.276716 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 13 04:09:53.276716 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application created_value = creator( Oct 13 04:09:53.276716 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 13 04:09:53.276716 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 13 04:09:53.276716 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 13 04:09:53.276716 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 13 04:09:53.276716 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 13 04:09:53.276716 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 13 04:09:53.276716 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: ResellerAdmin. Oct 13 04:09:53.276716 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application Oct 13 04:09:53.276716 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 422/822] 10.222.0.27 () {56 vars in 1120 bytes} [Thu Oct 13 04:09:53 2022] GET /identity/v3/roles/ResellerAdmin => generated 91 bytes in 155 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 13 04:09:53.281956 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-36eaaf13-c369-4d8c-97b1-b57310519b1a None admin] Authenticating user token {{(pid=81409) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:09:53.305056 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b23bae57-caaf-4dc9-b6e6-55ec0e338e6a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81408) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:09:53.305687 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b23bae57-caaf-4dc9-b6e6-55ec0e338e6a None admin] REQUEST_METHOD: `GET` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:09:53.305966 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b23bae57-caaf-4dc9-b6e6-55ec0e338e6a None admin] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:09:53.306257 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b23bae57-caaf-4dc9-b6e6-55ec0e338e6a None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:09:53.316174 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b23bae57-caaf-4dc9-b6e6-55ec0e338e6a None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 13 04:09:53.322330 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application [None req-b23bae57-caaf-4dc9-b6e6-55ec0e338e6a None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Oct 13 04:09:53.322330 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 13 04:09:53.322330 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 13 04:09:53.322330 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 13 04:09:53.322330 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 13 04:09:53.322330 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 13 04:09:53.322330 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 13 04:09:53.322330 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 13 04:09:53.322330 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 13 04:09:53.322330 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 13 04:09:53.322330 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 13 04:09:53.322330 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 13 04:09:53.322330 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 13 04:09:53.322330 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 13 04:09:53.322330 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 13 04:09:53.322330 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 13 04:09:53.323375 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 13 04:09:53.323375 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 13 04:09:53.323375 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 13 04:09:53.323375 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 13 04:09:53.323375 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 13 04:09:53.323375 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 13 04:09:53.323375 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 13 04:09:53.323375 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 13 04:09:53.323375 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 13 04:09:53.323375 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application return self.get_or_create( Oct 13 04:09:53.323375 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 13 04:09:53.323375 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application with Lock( Oct 13 04:09:53.323375 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 13 04:09:53.323375 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application return self._enter() Oct 13 04:09:53.323375 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 13 04:09:53.323375 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 13 04:09:53.324219 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 13 04:09:53.324219 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application return self.creator() Oct 13 04:09:53.324219 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 13 04:09:53.324219 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application created_value = creator( Oct 13 04:09:53.324219 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 13 04:09:53.324219 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 13 04:09:53.324219 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 13 04:09:53.324219 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 13 04:09:53.324219 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 13 04:09:53.324219 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 13 04:09:53.324219 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Oct 13 04:09:53.324219 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application Oct 13 04:09:53.324219 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 401/823] 10.222.0.27 () {56 vars in 1096 bytes} [Thu Oct 13 04:09:53 2022] GET /identity/v3/roles/admin => generated 83 bytes in 157 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Oct 13 04:09:53.329406 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b23bae57-caaf-4dc9-b6e6-55ec0e338e6a None admin] Authenticating user token {{(pid=81408) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:09:53.378456 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG dbcounter [-] [81409] Writing DB stats keystone:SELECT=100 {{(pid=81409) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:09:53.384765 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-36eaaf13-c369-4d8c-97b1-b57310519b1a None admin] Validating token access rules against request {{(pid=81409) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:09:53.419286 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-992f0150-bb42-44e3-b530-a6d30db46851 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81409) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:09:53.419889 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-992f0150-bb42-44e3-b530-a6d30db46851 None admin] REQUEST_METHOD: `GET` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:09:53.420156 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-992f0150-bb42-44e3-b530-a6d30db46851 None admin] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:09:53.420418 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-992f0150-bb42-44e3-b530-a6d30db46851 None admin] PATH_INFO: `/v3/roles` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:09:53.421067 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-992f0150-bb42-44e3-b530-a6d30db46851 None admin] RBAC: Adding query filter params (name=ResellerAdmin) {{(pid=81409) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 13 04:09:53.421363 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-992f0150-bb42-44e3-b530-a6d30db46851 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:09:53.422766 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-992f0150-bb42-44e3-b530-a6d30db46851 None admin] RBAC: Authorization granted {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:09:53.431254 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 423/824] 10.222.0.27 () {56 vars in 1115 bytes} [Thu Oct 13 04:09:53 2022] GET /identity/v3/roles?name=ResellerAdmin => generated 338 bytes in 150 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 13 04:09:53.436955 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-992f0150-bb42-44e3-b530-a6d30db46851 None admin] Authenticating user token {{(pid=81409) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:09:53.437230 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b23bae57-caaf-4dc9-b6e6-55ec0e338e6a None admin] Validating token access rules against request {{(pid=81408) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:09:53.447635 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG dbcounter [-] [81408] Writing DB stats keystone:SELECT=100 {{(pid=81408) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:09:53.473009 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3b224802-d7f0-46f2-b690-ff4c21677574 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81408) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:09:53.473603 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b224802-d7f0-46f2-b690-ff4c21677574 None admin] REQUEST_METHOD: `GET` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:09:53.473874 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b224802-d7f0-46f2-b690-ff4c21677574 None admin] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:09:53.474138 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b224802-d7f0-46f2-b690-ff4c21677574 None admin] PATH_INFO: `/v3/roles` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:09:53.474828 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3b224802-d7f0-46f2-b690-ff4c21677574 None admin] RBAC: Adding query filter params (name=admin) {{(pid=81408) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 13 04:09:53.475121 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3b224802-d7f0-46f2-b690-ff4c21677574 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:09:53.476841 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3b224802-d7f0-46f2-b690-ff4c21677574 None admin] RBAC: Authorization granted {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:09:53.485331 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 402/825] 10.222.0.27 () {56 vars in 1099 bytes} [Thu Oct 13 04:09:53 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 156 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 13 04:09:53.491173 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3b224802-d7f0-46f2-b690-ff4c21677574 None admin] Authenticating user token {{(pid=81408) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:09:53.535344 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-992f0150-bb42-44e3-b530-a6d30db46851 None admin] Validating token access rules against request {{(pid=81409) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:09:53.567886 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5225cfa2-4b35-4635-b3d9-10e335a1de1c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81409) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:09:53.568507 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5225cfa2-4b35-4635-b3d9-10e335a1de1c None admin] REQUEST_METHOD: `GET` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:09:53.568777 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5225cfa2-4b35-4635-b3d9-10e335a1de1c None admin] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:09:53.569046 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5225cfa2-4b35-4635-b3d9-10e335a1de1c None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:09:53.579893 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5225cfa2-4b35-4635-b3d9-10e335a1de1c None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:09:53.581317 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5225cfa2-4b35-4635-b3d9-10e335a1de1c None admin] RBAC: Authorization granted {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:09:53.591011 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3b224802-d7f0-46f2-b690-ff4c21677574 None admin] Validating token access rules against request {{(pid=81408) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:09:53.596141 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application [None req-5225cfa2-4b35-4635-b3d9-10e335a1de1c None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 13 04:09:53.596141 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 13 04:09:53.596141 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 13 04:09:53.596141 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 13 04:09:53.596141 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 13 04:09:53.596141 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 13 04:09:53.596141 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 13 04:09:53.596141 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 13 04:09:53.596141 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 13 04:09:53.596141 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 13 04:09:53.596141 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 13 04:09:53.596141 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 13 04:09:53.596141 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 13 04:09:53.596141 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 13 04:09:53.596141 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 13 04:09:53.596141 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 13 04:09:53.597198 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 13 04:09:53.597198 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 13 04:09:53.597198 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 13 04:09:53.597198 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 13 04:09:53.597198 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 13 04:09:53.597198 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application return self.get_or_create( Oct 13 04:09:53.597198 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 13 04:09:53.597198 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application with Lock( Oct 13 04:09:53.597198 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 13 04:09:53.597198 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application return self._enter() Oct 13 04:09:53.597198 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 13 04:09:53.597198 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 13 04:09:53.597198 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 13 04:09:53.597198 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application return self.creator() Oct 13 04:09:53.597198 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 13 04:09:53.597198 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application created_value = creator( Oct 13 04:09:53.597198 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 13 04:09:53.598094 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 13 04:09:53.598094 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 13 04:09:53.598094 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application Oct 13 04:09:53.598094 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 424/826] 10.222.0.27 () {56 vars in 1108 bytes} [Thu Oct 13 04:09:53 2022] GET /identity/v3/domains/Default => generated 87 bytes in 161 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Oct 13 04:09:53.604104 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5225cfa2-4b35-4635-b3d9-10e335a1de1c None admin] Authenticating user token {{(pid=81409) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:09:53.616019 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG dbcounter [-] [81409] Writing DB stats keystone:SELECT=100 {{(pid=81409) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:09:53.626929 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cee2108e-fbc3-462f-97d7-111e2fb782ca None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81408) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:09:53.627560 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cee2108e-fbc3-462f-97d7-111e2fb782ca None admin] REQUEST_METHOD: `GET` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:09:53.627834 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cee2108e-fbc3-462f-97d7-111e2fb782ca None admin] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:09:53.628127 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cee2108e-fbc3-462f-97d7-111e2fb782ca None admin] PATH_INFO: `/v3/users/778ef39a805140eab48c78a122f83abc` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:09:53.629299 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cee2108e-fbc3-462f-97d7-111e2fb782ca None admin] RBAC: Authorizing `identity:get_user(user_id=778ef39a805140eab48c78a122f83abc)` {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:09:53.630709 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cee2108e-fbc3-462f-97d7-111e2fb782ca None admin] RBAC: Authorization granted {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:09:53.632281 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 403/827] 10.222.0.27 () {56 vars in 1177 bytes} [Thu Oct 13 04:09:53 2022] GET /identity/v3/users/778ef39a805140eab48c78a122f83abc => generated 285 bytes in 141 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 13 04:09:53.637925 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cee2108e-fbc3-462f-97d7-111e2fb782ca None admin] Authenticating user token {{(pid=81408) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:09:53.676826 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG dbcounter [-] [81408] Writing DB stats keystone:SELECT=100 {{(pid=81408) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:09:53.702439 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5225cfa2-4b35-4635-b3d9-10e335a1de1c None admin] Validating token access rules against request {{(pid=81409) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:09:53.735950 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b0d1f3af-2433-428b-86b9-d68fdf44076c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81409) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:09:53.736544 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0d1f3af-2433-428b-86b9-d68fdf44076c None admin] REQUEST_METHOD: `GET` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:09:53.736816 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0d1f3af-2433-428b-86b9-d68fdf44076c None admin] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:09:53.736911 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cee2108e-fbc3-462f-97d7-111e2fb782ca None admin] Validating token access rules against request {{(pid=81408) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:09:53.737083 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0d1f3af-2433-428b-86b9-d68fdf44076c None admin] PATH_INFO: `/v3/domains` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:09:53.737709 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b0d1f3af-2433-428b-86b9-d68fdf44076c None admin] RBAC: Adding query filter params (name=Default) {{(pid=81409) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 13 04:09:53.738006 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b0d1f3af-2433-428b-86b9-d68fdf44076c None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:09:53.739392 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b0d1f3af-2433-428b-86b9-d68fdf44076c None admin] RBAC: Authorization granted {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:09:53.750825 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 425/828] 10.222.0.27 () {56 vars in 1109 bytes} [Thu Oct 13 04:09:53 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 147 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 13 04:09:53.756381 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b0d1f3af-2433-428b-86b9-d68fdf44076c None admin] Authenticating user token {{(pid=81409) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:09:53.769899 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-758d0acb-2a69-46e8-b82e-60d6ce8bb14e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81408) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:09:53.770541 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-758d0acb-2a69-46e8-b82e-60d6ce8bb14e None admin] REQUEST_METHOD: `GET` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:09:53.770821 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-758d0acb-2a69-46e8-b82e-60d6ce8bb14e None admin] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:09:53.771093 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-758d0acb-2a69-46e8-b82e-60d6ce8bb14e None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:09:53.823602 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG dbcounter [-] [81409] Writing DB stats keystone:SELECT=100 {{(pid=81409) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:09:53.838361 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-758d0acb-2a69-46e8-b82e-60d6ce8bb14e None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:09:53.839803 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-758d0acb-2a69-46e8-b82e-60d6ce8bb14e None admin] RBAC: Authorization granted {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:09:53.854924 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b0d1f3af-2433-428b-86b9-d68fdf44076c None admin] Validating token access rules against request {{(pid=81409) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:09:53.876886 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG dbcounter [-] [81408] Writing DB stats keystone:SELECT=100 {{(pid=81408) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:09:53.889791 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d1cd0beb-c240-45b6-9f5b-f5554a8ade26 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81409) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:09:53.890382 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1cd0beb-c240-45b6-9f5b-f5554a8ade26 None admin] REQUEST_METHOD: `GET` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:09:53.890654 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1cd0beb-c240-45b6-9f5b-f5554a8ade26 None admin] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:09:53.890918 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1cd0beb-c240-45b6-9f5b-f5554a8ade26 None admin] PATH_INFO: `/v3/users/nova` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:09:53.897677 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d1cd0beb-c240-45b6-9f5b-f5554a8ade26 None admin] RBAC: Authorizing `identity:get_user(user_id=nova)` {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:09:53.900004 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d1cd0beb-c240-45b6-9f5b-f5554a8ade26 None admin] RBAC: Authorization granted {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:09:53.911081 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application [None req-d1cd0beb-c240-45b6-9f5b-f5554a8ade26 None admin] Could not find user: nova.: keystone.exception.UserNotFound: Could not find user: nova. Oct 13 04:09:53.911081 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 13 04:09:53.911081 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 13 04:09:53.911081 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 13 04:09:53.911081 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 13 04:09:53.911081 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 13 04:09:53.911081 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 13 04:09:53.911081 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 13 04:09:53.911081 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 13 04:09:53.911081 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 13 04:09:53.911081 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 13 04:09:53.911081 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 13 04:09:53.911081 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Oct 13 04:09:53.911081 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application return self._get_user(user_id) Oct 13 04:09:53.911081 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Oct 13 04:09:53.911081 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Oct 13 04:09:53.912219 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 13 04:09:53.912219 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 13 04:09:53.912219 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Oct 13 04:09:53.912219 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 13 04:09:53.912219 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Oct 13 04:09:53.912219 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 13 04:09:53.912219 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 13 04:09:53.912219 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 13 04:09:53.912219 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 13 04:09:53.912219 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application return self.get_or_create( Oct 13 04:09:53.912219 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 13 04:09:53.912219 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application with Lock( Oct 13 04:09:53.912219 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 13 04:09:53.912219 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application return self._enter() Oct 13 04:09:53.912219 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 13 04:09:53.912219 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 13 04:09:53.912219 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 13 04:09:53.912219 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application return self.creator() Oct 13 04:09:53.913203 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 13 04:09:53.913203 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application created_value = creator( Oct 13 04:09:53.913203 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Oct 13 04:09:53.913203 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Oct 13 04:09:53.913203 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Oct 13 04:09:53.913203 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Oct 13 04:09:53.913203 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Oct 13 04:09:53.913203 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Oct 13 04:09:53.913203 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: nova. Oct 13 04:09:53.913203 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application Oct 13 04:09:53.913203 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 426/829] 10.222.0.27 () {56 vars in 1093 bytes} [Thu Oct 13 04:09:53 2022] GET /identity/v3/users/nova => generated 82 bytes in 156 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Oct 13 04:09:53.918247 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d1cd0beb-c240-45b6-9f5b-f5554a8ade26 None admin] Authenticating user token {{(pid=81409) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:09:53.939207 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 404/830] 10.222.0.27 () {58 vars in 1282 bytes} [Thu Oct 13 04:09:53 2022] GET /identity/v3/auth/tokens => generated 2118 bytes in 302 msecs (HTTP/1.1 200) 6 headers in 359 bytes (2 switches on core 0) Oct 13 04:09:53.944975 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-758d0acb-2a69-46e8-b82e-60d6ce8bb14e None admin] Authenticating user token {{(pid=81408) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:09:54.020442 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d1cd0beb-c240-45b6-9f5b-f5554a8ade26 None admin] Validating token access rules against request {{(pid=81409) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:09:54.041228 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-758d0acb-2a69-46e8-b82e-60d6ce8bb14e None admin] Validating token access rules against request {{(pid=81408) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:09:54.049009 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG dbcounter [-] [81409] Writing DB stats keystone:SELECT=100 {{(pid=81409) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:09:54.055405 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c44f1146-1266-4331-b56a-ed5abd3aca40 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81409) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:09:54.056009 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c44f1146-1266-4331-b56a-ed5abd3aca40 None admin] REQUEST_METHOD: `GET` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:09:54.056284 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c44f1146-1266-4331-b56a-ed5abd3aca40 None admin] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:09:54.056552 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c44f1146-1266-4331-b56a-ed5abd3aca40 None admin] PATH_INFO: `/v3/users` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:09:54.057265 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c44f1146-1266-4331-b56a-ed5abd3aca40 None admin] RBAC: Adding query filter params (domain_id=default, name=nova) {{(pid=81409) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 13 04:09:54.057566 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c44f1146-1266-4331-b56a-ed5abd3aca40 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:09:54.059024 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c44f1146-1266-4331-b56a-ed5abd3aca40 None admin] RBAC: Authorization granted {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:09:54.076137 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e86a5a8-f5c4-43b1-aa64-17c775b719da None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81408) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:09:54.076457 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 427/831] 10.222.0.27 () {56 vars in 1133 bytes} [Thu Oct 13 04:09:53 2022] GET /identity/v3/users?domain_id=default&name=nova => generated 368 bytes in 159 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 13 04:09:54.076763 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e86a5a8-f5c4-43b1-aa64-17c775b719da None admin] REQUEST_METHOD: `GET` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:09:54.077038 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e86a5a8-f5c4-43b1-aa64-17c775b719da None admin] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:09:54.077308 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e86a5a8-f5c4-43b1-aa64-17c775b719da None admin] PATH_INFO: `/v3/projects/78b3aa8e26634401b668ae1402864d8b` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:09:54.078466 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1e86a5a8-f5c4-43b1-aa64-17c775b719da None admin] RBAC: Authorizing `identity:get_project(project_id=78b3aa8e26634401b668ae1402864d8b)` {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:09:54.079896 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1e86a5a8-f5c4-43b1-aa64-17c775b719da None admin] RBAC: Authorization granted {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:09:54.086569 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 405/832] 10.222.0.27 () {56 vars in 1186 bytes} [Thu Oct 13 04:09:53 2022] GET /identity/v3/projects/78b3aa8e26634401b668ae1402864d8b => generated 311 bytes in 137 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 13 04:09:54.086718 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c44f1146-1266-4331-b56a-ed5abd3aca40 None admin] Authenticating user token {{(pid=81409) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:09:54.087501 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e86a5a8-f5c4-43b1-aa64-17c775b719da None admin] Authenticating user token {{(pid=81408) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:09:54.096821 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG dbcounter [-] [81408] Writing DB stats keystone:SELECT=100 {{(pid=81408) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:09:54.182441 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c44f1146-1266-4331-b56a-ed5abd3aca40 None admin] Validating token access rules against request {{(pid=81409) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:09:54.188917 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e86a5a8-f5c4-43b1-aa64-17c775b719da None admin] Validating token access rules against request {{(pid=81408) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:09:54.216651 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-60acaf36-e99b-423c-b88b-0c4fcec19cd3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81409) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:09:54.217243 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60acaf36-e99b-423c-b88b-0c4fcec19cd3 None admin] REQUEST_METHOD: `GET` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:09:54.217524 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60acaf36-e99b-423c-b88b-0c4fcec19cd3 None admin] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:09:54.217799 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60acaf36-e99b-423c-b88b-0c4fcec19cd3 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:09:54.222032 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d4f9bdcf-0b77-4a50-a6d0-248558d544cf None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81408) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:09:54.222651 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4f9bdcf-0b77-4a50-a6d0-248558d544cf None admin] REQUEST_METHOD: `GET` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:09:54.222928 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4f9bdcf-0b77-4a50-a6d0-248558d544cf None admin] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:09:54.223221 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4f9bdcf-0b77-4a50-a6d0-248558d544cf None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:09:54.223933 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d4f9bdcf-0b77-4a50-a6d0-248558d544cf None admin] RBAC: Adding query filter params (role.id=ef55fbd94e1040be83695754075a7ed5, scope.project.id=78b3aa8e26634401b668ae1402864d8b, user.id=778ef39a805140eab48c78a122f83abc) {{(pid=81408) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 13 04:09:54.224237 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d4f9bdcf-0b77-4a50-a6d0-248558d544cf None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:09:54.225671 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d4f9bdcf-0b77-4a50-a6d0-248558d544cf None admin] RBAC: Authorization granted {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:09:54.232512 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 406/833] 10.222.0.27 () {56 vars in 1374 bytes} [Thu Oct 13 04:09:54 2022] GET /identity/v3/role_assignments?user.id=778ef39a805140eab48c78a122f83abc&scope.project.id=78b3aa8e26634401b668ae1402864d8b&role.id=ef55fbd94e1040be83695754075a7ed5 => generated 608 bytes in 145 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 13 04:09:54.275844 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG dbcounter [-] [81409] Writing DB stats keystone:SELECT=100 {{(pid=81409) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:09:54.288810 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-60acaf36-e99b-423c-b88b-0c4fcec19cd3 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:09:54.291008 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-60acaf36-e99b-423c-b88b-0c4fcec19cd3 None admin] RBAC: Authorization granted {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:09:54.394874 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 428/834] 10.222.0.27 () {58 vars in 1282 bytes} [Thu Oct 13 04:09:54 2022] GET /identity/v3/auth/tokens => generated 2118 bytes in 313 msecs (HTTP/1.1 200) 6 headers in 359 bytes (2 switches on core 0) Oct 13 04:09:54.401323 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d4f9bdcf-0b77-4a50-a6d0-248558d544cf None admin] Authenticating user token {{(pid=81408) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:09:54.473125 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG dbcounter [-] [81408] Writing DB stats keystone:SELECT=100 {{(pid=81408) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:09:54.621500 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d4f9bdcf-0b77-4a50-a6d0-248558d544cf None admin] Validating token access rules against request {{(pid=81408) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:09:54.655398 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-80a1f02f-b910-476b-b92e-49fca34a83bb None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81408) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:09:54.656015 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80a1f02f-b910-476b-b92e-49fca34a83bb None admin] REQUEST_METHOD: `GET` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:09:54.656289 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80a1f02f-b910-476b-b92e-49fca34a83bb None admin] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:09:54.656558 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80a1f02f-b910-476b-b92e-49fca34a83bb None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:09:54.666677 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-80a1f02f-b910-476b-b92e-49fca34a83bb None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:09:54.668125 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-80a1f02f-b910-476b-b92e-49fca34a83bb None admin] RBAC: Authorization granted {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:09:54.687116 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application [None req-80a1f02f-b910-476b-b92e-49fca34a83bb None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 13 04:09:54.687116 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 13 04:09:54.687116 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 13 04:09:54.687116 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 13 04:09:54.687116 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 13 04:09:54.687116 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 13 04:09:54.687116 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 13 04:09:54.687116 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 13 04:09:54.687116 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 13 04:09:54.687116 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 13 04:09:54.687116 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 13 04:09:54.687116 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 13 04:09:54.687116 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 13 04:09:54.687116 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 13 04:09:54.687116 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 13 04:09:54.687116 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 13 04:09:54.688277 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 13 04:09:54.688277 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 13 04:09:54.688277 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 13 04:09:54.688277 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 13 04:09:54.688277 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 13 04:09:54.688277 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application return self.get_or_create( Oct 13 04:09:54.688277 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 13 04:09:54.688277 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application with Lock( Oct 13 04:09:54.688277 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 13 04:09:54.688277 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application return self._enter() Oct 13 04:09:54.688277 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 13 04:09:54.688277 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 13 04:09:54.688277 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 13 04:09:54.688277 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application return self.creator() Oct 13 04:09:54.688277 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 13 04:09:54.688277 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application created_value = creator( Oct 13 04:09:54.688277 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 13 04:09:54.689371 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 13 04:09:54.689371 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 13 04:09:54.689371 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application Oct 13 04:09:54.690364 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 407/835] 10.222.0.27 () {56 vars in 1108 bytes} [Thu Oct 13 04:09:54 2022] GET /identity/v3/domains/Default => generated 87 bytes in 290 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 13 04:09:54.693867 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-60acaf36-e99b-423c-b88b-0c4fcec19cd3 None admin] Authenticating user token {{(pid=81409) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:09:54.782136 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG dbcounter [-] [81409] Writing DB stats keystone:SELECT=100 {{(pid=81409) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:09:54.795699 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-60acaf36-e99b-423c-b88b-0c4fcec19cd3 None admin] Validating token access rules against request {{(pid=81409) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:09:54.828865 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ae09475d-a713-446c-b2f3-acdcee4532e0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81409) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:09:54.829475 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae09475d-a713-446c-b2f3-acdcee4532e0 None admin] REQUEST_METHOD: `GET` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:09:54.829721 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae09475d-a713-446c-b2f3-acdcee4532e0 None admin] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:09:54.829988 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae09475d-a713-446c-b2f3-acdcee4532e0 None admin] PATH_INFO: `/v3/domains` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:09:54.830656 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ae09475d-a713-446c-b2f3-acdcee4532e0 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81409) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 13 04:09:54.830956 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ae09475d-a713-446c-b2f3-acdcee4532e0 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:09:54.832306 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ae09475d-a713-446c-b2f3-acdcee4532e0 None admin] RBAC: Authorization granted {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:09:54.843818 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 429/836] 10.222.0.27 () {56 vars in 1109 bytes} [Thu Oct 13 04:09:54 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 150 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 13 04:09:54.849390 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-80a1f02f-b910-476b-b92e-49fca34a83bb None admin] Authenticating user token {{(pid=81408) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:09:54.946935 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-80a1f02f-b910-476b-b92e-49fca34a83bb None admin] Validating token access rules against request {{(pid=81408) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:09:54.969612 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG dbcounter [-] [81408] Writing DB stats keystone:SELECT=100 {{(pid=81408) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:09:54.982160 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be3f45b9-2323-4b80-a9dc-5d1eab506595 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81408) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:09:54.982810 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be3f45b9-2323-4b80-a9dc-5d1eab506595 None admin] REQUEST_METHOD: `GET` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:09:54.983085 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be3f45b9-2323-4b80-a9dc-5d1eab506595 None admin] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:09:54.983353 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be3f45b9-2323-4b80-a9dc-5d1eab506595 None admin] PATH_INFO: `/v3/projects/service` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:09:54.989973 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-be3f45b9-2323-4b80-a9dc-5d1eab506595 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:09:54.991454 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-be3f45b9-2323-4b80-a9dc-5d1eab506595 None admin] RBAC: Authorization granted {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:09:55.002788 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application [None req-be3f45b9-2323-4b80-a9dc-5d1eab506595 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Oct 13 04:09:55.002788 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 13 04:09:55.002788 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 13 04:09:55.002788 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 13 04:09:55.002788 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 13 04:09:55.002788 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 13 04:09:55.002788 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 13 04:09:55.002788 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 13 04:09:55.002788 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 13 04:09:55.002788 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 13 04:09:55.002788 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 13 04:09:55.002788 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 13 04:09:55.002788 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Oct 13 04:09:55.002788 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application return self._get_project(project_id) Oct 13 04:09:55.002788 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Oct 13 04:09:55.003838 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Oct 13 04:09:55.003838 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 13 04:09:55.003838 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 13 04:09:55.003838 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 13 04:09:55.003838 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 13 04:09:55.003838 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 13 04:09:55.003838 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application return self.get_or_create( Oct 13 04:09:55.003838 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 13 04:09:55.003838 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application with Lock( Oct 13 04:09:55.003838 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 13 04:09:55.003838 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application return self._enter() Oct 13 04:09:55.003838 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 13 04:09:55.003838 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 13 04:09:55.003838 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 13 04:09:55.003838 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application return self.creator() Oct 13 04:09:55.003838 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 13 04:09:55.003838 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application created_value = creator( Oct 13 04:09:55.010224 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Oct 13 04:09:55.010224 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Oct 13 04:09:55.010224 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Oct 13 04:09:55.010224 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Oct 13 04:09:55.010224 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Oct 13 04:09:55.010224 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Oct 13 04:09:55.010224 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Oct 13 04:09:55.010224 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application Oct 13 04:09:55.010224 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 408/837] 10.222.0.27 () {56 vars in 1111 bytes} [Thu Oct 13 04:09:54 2022] GET /identity/v3/projects/service => generated 88 bytes in 155 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Oct 13 04:09:55.010712 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ae09475d-a713-446c-b2f3-acdcee4532e0 None admin] Authenticating user token {{(pid=81409) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:09:55.106843 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ae09475d-a713-446c-b2f3-acdcee4532e0 None admin] Validating token access rules against request {{(pid=81409) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:09:55.139343 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-41484249-f95f-4345-aca9-8031c63c75fb None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81409) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:09:55.139960 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41484249-f95f-4345-aca9-8031c63c75fb None admin] REQUEST_METHOD: `GET` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:09:55.140231 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41484249-f95f-4345-aca9-8031c63c75fb None admin] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:09:55.140496 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41484249-f95f-4345-aca9-8031c63c75fb None admin] PATH_INFO: `/v3/projects` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:09:55.141192 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-41484249-f95f-4345-aca9-8031c63c75fb None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81409) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 13 04:09:55.141488 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-41484249-f95f-4345-aca9-8031c63c75fb None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:09:55.142973 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-41484249-f95f-4345-aca9-8031c63c75fb None admin] RBAC: Authorization granted {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:09:55.156209 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 430/838] 10.222.0.27 () {56 vars in 1148 bytes} [Thu Oct 13 04:09:55 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 146 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 13 04:09:55.162251 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be3f45b9-2323-4b80-a9dc-5d1eab506595 None admin] Authenticating user token {{(pid=81408) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:09:55.260640 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be3f45b9-2323-4b80-a9dc-5d1eab506595 None admin] Validating token access rules against request {{(pid=81408) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:09:55.293507 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-809166ba-7bec-4e52-9d43-b9024a191c76 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81408) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:09:55.294089 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-809166ba-7bec-4e52-9d43-b9024a191c76 None admin] REQUEST_METHOD: `GET` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:09:55.294389 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-809166ba-7bec-4e52-9d43-b9024a191c76 None admin] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:09:55.294659 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-809166ba-7bec-4e52-9d43-b9024a191c76 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:09:55.295367 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-809166ba-7bec-4e52-9d43-b9024a191c76 None admin] RBAC: Adding query filter params (role.id=0b094166c73d4276a8e8eec82da7f5ff, scope.project.id=dcec0431fb6849499d6aec35a234daa6, user.id=cb740ed7f7684b079e45f887be83f795) {{(pid=81408) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 13 04:09:55.295671 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-809166ba-7bec-4e52-9d43-b9024a191c76 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:09:55.297091 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-809166ba-7bec-4e52-9d43-b9024a191c76 None admin] RBAC: Authorization granted {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:09:55.302655 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 409/839] 10.222.0.27 () {56 vars in 1374 bytes} [Thu Oct 13 04:09:55 2022] GET /identity/v3/role_assignments?user.id=cb740ed7f7684b079e45f887be83f795&scope.project.id=dcec0431fb6849499d6aec35a234daa6&role.id=0b094166c73d4276a8e8eec82da7f5ff => generated 608 bytes in 141 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 13 04:09:56.434481 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c503a92d-dece-4b52-828e-fde3917c565f None None] REQUEST_METHOD: `GET` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:09:56.434972 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c503a92d-dece-4b52-828e-fde3917c565f None None] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:09:56.435066 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c503a92d-dece-4b52-828e-fde3917c565f None None] PATH_INFO: `/` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:09:56.436341 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 431/840] 10.222.0.27 () {54 vars in 927 bytes} [Thu Oct 13 04:09:56 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 13 04:09:56.441517 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e64d64ae-c873-4c96-8fa4-e91f4f9d8e6a None None] REQUEST_METHOD: `POST` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:09:56.441891 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e64d64ae-c873-4c96-8fa4-e91f4f9d8e6a None None] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:09:56.442050 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e64d64ae-c873-4c96-8fa4-e91f4f9d8e6a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:09:56.460691 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.auth.core [None req-e64d64ae-c873-4c96-8fa4-e91f4f9d8e6a None None] MFA Rules not processed for user `0e2d979563d6421995451a0e86bf3022`. Rule list: `[]` (Enabled: `True`). {{(pid=81408) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 13 04:09:56.462150 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.fernet_utils [None req-e64d64ae-c873-4c96-8fa4-e91f4f9d8e6a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81408) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:09:56.502846 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG dbcounter [-] [81408] Writing DB stats keystone:SELECT=100 {{(pid=81408) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:09:56.527200 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 410/841] 10.222.0.27 () {58 vars in 1024 bytes} [Thu Oct 13 04:09:56 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 87 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Oct 13 04:09:56.801633 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c503a92d-dece-4b52-828e-fde3917c565f None None] Authenticating user token {{(pid=81409) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:09:56.813129 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG dbcounter [-] [81409] Writing DB stats keystone:SELECT=100 {{(pid=81409) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:09:56.901840 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c503a92d-dece-4b52-828e-fde3917c565f None None] Validating token access rules against request {{(pid=81409) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:09:56.934790 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ce37874e-88cd-4318-8471-bf0f265722bb None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81409) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:09:56.935396 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce37874e-88cd-4318-8471-bf0f265722bb None admin] REQUEST_METHOD: `GET` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:09:56.935665 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce37874e-88cd-4318-8471-bf0f265722bb None admin] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:09:56.935928 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce37874e-88cd-4318-8471-bf0f265722bb None admin] PATH_INFO: `/v3/domains/default` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:09:56.937010 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ce37874e-88cd-4318-8471-bf0f265722bb None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:09:56.938437 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ce37874e-88cd-4318-8471-bf0f265722bb None admin] RBAC: Authorization granted {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:09:56.939971 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 432/842] 10.222.0.27 () {56 vars in 1108 bytes} [Thu Oct 13 04:09:56 2022] GET /identity/v3/domains/default => generated 199 bytes in 139 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 13 04:09:56.945787 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e64d64ae-c873-4c96-8fa4-e91f4f9d8e6a None None] Authenticating user token {{(pid=81408) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:09:57.043273 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e64d64ae-c873-4c96-8fa4-e91f4f9d8e6a None None] Validating token access rules against request {{(pid=81408) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:09:57.076744 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-33ee84bc-43e9-40e8-9186-6f2fc185fcf6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81408) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:09:57.077270 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33ee84bc-43e9-40e8-9186-6f2fc185fcf6 None admin] REQUEST_METHOD: `POST` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:09:57.077545 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33ee84bc-43e9-40e8-9186-6f2fc185fcf6 None admin] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:09:57.077814 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33ee84bc-43e9-40e8-9186-6f2fc185fcf6 None admin] PATH_INFO: `/v3/users` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:09:57.078495 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-33ee84bc-43e9-40e8-9186-6f2fc185fcf6 None admin] RBAC: Authorizing `identity:create_user()` {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:09:57.079908 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-33ee84bc-43e9-40e8-9186-6f2fc185fcf6 None admin] RBAC: Authorization granted {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:09:57.100996 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 411/843] 10.222.0.27 () {60 vars in 1132 bytes} [Thu Oct 13 04:09:56 2022] POST /identity/v3/users => generated 286 bytes in 156 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 13 04:09:59.245458 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-163d7a76-1b44-4eff-bb5c-dab5dccfba26 None None] REQUEST_METHOD: `GET` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:09:59.246010 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-163d7a76-1b44-4eff-bb5c-dab5dccfba26 None None] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:09:59.246010 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-163d7a76-1b44-4eff-bb5c-dab5dccfba26 None None] PATH_INFO: `/` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:09:59.247355 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 433/844] 10.222.0.27 () {54 vars in 927 bytes} [Thu Oct 13 04:09:59 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 13 04:09:59.252601 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2928beaa-ca84-483e-babe-e6302366500c None None] REQUEST_METHOD: `POST` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:09:59.252962 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2928beaa-ca84-483e-babe-e6302366500c None None] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:09:59.253151 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2928beaa-ca84-483e-babe-e6302366500c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:09:59.271353 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.auth.core [None req-2928beaa-ca84-483e-babe-e6302366500c None None] MFA Rules not processed for user `0e2d979563d6421995451a0e86bf3022`. Rule list: `[]` (Enabled: `True`). {{(pid=81408) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 13 04:09:59.272530 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.fernet_utils [None req-2928beaa-ca84-483e-babe-e6302366500c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81408) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:09:59.298374 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG dbcounter [-] [81408] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81408) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:09:59.340229 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 412/845] 10.222.0.27 () {58 vars in 1024 bytes} [Thu Oct 13 04:09:59 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 89 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Oct 13 04:09:59.610802 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-163d7a76-1b44-4eff-bb5c-dab5dccfba26 None None] Authenticating user token {{(pid=81409) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:09:59.687744 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG dbcounter [-] [81409] Writing DB stats keystone:SELECT=100 {{(pid=81409) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:09:59.711454 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-163d7a76-1b44-4eff-bb5c-dab5dccfba26 None None] Validating token access rules against request {{(pid=81409) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:09:59.744254 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-11584d14-18ef-43f1-ac1c-4b54969f27de None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81409) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:09:59.744857 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11584d14-18ef-43f1-ac1c-4b54969f27de None admin] REQUEST_METHOD: `GET` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:09:59.745127 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11584d14-18ef-43f1-ac1c-4b54969f27de None admin] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:09:59.745392 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11584d14-18ef-43f1-ac1c-4b54969f27de None admin] PATH_INFO: `/v3/roles/743995e93fef47fd91204c3d701054b0` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:09:59.746779 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-11584d14-18ef-43f1-ac1c-4b54969f27de None admin] RBAC: Authorizing `identity:get_role(role_id=743995e93fef47fd91204c3d701054b0)` {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:09:59.748140 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-11584d14-18ef-43f1-ac1c-4b54969f27de None admin] RBAC: Authorization granted {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:09:59.749426 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 434/846] 10.222.0.27 () {56 vars in 1177 bytes} [Thu Oct 13 04:09:59 2022] GET /identity/v3/roles/743995e93fef47fd91204c3d701054b0 => generated 223 bytes in 139 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 13 04:09:59.755188 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2928beaa-ca84-483e-babe-e6302366500c None None] Authenticating user token {{(pid=81408) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:09:59.851031 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2928beaa-ca84-483e-babe-e6302366500c None None] Validating token access rules against request {{(pid=81408) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:09:59.884674 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8809d66b-4588-4e7e-91b1-534374e3ab4e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81408) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:09:59.885227 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8809d66b-4588-4e7e-91b1-534374e3ab4e None admin] REQUEST_METHOD: `GET` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:09:59.885496 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8809d66b-4588-4e7e-91b1-534374e3ab4e None admin] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:09:59.885757 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8809d66b-4588-4e7e-91b1-534374e3ab4e None admin] PATH_INFO: `/v3/users/65e3207dbce144c0afcbf282f577a2ba` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:09:59.898137 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8809d66b-4588-4e7e-91b1-534374e3ab4e None admin] RBAC: Authorizing `identity:get_user(user_id=65e3207dbce144c0afcbf282f577a2ba)` {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:09:59.899919 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8809d66b-4588-4e7e-91b1-534374e3ab4e None admin] RBAC: Authorization granted {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:09:59.901457 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 413/847] 10.222.0.27 () {56 vars in 1177 bytes} [Thu Oct 13 04:09:59 2022] GET /identity/v3/users/65e3207dbce144c0afcbf282f577a2ba => generated 286 bytes in 147 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 13 04:09:59.907131 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-11584d14-18ef-43f1-ac1c-4b54969f27de None admin] Authenticating user token {{(pid=81409) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:10:00.003857 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-11584d14-18ef-43f1-ac1c-4b54969f27de None admin] Validating token access rules against request {{(pid=81409) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:10:00.038713 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c8568b0d-b349-46ec-9ab4-6a2d1e06da7a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81409) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:10:00.040042 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8568b0d-b349-46ec-9ab4-6a2d1e06da7a None admin] REQUEST_METHOD: `GET` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:10:00.040317 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8568b0d-b349-46ec-9ab4-6a2d1e06da7a None admin] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:10:00.040636 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8568b0d-b349-46ec-9ab4-6a2d1e06da7a None admin] PATH_INFO: `/v3/projects/78b3aa8e26634401b668ae1402864d8b` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:10:00.041781 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c8568b0d-b349-46ec-9ab4-6a2d1e06da7a None admin] RBAC: Authorizing `identity:get_project(project_id=78b3aa8e26634401b668ae1402864d8b)` {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:10:00.043214 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c8568b0d-b349-46ec-9ab4-6a2d1e06da7a None admin] RBAC: Authorization granted {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:10:00.044817 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 435/848] 10.222.0.27 () {56 vars in 1186 bytes} [Thu Oct 13 04:09:59 2022] GET /identity/v3/projects/78b3aa8e26634401b668ae1402864d8b => generated 311 bytes in 138 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 13 04:10:00.050712 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8809d66b-4588-4e7e-91b1-534374e3ab4e None admin] Authenticating user token {{(pid=81408) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:10:00.072211 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG dbcounter [-] [81408] Writing DB stats keystone:SELECT=100 {{(pid=81408) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:10:00.147067 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8809d66b-4588-4e7e-91b1-534374e3ab4e None admin] Validating token access rules against request {{(pid=81408) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:10:00.180114 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-29f95bf1-3a05-4f84-8bc1-04764b508e87 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81408) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:10:00.180743 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29f95bf1-3a05-4f84-8bc1-04764b508e87 None admin] REQUEST_METHOD: `PUT` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:10:00.181020 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29f95bf1-3a05-4f84-8bc1-04764b508e87 None admin] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:10:00.181292 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29f95bf1-3a05-4f84-8bc1-04764b508e87 None admin] PATH_INFO: `/v3/projects/78b3aa8e26634401b668ae1402864d8b/users/65e3207dbce144c0afcbf282f577a2ba/roles/743995e93fef47fd91204c3d701054b0` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:10:00.183449 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-29f95bf1-3a05-4f84-8bc1-04764b508e87 None admin] RBAC: Authorizing `identity:create_grant(project_id=78b3aa8e26634401b668ae1402864d8b, user_id=65e3207dbce144c0afcbf282f577a2ba, role_id=743995e93fef47fd91204c3d701054b0)` {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:10:00.184882 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-29f95bf1-3a05-4f84-8bc1-04764b508e87 None admin] RBAC: Authorization granted {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:10:00.197603 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 414/849] 10.222.0.27 () {58 vars in 1439 bytes} [Thu Oct 13 04:10:00 2022] PUT /identity/v3/projects/78b3aa8e26634401b668ae1402864d8b/users/65e3207dbce144c0afcbf282f577a2ba/roles/743995e93fef47fd91204c3d701054b0 => generated 0 bytes in 147 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 13 04:10:02.298921 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-547b791f-e1b9-45d8-a64f-33d324340707 None None] REQUEST_METHOD: `GET` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:10:02.299456 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-547b791f-e1b9-45d8-a64f-33d324340707 None None] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:10:02.299456 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-547b791f-e1b9-45d8-a64f-33d324340707 None None] PATH_INFO: `/` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:10:02.300795 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 436/850] 10.222.0.27 () {54 vars in 927 bytes} [Thu Oct 13 04:10:02 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 13 04:10:02.306796 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f362345-877f-425a-b2b0-9166a91af447 None None] REQUEST_METHOD: `POST` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:10:02.307440 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f362345-877f-425a-b2b0-9166a91af447 None None] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:10:02.307715 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f362345-877f-425a-b2b0-9166a91af447 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:10:02.325249 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.auth.core [None req-4f362345-877f-425a-b2b0-9166a91af447 None None] MFA Rules not processed for user `0e2d979563d6421995451a0e86bf3022`. Rule list: `[]` (Enabled: `True`). {{(pid=81408) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 13 04:10:02.326461 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.fernet_utils [None req-4f362345-877f-425a-b2b0-9166a91af447 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81408) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:10:02.389499 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 415/851] 10.222.0.27 () {58 vars in 1024 bytes} [Thu Oct 13 04:10:02 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 84 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Oct 13 04:10:02.661032 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-547b791f-e1b9-45d8-a64f-33d324340707 None None] Authenticating user token {{(pid=81409) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:10:02.675742 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG dbcounter [-] [81409] Writing DB stats keystone:SELECT=100 {{(pid=81409) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:10:02.761673 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-547b791f-e1b9-45d8-a64f-33d324340707 None None] Validating token access rules against request {{(pid=81409) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:10:02.794417 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d1db51d8-e940-4810-b681-362d0042e692 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81409) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:10:02.795031 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1db51d8-e940-4810-b681-362d0042e692 None admin] REQUEST_METHOD: `GET` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:10:02.795304 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1db51d8-e940-4810-b681-362d0042e692 None admin] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:10:02.795572 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1db51d8-e940-4810-b681-362d0042e692 None admin] PATH_INFO: `/v3/roles/743995e93fef47fd91204c3d701054b0` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:10:02.796971 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d1db51d8-e940-4810-b681-362d0042e692 None admin] RBAC: Authorizing `identity:get_role(role_id=743995e93fef47fd91204c3d701054b0)` {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:10:02.798364 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d1db51d8-e940-4810-b681-362d0042e692 None admin] RBAC: Authorization granted {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:10:02.799673 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 437/852] 10.222.0.27 () {56 vars in 1177 bytes} [Thu Oct 13 04:10:02 2022] GET /identity/v3/roles/743995e93fef47fd91204c3d701054b0 => generated 223 bytes in 139 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 13 04:10:02.805999 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4f362345-877f-425a-b2b0-9166a91af447 None None] Authenticating user token {{(pid=81408) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:10:02.817247 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG dbcounter [-] [81408] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81408) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:10:02.902255 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4f362345-877f-425a-b2b0-9166a91af447 None None] Validating token access rules against request {{(pid=81408) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:10:02.936114 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9edfc3dc-201d-437f-b67e-638dcffd16ca None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81408) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:10:02.936664 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9edfc3dc-201d-437f-b67e-638dcffd16ca None admin] REQUEST_METHOD: `GET` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:10:02.936936 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9edfc3dc-201d-437f-b67e-638dcffd16ca None admin] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:10:02.937201 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9edfc3dc-201d-437f-b67e-638dcffd16ca None admin] PATH_INFO: `/v3/users/65e3207dbce144c0afcbf282f577a2ba` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:10:02.938317 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9edfc3dc-201d-437f-b67e-638dcffd16ca None admin] RBAC: Authorizing `identity:get_user(user_id=65e3207dbce144c0afcbf282f577a2ba)` {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:10:02.939733 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9edfc3dc-201d-437f-b67e-638dcffd16ca None admin] RBAC: Authorization granted {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:10:02.941262 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 416/853] 10.222.0.27 () {56 vars in 1177 bytes} [Thu Oct 13 04:10:02 2022] GET /identity/v3/users/65e3207dbce144c0afcbf282f577a2ba => generated 286 bytes in 136 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 13 04:10:02.947233 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d1db51d8-e940-4810-b681-362d0042e692 None admin] Authenticating user token {{(pid=81409) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:10:03.027530 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG dbcounter [-] [81409] Writing DB stats keystone:SELECT=100 {{(pid=81409) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:10:03.047552 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d1db51d8-e940-4810-b681-362d0042e692 None admin] Validating token access rules against request {{(pid=81409) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:10:03.080506 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4f05e2ea-ef8a-46c6-9b12-0fbe20b7a6ab None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81409) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:10:03.081096 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f05e2ea-ef8a-46c6-9b12-0fbe20b7a6ab None admin] REQUEST_METHOD: `GET` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:10:03.081377 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f05e2ea-ef8a-46c6-9b12-0fbe20b7a6ab None admin] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:10:03.081649 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f05e2ea-ef8a-46c6-9b12-0fbe20b7a6ab None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:10:03.145932 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4f05e2ea-ef8a-46c6-9b12-0fbe20b7a6ab None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:10:03.147386 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4f05e2ea-ef8a-46c6-9b12-0fbe20b7a6ab None admin] RBAC: Authorization granted {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:10:03.223251 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG dbcounter [-] [81409] Writing DB stats keystone:SELECT=100 {{(pid=81409) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:10:03.242796 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 438/854] 10.222.0.27 () {58 vars in 1282 bytes} [Thu Oct 13 04:10:02 2022] GET /identity/v3/auth/tokens => generated 2118 bytes in 296 msecs (HTTP/1.1 200) 6 headers in 359 bytes (1 switches on core 0) Oct 13 04:10:03.248689 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9edfc3dc-201d-437f-b67e-638dcffd16ca None admin] Authenticating user token {{(pid=81408) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:10:03.323183 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG dbcounter [-] [81408] Writing DB stats keystone:SELECT=100 {{(pid=81408) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:10:03.343850 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9edfc3dc-201d-437f-b67e-638dcffd16ca None admin] Validating token access rules against request {{(pid=81408) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:10:03.376443 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b4f3ea8e-c7cd-443e-9bad-30da70e910b2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81408) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:10:03.377053 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4f3ea8e-c7cd-443e-9bad-30da70e910b2 None admin] REQUEST_METHOD: `GET` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:10:03.377329 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4f3ea8e-c7cd-443e-9bad-30da70e910b2 None admin] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:10:03.377599 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4f3ea8e-c7cd-443e-9bad-30da70e910b2 None admin] PATH_INFO: `/v3/projects/78b3aa8e26634401b668ae1402864d8b` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:10:03.378780 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b4f3ea8e-c7cd-443e-9bad-30da70e910b2 None admin] RBAC: Authorizing `identity:get_project(project_id=78b3aa8e26634401b668ae1402864d8b)` {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:10:03.380205 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b4f3ea8e-c7cd-443e-9bad-30da70e910b2 None admin] RBAC: Authorization granted {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:10:03.381802 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 417/855] 10.222.0.27 () {56 vars in 1186 bytes} [Thu Oct 13 04:10:03 2022] GET /identity/v3/projects/78b3aa8e26634401b668ae1402864d8b => generated 311 bytes in 134 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 13 04:10:03.388746 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4f05e2ea-ef8a-46c6-9b12-0fbe20b7a6ab None admin] Authenticating user token {{(pid=81409) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:10:03.485778 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4f05e2ea-ef8a-46c6-9b12-0fbe20b7a6ab None admin] Validating token access rules against request {{(pid=81409) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:10:03.519294 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f1a4e0ee-9b59-4b3f-9d48-79e37f22b116 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81409) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:10:03.519880 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1a4e0ee-9b59-4b3f-9d48-79e37f22b116 None admin] REQUEST_METHOD: `GET` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:10:03.520150 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1a4e0ee-9b59-4b3f-9d48-79e37f22b116 None admin] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:10:03.520413 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1a4e0ee-9b59-4b3f-9d48-79e37f22b116 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:10:03.521122 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f1a4e0ee-9b59-4b3f-9d48-79e37f22b116 None admin] RBAC: Adding query filter params (role.id=743995e93fef47fd91204c3d701054b0, scope.project.id=78b3aa8e26634401b668ae1402864d8b, user.id=65e3207dbce144c0afcbf282f577a2ba) {{(pid=81409) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 13 04:10:03.521423 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f1a4e0ee-9b59-4b3f-9d48-79e37f22b116 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:10:03.522850 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f1a4e0ee-9b59-4b3f-9d48-79e37f22b116 None admin] RBAC: Authorization granted {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:10:03.528708 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 439/856] 10.222.0.27 () {56 vars in 1374 bytes} [Thu Oct 13 04:10:03 2022] GET /identity/v3/role_assignments?user.id=65e3207dbce144c0afcbf282f577a2ba&scope.project.id=78b3aa8e26634401b668ae1402864d8b&role.id=743995e93fef47fd91204c3d701054b0 => generated 608 bytes in 140 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 13 04:10:05.651279 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23340a08-27a6-4bbe-b5da-3a131de696cc None None] REQUEST_METHOD: `GET` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:10:05.651850 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23340a08-27a6-4bbe-b5da-3a131de696cc None None] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:10:05.651850 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23340a08-27a6-4bbe-b5da-3a131de696cc None None] PATH_INFO: `/` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:10:05.653138 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 418/857] 10.222.0.27 () {54 vars in 927 bytes} [Thu Oct 13 04:10:05 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 13 04:10:05.658306 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5756774d-0972-40ef-8b38-92c8fe92d81f None None] REQUEST_METHOD: `POST` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:10:05.658707 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5756774d-0972-40ef-8b38-92c8fe92d81f None None] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:10:05.658849 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5756774d-0972-40ef-8b38-92c8fe92d81f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:10:05.677148 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.auth.core [None req-5756774d-0972-40ef-8b38-92c8fe92d81f None None] MFA Rules not processed for user `0e2d979563d6421995451a0e86bf3022`. Rule list: `[]` (Enabled: `True`). {{(pid=81409) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 13 04:10:05.678289 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.fernet_utils [None req-5756774d-0972-40ef-8b38-92c8fe92d81f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81409) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:10:05.716590 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG dbcounter [-] [81409] Writing DB stats keystone:SELECT=100 {{(pid=81409) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:10:05.743438 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 440/858] 10.222.0.27 () {58 vars in 1024 bytes} [Thu Oct 13 04:10:05 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 86 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Oct 13 04:10:06.011582 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-23340a08-27a6-4bbe-b5da-3a131de696cc None None] Authenticating user token {{(pid=81408) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:10:06.109767 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-23340a08-27a6-4bbe-b5da-3a131de696cc None None] Validating token access rules against request {{(pid=81408) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:10:06.143315 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6a3adf54-e1f1-46c7-bd44-fbb1e7740bc1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81408) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:10:06.143917 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a3adf54-e1f1-46c7-bd44-fbb1e7740bc1 None admin] REQUEST_METHOD: `GET` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:10:06.144189 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a3adf54-e1f1-46c7-bd44-fbb1e7740bc1 None admin] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:10:06.144451 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a3adf54-e1f1-46c7-bd44-fbb1e7740bc1 None admin] PATH_INFO: `/v3/domains/default` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:10:06.145532 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6a3adf54-e1f1-46c7-bd44-fbb1e7740bc1 None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:10:06.146969 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6a3adf54-e1f1-46c7-bd44-fbb1e7740bc1 None admin] RBAC: Authorization granted {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:10:06.149250 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 419/859] 10.222.0.27 () {56 vars in 1108 bytes} [Thu Oct 13 04:10:06 2022] GET /identity/v3/domains/default => generated 199 bytes in 138 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 13 04:10:06.154914 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5756774d-0972-40ef-8b38-92c8fe92d81f None None] Authenticating user token {{(pid=81409) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:10:06.251016 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5756774d-0972-40ef-8b38-92c8fe92d81f None None] Validating token access rules against request {{(pid=81409) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:10:06.284704 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b968ca2c-f3f5-4d4f-b6a2-908c85714226 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81409) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:10:06.285242 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b968ca2c-f3f5-4d4f-b6a2-908c85714226 None admin] REQUEST_METHOD: `POST` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:10:06.285517 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b968ca2c-f3f5-4d4f-b6a2-908c85714226 None admin] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:10:06.285786 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b968ca2c-f3f5-4d4f-b6a2-908c85714226 None admin] PATH_INFO: `/v3/projects` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:10:06.286450 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b968ca2c-f3f5-4d4f-b6a2-908c85714226 None admin] RBAC: Authorizing `identity:create_project()` {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:10:06.287861 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b968ca2c-f3f5-4d4f-b6a2-908c85714226 None admin] RBAC: Authorization granted {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:10:06.313669 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 441/860] 10.222.0.27 () {60 vars in 1141 bytes} [Thu Oct 13 04:10:06 2022] POST /identity/v3/projects => generated 311 bytes in 159 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 13 04:10:08.481487 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-801683f1-a59e-43d2-bcdf-4241fabadc67 None None] REQUEST_METHOD: `GET` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:10:08.482001 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-801683f1-a59e-43d2-bcdf-4241fabadc67 None None] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:10:08.482001 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-801683f1-a59e-43d2-bcdf-4241fabadc67 None None] PATH_INFO: `/` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:10:08.483392 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 420/861] 10.222.0.27 () {54 vars in 927 bytes} [Thu Oct 13 04:10:08 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 13 04:10:08.488517 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61cd093e-7e08-48a6-bf1f-b4c5f6e33a16 None None] REQUEST_METHOD: `POST` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:10:08.488936 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61cd093e-7e08-48a6-bf1f-b4c5f6e33a16 None None] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:10:08.489035 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61cd093e-7e08-48a6-bf1f-b4c5f6e33a16 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:10:08.507435 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.auth.core [None req-61cd093e-7e08-48a6-bf1f-b4c5f6e33a16 None None] MFA Rules not processed for user `0e2d979563d6421995451a0e86bf3022`. Rule list: `[]` (Enabled: `True`). {{(pid=81409) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 13 04:10:08.509184 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.fernet_utils [None req-61cd093e-7e08-48a6-bf1f-b4c5f6e33a16 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81409) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:10:08.528932 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG dbcounter [-] [81409] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81409) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:10:08.575224 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 442/862] 10.222.0.27 () {58 vars in 1024 bytes} [Thu Oct 13 04:10:08 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 88 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Oct 13 04:10:08.850178 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-801683f1-a59e-43d2-bcdf-4241fabadc67 None None] Authenticating user token {{(pid=81408) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:10:08.866213 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG dbcounter [-] [81408] Writing DB stats keystone:SELECT=100 {{(pid=81408) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:10:08.949629 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-801683f1-a59e-43d2-bcdf-4241fabadc67 None None] Validating token access rules against request {{(pid=81408) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:10:08.982625 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7ef20036-ed0f-4119-b474-55bbb95f64c5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81408) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:10:08.983242 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ef20036-ed0f-4119-b474-55bbb95f64c5 None admin] REQUEST_METHOD: `GET` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:10:08.983521 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ef20036-ed0f-4119-b474-55bbb95f64c5 None admin] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:10:08.983792 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ef20036-ed0f-4119-b474-55bbb95f64c5 None admin] PATH_INFO: `/v3/domains/default` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:10:08.984898 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7ef20036-ed0f-4119-b474-55bbb95f64c5 None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:10:08.986349 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7ef20036-ed0f-4119-b474-55bbb95f64c5 None admin] RBAC: Authorization granted {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:10:08.987897 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 421/863] 10.222.0.27 () {56 vars in 1108 bytes} [Thu Oct 13 04:10:08 2022] GET /identity/v3/domains/default => generated 199 bytes in 138 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 13 04:10:08.994157 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-61cd093e-7e08-48a6-bf1f-b4c5f6e33a16 None None] Authenticating user token {{(pid=81409) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:10:09.089312 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-61cd093e-7e08-48a6-bf1f-b4c5f6e33a16 None None] Validating token access rules against request {{(pid=81409) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:10:09.122579 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dd3f281b-0909-4ca6-bcb2-3da99a2db8c3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81409) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:10:09.123107 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd3f281b-0909-4ca6-bcb2-3da99a2db8c3 None admin] REQUEST_METHOD: `POST` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:10:09.123379 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd3f281b-0909-4ca6-bcb2-3da99a2db8c3 None admin] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:10:09.123644 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd3f281b-0909-4ca6-bcb2-3da99a2db8c3 None admin] PATH_INFO: `/v3/users` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:10:09.124284 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dd3f281b-0909-4ca6-bcb2-3da99a2db8c3 None admin] RBAC: Authorizing `identity:create_user()` {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:10:09.125691 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dd3f281b-0909-4ca6-bcb2-3da99a2db8c3 None admin] RBAC: Authorization granted {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:10:09.145423 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 443/864] 10.222.0.27 () {60 vars in 1132 bytes} [Thu Oct 13 04:10:08 2022] POST /identity/v3/users => generated 286 bytes in 152 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 13 04:10:11.283364 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0cf1858c-cbb9-4d15-b7b2-0a6f694706b8 None None] REQUEST_METHOD: `GET` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:10:11.283915 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0cf1858c-cbb9-4d15-b7b2-0a6f694706b8 None None] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:10:11.283915 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0cf1858c-cbb9-4d15-b7b2-0a6f694706b8 None None] PATH_INFO: `/` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:10:11.285248 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 422/865] 10.222.0.27 () {54 vars in 927 bytes} [Thu Oct 13 04:10:11 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 13 04:10:11.290896 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6219006-85e8-451b-a82a-0b406cda32d7 None None] REQUEST_METHOD: `POST` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:10:11.291312 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6219006-85e8-451b-a82a-0b406cda32d7 None None] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:10:11.291441 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6219006-85e8-451b-a82a-0b406cda32d7 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:10:11.309861 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.auth.core [None req-f6219006-85e8-451b-a82a-0b406cda32d7 None None] MFA Rules not processed for user `0e2d979563d6421995451a0e86bf3022`. Rule list: `[]` (Enabled: `True`). {{(pid=81409) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 13 04:10:11.311071 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.fernet_utils [None req-f6219006-85e8-451b-a82a-0b406cda32d7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81409) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:10:11.318104 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG dbcounter [-] [81409] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81409) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:10:11.377231 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 444/866] 10.222.0.27 () {58 vars in 1024 bytes} [Thu Oct 13 04:10:11 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 87 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Oct 13 04:10:11.650749 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0cf1858c-cbb9-4d15-b7b2-0a6f694706b8 None None] Authenticating user token {{(pid=81408) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:10:11.731514 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG dbcounter [-] [81408] Writing DB stats keystone:SELECT=100 {{(pid=81408) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:10:11.750053 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0cf1858c-cbb9-4d15-b7b2-0a6f694706b8 None None] Validating token access rules against request {{(pid=81408) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:10:11.783808 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f555cc37-792f-426f-813d-c295af91da4f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81408) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:10:11.784481 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f555cc37-792f-426f-813d-c295af91da4f None admin] REQUEST_METHOD: `GET` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:10:11.784755 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f555cc37-792f-426f-813d-c295af91da4f None admin] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:10:11.785021 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f555cc37-792f-426f-813d-c295af91da4f None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:10:11.794899 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f555cc37-792f-426f-813d-c295af91da4f None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 13 04:10:11.799585 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application [None req-f555cc37-792f-426f-813d-c295af91da4f None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Oct 13 04:10:11.799585 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 13 04:10:11.799585 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 13 04:10:11.799585 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 13 04:10:11.799585 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 13 04:10:11.799585 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 13 04:10:11.799585 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 13 04:10:11.799585 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 13 04:10:11.799585 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 13 04:10:11.799585 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 13 04:10:11.799585 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 13 04:10:11.799585 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 13 04:10:11.799585 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 13 04:10:11.799585 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 13 04:10:11.799585 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 13 04:10:11.799585 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 13 04:10:11.800720 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 13 04:10:11.800720 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 13 04:10:11.800720 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 13 04:10:11.800720 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 13 04:10:11.800720 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 13 04:10:11.800720 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 13 04:10:11.800720 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 13 04:10:11.800720 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 13 04:10:11.800720 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 13 04:10:11.800720 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application return self.get_or_create( Oct 13 04:10:11.800720 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 13 04:10:11.800720 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application with Lock( Oct 13 04:10:11.800720 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 13 04:10:11.800720 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application return self._enter() Oct 13 04:10:11.800720 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 13 04:10:11.800720 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 13 04:10:11.801840 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 13 04:10:11.801840 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application return self.creator() Oct 13 04:10:11.801840 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 13 04:10:11.801840 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application created_value = creator( Oct 13 04:10:11.801840 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 13 04:10:11.801840 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 13 04:10:11.801840 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 13 04:10:11.801840 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 13 04:10:11.801840 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 13 04:10:11.801840 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 13 04:10:11.801840 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Oct 13 04:10:11.801840 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application Oct 13 04:10:11.801840 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 423/867] 10.222.0.27 () {56 vars in 1096 bytes} [Thu Oct 13 04:10:11 2022] GET /identity/v3/roles/admin => generated 83 bytes in 151 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 13 04:10:11.806709 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f6219006-85e8-451b-a82a-0b406cda32d7 None None] Authenticating user token {{(pid=81409) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:10:11.903589 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f6219006-85e8-451b-a82a-0b406cda32d7 None None] Validating token access rules against request {{(pid=81409) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:10:11.936149 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-390d08b4-f112-4c4d-abed-94f5dd54d21e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81409) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:10:11.936738 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-390d08b4-f112-4c4d-abed-94f5dd54d21e None admin] REQUEST_METHOD: `GET` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:10:11.937009 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-390d08b4-f112-4c4d-abed-94f5dd54d21e None admin] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:10:11.937275 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-390d08b4-f112-4c4d-abed-94f5dd54d21e None admin] PATH_INFO: `/v3/roles` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:10:11.937924 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-390d08b4-f112-4c4d-abed-94f5dd54d21e None admin] RBAC: Adding query filter params (name=admin) {{(pid=81409) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 13 04:10:11.938240 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-390d08b4-f112-4c4d-abed-94f5dd54d21e None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:10:11.939592 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-390d08b4-f112-4c4d-abed-94f5dd54d21e None admin] RBAC: Authorization granted {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:10:11.947896 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 445/868] 10.222.0.27 () {56 vars in 1099 bytes} [Thu Oct 13 04:10:11 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 142 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 13 04:10:11.953676 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f555cc37-792f-426f-813d-c295af91da4f None admin] Authenticating user token {{(pid=81408) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:10:12.052247 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f555cc37-792f-426f-813d-c295af91da4f None admin] Validating token access rules against request {{(pid=81408) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:10:12.085245 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cdb5c967-d9b7-4fbf-bde2-63b93a5fb147 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81408) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:10:12.086030 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cdb5c967-d9b7-4fbf-bde2-63b93a5fb147 None admin] REQUEST_METHOD: `GET` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:10:12.086337 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cdb5c967-d9b7-4fbf-bde2-63b93a5fb147 None admin] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:10:12.086613 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cdb5c967-d9b7-4fbf-bde2-63b93a5fb147 None admin] PATH_INFO: `/v3/users/4fa9021740354c13acba14b90de6ff49` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:10:12.097859 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG dbcounter [-] [81408] Writing DB stats keystone:SELECT=100 {{(pid=81408) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:10:12.100341 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cdb5c967-d9b7-4fbf-bde2-63b93a5fb147 None admin] RBAC: Authorizing `identity:get_user(user_id=4fa9021740354c13acba14b90de6ff49)` {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:10:12.102889 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cdb5c967-d9b7-4fbf-bde2-63b93a5fb147 None admin] RBAC: Authorization granted {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:10:12.104768 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 424/869] 10.222.0.27 () {56 vars in 1177 bytes} [Thu Oct 13 04:10:11 2022] GET /identity/v3/users/4fa9021740354c13acba14b90de6ff49 => generated 286 bytes in 152 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 13 04:10:12.110568 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-390d08b4-f112-4c4d-abed-94f5dd54d21e None admin] Authenticating user token {{(pid=81409) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:10:12.114243 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG dbcounter [-] [81409] Writing DB stats keystone:SELECT=100 {{(pid=81409) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:10:12.206379 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-390d08b4-f112-4c4d-abed-94f5dd54d21e None admin] Validating token access rules against request {{(pid=81409) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:10:12.239191 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0d060c78-9cf6-4eee-a167-fe4de35190cf None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81409) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:10:12.239810 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d060c78-9cf6-4eee-a167-fe4de35190cf None admin] REQUEST_METHOD: `GET` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:10:12.240083 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d060c78-9cf6-4eee-a167-fe4de35190cf None admin] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:10:12.240349 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d060c78-9cf6-4eee-a167-fe4de35190cf None admin] PATH_INFO: `/v3/projects/c86a50a7e795496a9f632329061dc2d5` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:10:12.241487 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0d060c78-9cf6-4eee-a167-fe4de35190cf None admin] RBAC: Authorizing `identity:get_project(project_id=c86a50a7e795496a9f632329061dc2d5)` {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:10:12.243100 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0d060c78-9cf6-4eee-a167-fe4de35190cf None admin] RBAC: Authorization granted {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:10:12.244719 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 446/870] 10.222.0.27 () {56 vars in 1186 bytes} [Thu Oct 13 04:10:12 2022] GET /identity/v3/projects/c86a50a7e795496a9f632329061dc2d5 => generated 311 bytes in 135 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 13 04:10:12.250573 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cdb5c967-d9b7-4fbf-bde2-63b93a5fb147 None admin] Authenticating user token {{(pid=81408) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:10:12.346891 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cdb5c967-d9b7-4fbf-bde2-63b93a5fb147 None admin] Validating token access rules against request {{(pid=81408) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:10:12.379931 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-48fb463b-bbcb-40c6-b7d0-f079e55fc5b9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81408) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:10:12.380569 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48fb463b-bbcb-40c6-b7d0-f079e55fc5b9 None admin] REQUEST_METHOD: `PUT` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:10:12.380844 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48fb463b-bbcb-40c6-b7d0-f079e55fc5b9 None admin] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:10:12.381114 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48fb463b-bbcb-40c6-b7d0-f079e55fc5b9 None admin] PATH_INFO: `/v3/projects/c86a50a7e795496a9f632329061dc2d5/users/4fa9021740354c13acba14b90de6ff49/roles/ef55fbd94e1040be83695754075a7ed5` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:10:12.383715 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-48fb463b-bbcb-40c6-b7d0-f079e55fc5b9 None admin] RBAC: Authorizing `identity:create_grant(project_id=c86a50a7e795496a9f632329061dc2d5, user_id=4fa9021740354c13acba14b90de6ff49, role_id=ef55fbd94e1040be83695754075a7ed5)` {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:10:12.385138 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-48fb463b-bbcb-40c6-b7d0-f079e55fc5b9 None admin] RBAC: Authorization granted {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:10:12.397337 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 425/871] 10.222.0.27 () {58 vars in 1439 bytes} [Thu Oct 13 04:10:12 2022] PUT /identity/v3/projects/c86a50a7e795496a9f632329061dc2d5/users/4fa9021740354c13acba14b90de6ff49/roles/ef55fbd94e1040be83695754075a7ed5 => generated 0 bytes in 147 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 13 04:10:14.502290 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d3f2d66-bc0a-4605-849a-a9bc947f8c3d None None] REQUEST_METHOD: `GET` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:10:14.502799 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d3f2d66-bc0a-4605-849a-a9bc947f8c3d None None] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:10:14.502917 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d3f2d66-bc0a-4605-849a-a9bc947f8c3d None None] PATH_INFO: `/` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:10:14.504180 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 447/872] 10.222.0.27 () {54 vars in 927 bytes} [Thu Oct 13 04:10:14 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 13 04:10:14.509895 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1415124e-ee2b-4109-b4ac-af2ce8bb0ceb None None] REQUEST_METHOD: `POST` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:10:14.510334 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1415124e-ee2b-4109-b4ac-af2ce8bb0ceb None None] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:10:14.510476 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1415124e-ee2b-4109-b4ac-af2ce8bb0ceb None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:10:14.528282 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.auth.core [None req-1415124e-ee2b-4109-b4ac-af2ce8bb0ceb None None] MFA Rules not processed for user `0e2d979563d6421995451a0e86bf3022`. Rule list: `[]` (Enabled: `True`). {{(pid=81408) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 13 04:10:14.529466 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.fernet_utils [None req-1415124e-ee2b-4109-b4ac-af2ce8bb0ceb None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81408) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:10:14.583006 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG dbcounter [-] [81408] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81408) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:10:14.595372 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 426/873] 10.222.0.27 () {58 vars in 1024 bytes} [Thu Oct 13 04:10:14 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 86 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Oct 13 04:10:14.863927 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8d3f2d66-bc0a-4605-849a-a9bc947f8c3d None None] Authenticating user token {{(pid=81409) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:10:14.933237 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG dbcounter [-] [81409] Writing DB stats keystone:SELECT=100 {{(pid=81409) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:10:14.963320 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8d3f2d66-bc0a-4605-849a-a9bc947f8c3d None None] Validating token access rules against request {{(pid=81409) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:10:14.995488 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5ddf3927-cfec-43ec-b0bb-8d605a53d8f3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81409) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:10:14.996090 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ddf3927-cfec-43ec-b0bb-8d605a53d8f3 None admin] REQUEST_METHOD: `GET` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:10:14.996364 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ddf3927-cfec-43ec-b0bb-8d605a53d8f3 None admin] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:10:14.996630 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ddf3927-cfec-43ec-b0bb-8d605a53d8f3 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:10:15.006732 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5ddf3927-cfec-43ec-b0bb-8d605a53d8f3 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 13 04:10:15.011968 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application [None req-5ddf3927-cfec-43ec-b0bb-8d605a53d8f3 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Oct 13 04:10:15.011968 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 13 04:10:15.011968 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 13 04:10:15.011968 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 13 04:10:15.011968 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 13 04:10:15.011968 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 13 04:10:15.011968 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 13 04:10:15.011968 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 13 04:10:15.011968 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 13 04:10:15.011968 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 13 04:10:15.011968 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 13 04:10:15.011968 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 13 04:10:15.011968 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 13 04:10:15.011968 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 13 04:10:15.011968 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 13 04:10:15.011968 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 13 04:10:15.013052 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 13 04:10:15.013052 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 13 04:10:15.013052 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 13 04:10:15.013052 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 13 04:10:15.013052 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 13 04:10:15.013052 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 13 04:10:15.013052 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 13 04:10:15.013052 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 13 04:10:15.013052 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 13 04:10:15.013052 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application return self.get_or_create( Oct 13 04:10:15.013052 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 13 04:10:15.013052 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application with Lock( Oct 13 04:10:15.013052 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 13 04:10:15.013052 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application return self._enter() Oct 13 04:10:15.013052 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 13 04:10:15.013052 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 13 04:10:15.013927 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 13 04:10:15.013927 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application return self.creator() Oct 13 04:10:15.013927 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 13 04:10:15.013927 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application created_value = creator( Oct 13 04:10:15.013927 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 13 04:10:15.013927 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 13 04:10:15.013927 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 13 04:10:15.013927 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 13 04:10:15.013927 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 13 04:10:15.013927 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 13 04:10:15.013927 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Oct 13 04:10:15.013927 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application Oct 13 04:10:15.013927 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 448/874] 10.222.0.27 () {56 vars in 1096 bytes} [Thu Oct 13 04:10:14 2022] GET /identity/v3/roles/admin => generated 83 bytes in 150 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 13 04:10:15.019094 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1415124e-ee2b-4109-b4ac-af2ce8bb0ceb None None] Authenticating user token {{(pid=81408) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:10:15.115207 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1415124e-ee2b-4109-b4ac-af2ce8bb0ceb None None] Validating token access rules against request {{(pid=81408) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:10:15.148295 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c060b296-2dea-40fd-a1ab-28cde6a6aa3c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81408) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:10:15.148875 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c060b296-2dea-40fd-a1ab-28cde6a6aa3c None admin] REQUEST_METHOD: `GET` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:10:15.149151 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c060b296-2dea-40fd-a1ab-28cde6a6aa3c None admin] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:10:15.149426 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c060b296-2dea-40fd-a1ab-28cde6a6aa3c None admin] PATH_INFO: `/v3/roles` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:10:15.150081 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c060b296-2dea-40fd-a1ab-28cde6a6aa3c None admin] RBAC: Adding query filter params (name=admin) {{(pid=81408) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 13 04:10:15.150409 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c060b296-2dea-40fd-a1ab-28cde6a6aa3c None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:10:15.151768 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c060b296-2dea-40fd-a1ab-28cde6a6aa3c None admin] RBAC: Authorization granted {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:10:15.159475 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 427/875] 10.222.0.27 () {56 vars in 1099 bytes} [Thu Oct 13 04:10:15 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 141 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 13 04:10:15.165317 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5ddf3927-cfec-43ec-b0bb-8d605a53d8f3 None admin] Authenticating user token {{(pid=81409) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:10:15.262615 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5ddf3927-cfec-43ec-b0bb-8d605a53d8f3 None admin] Validating token access rules against request {{(pid=81409) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:10:15.289840 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG dbcounter [-] [81409] Writing DB stats keystone:SELECT=100 {{(pid=81409) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:10:15.295095 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a53a8146-f4b2-4c23-bd1b-5205fa36d3a3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81409) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:10:15.295701 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a53a8146-f4b2-4c23-bd1b-5205fa36d3a3 None admin] REQUEST_METHOD: `GET` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:10:15.295973 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a53a8146-f4b2-4c23-bd1b-5205fa36d3a3 None admin] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:10:15.296236 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a53a8146-f4b2-4c23-bd1b-5205fa36d3a3 None admin] PATH_INFO: `/v3/users/4fa9021740354c13acba14b90de6ff49` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:10:15.297327 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a53a8146-f4b2-4c23-bd1b-5205fa36d3a3 None admin] RBAC: Authorizing `identity:get_user(user_id=4fa9021740354c13acba14b90de6ff49)` {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:10:15.298777 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a53a8146-f4b2-4c23-bd1b-5205fa36d3a3 None admin] RBAC: Authorization granted {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:10:15.300322 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 449/876] 10.222.0.27 () {56 vars in 1177 bytes} [Thu Oct 13 04:10:15 2022] GET /identity/v3/users/4fa9021740354c13acba14b90de6ff49 => generated 286 bytes in 135 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 13 04:10:15.306665 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c060b296-2dea-40fd-a1ab-28cde6a6aa3c None admin] Authenticating user token {{(pid=81408) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:10:15.358034 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG dbcounter [-] [81408] Writing DB stats keystone:SELECT=100 {{(pid=81408) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:10:15.404978 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c060b296-2dea-40fd-a1ab-28cde6a6aa3c None admin] Validating token access rules against request {{(pid=81408) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:10:15.438298 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-583ae137-3dad-47d7-b475-09331f729f8f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81408) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:10:15.438902 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-583ae137-3dad-47d7-b475-09331f729f8f None admin] REQUEST_METHOD: `GET` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:10:15.439178 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-583ae137-3dad-47d7-b475-09331f729f8f None admin] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:10:15.439450 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-583ae137-3dad-47d7-b475-09331f729f8f None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:10:15.504283 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-583ae137-3dad-47d7-b475-09331f729f8f None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:10:15.505749 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-583ae137-3dad-47d7-b475-09331f729f8f None admin] RBAC: Authorization granted {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:10:15.555256 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG dbcounter [-] [81408] Writing DB stats keystone:SELECT=100 {{(pid=81408) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:10:15.600964 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 428/877] 10.222.0.27 () {58 vars in 1282 bytes} [Thu Oct 13 04:10:15 2022] GET /identity/v3/auth/tokens => generated 2118 bytes in 295 msecs (HTTP/1.1 200) 6 headers in 359 bytes (1 switches on core 0) Oct 13 04:10:15.607814 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a53a8146-f4b2-4c23-bd1b-5205fa36d3a3 None admin] Authenticating user token {{(pid=81409) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:10:15.704262 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a53a8146-f4b2-4c23-bd1b-5205fa36d3a3 None admin] Validating token access rules against request {{(pid=81409) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:10:15.736204 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7011df2e-c1d0-4472-9e75-ebc5ee0bdf37 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81409) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:10:15.736808 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7011df2e-c1d0-4472-9e75-ebc5ee0bdf37 None admin] REQUEST_METHOD: `GET` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:10:15.737079 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7011df2e-c1d0-4472-9e75-ebc5ee0bdf37 None admin] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:10:15.737344 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7011df2e-c1d0-4472-9e75-ebc5ee0bdf37 None admin] PATH_INFO: `/v3/projects/c86a50a7e795496a9f632329061dc2d5` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:10:15.738452 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7011df2e-c1d0-4472-9e75-ebc5ee0bdf37 None admin] RBAC: Authorizing `identity:get_project(project_id=c86a50a7e795496a9f632329061dc2d5)` {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:10:15.739853 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7011df2e-c1d0-4472-9e75-ebc5ee0bdf37 None admin] RBAC: Authorization granted {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:10:15.741436 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 450/878] 10.222.0.27 () {56 vars in 1186 bytes} [Thu Oct 13 04:10:15 2022] GET /identity/v3/projects/c86a50a7e795496a9f632329061dc2d5 => generated 311 bytes in 134 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 13 04:10:15.748165 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-583ae137-3dad-47d7-b475-09331f729f8f None admin] Authenticating user token {{(pid=81408) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:10:15.843186 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-583ae137-3dad-47d7-b475-09331f729f8f None admin] Validating token access rules against request {{(pid=81408) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:10:15.876215 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-42b7f7aa-749a-453e-b426-c8182e418d97 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81408) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:10:15.876798 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42b7f7aa-749a-453e-b426-c8182e418d97 None admin] REQUEST_METHOD: `GET` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:10:15.877070 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42b7f7aa-749a-453e-b426-c8182e418d97 None admin] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:10:15.877336 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42b7f7aa-749a-453e-b426-c8182e418d97 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:10:15.878042 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-42b7f7aa-749a-453e-b426-c8182e418d97 None admin] RBAC: Adding query filter params (role.id=ef55fbd94e1040be83695754075a7ed5, scope.project.id=c86a50a7e795496a9f632329061dc2d5, user.id=4fa9021740354c13acba14b90de6ff49) {{(pid=81408) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 13 04:10:15.878376 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-42b7f7aa-749a-453e-b426-c8182e418d97 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:10:15.879783 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-42b7f7aa-749a-453e-b426-c8182e418d97 None admin] RBAC: Authorization granted {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:10:15.885326 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 429/879] 10.222.0.27 () {56 vars in 1374 bytes} [Thu Oct 13 04:10:15 2022] GET /identity/v3/role_assignments?user.id=4fa9021740354c13acba14b90de6ff49&scope.project.id=c86a50a7e795496a9f632329061dc2d5&role.id=ef55fbd94e1040be83695754075a7ed5 => generated 608 bytes in 138 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 13 04:10:18.010679 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6085c856-00a5-4df9-b644-11402a43ade0 None None] REQUEST_METHOD: `GET` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:10:18.011230 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6085c856-00a5-4df9-b644-11402a43ade0 None None] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:10:18.011230 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6085c856-00a5-4df9-b644-11402a43ade0 None None] PATH_INFO: `/` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:10:18.012542 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 451/880] 10.222.0.27 () {54 vars in 927 bytes} [Thu Oct 13 04:10:18 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 13 04:10:18.017631 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19a42af6-e349-4379-9852-003fdc9d9106 None None] REQUEST_METHOD: `POST` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:10:18.017994 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19a42af6-e349-4379-9852-003fdc9d9106 None None] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:10:18.018173 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19a42af6-e349-4379-9852-003fdc9d9106 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:10:18.036339 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.auth.core [None req-19a42af6-e349-4379-9852-003fdc9d9106 None None] MFA Rules not processed for user `0e2d979563d6421995451a0e86bf3022`. Rule list: `[]` (Enabled: `True`). {{(pid=81408) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 13 04:10:18.037514 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.fernet_utils [None req-19a42af6-e349-4379-9852-003fdc9d9106 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81408) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:10:18.047942 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG dbcounter [-] [81408] Writing DB stats keystone:SELECT=100 {{(pid=81408) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:10:18.102565 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 430/881] 10.222.0.27 () {58 vars in 1024 bytes} [Thu Oct 13 04:10:18 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 86 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Oct 13 04:10:18.369991 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6085c856-00a5-4df9-b644-11402a43ade0 None None] Authenticating user token {{(pid=81409) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:10:18.435700 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG dbcounter [-] [81409] Writing DB stats keystone:SELECT=100 {{(pid=81409) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:10:18.468867 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6085c856-00a5-4df9-b644-11402a43ade0 None None] Validating token access rules against request {{(pid=81409) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:10:18.501275 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bd1536f3-bcfa-4a3f-8cbe-e7932630fbf3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81409) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:10:18.501861 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd1536f3-bcfa-4a3f-8cbe-e7932630fbf3 None admin] REQUEST_METHOD: `POST` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:10:18.502129 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd1536f3-bcfa-4a3f-8cbe-e7932630fbf3 None admin] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:10:18.502419 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd1536f3-bcfa-4a3f-8cbe-e7932630fbf3 None admin] PATH_INFO: `/v3/domains` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:10:18.503079 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bd1536f3-bcfa-4a3f-8cbe-e7932630fbf3 None admin] RBAC: Authorizing `identity:create_domain()` {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:10:18.504416 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bd1536f3-bcfa-4a3f-8cbe-e7932630fbf3 None admin] RBAC: Authorization granted {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:10:18.519488 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 452/882] 10.222.0.27 () {60 vars in 1138 bytes} [Thu Oct 13 04:10:18 2022] POST /identity/v3/domains => generated 267 bytes in 150 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 13 04:10:20.709320 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e45df75-61c5-47f5-8461-2c9f2ecc5b61 None None] REQUEST_METHOD: `GET` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:10:20.709837 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e45df75-61c5-47f5-8461-2c9f2ecc5b61 None None] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:10:20.709837 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e45df75-61c5-47f5-8461-2c9f2ecc5b61 None None] PATH_INFO: `/` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:10:20.711218 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 431/883] 10.222.0.27 () {54 vars in 927 bytes} [Thu Oct 13 04:10:20 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 13 04:10:20.716417 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-581b6f8e-ff18-4086-8848-91fac725c80f None None] REQUEST_METHOD: `POST` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:10:20.716840 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-581b6f8e-ff18-4086-8848-91fac725c80f None None] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:10:20.716942 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-581b6f8e-ff18-4086-8848-91fac725c80f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:10:20.736615 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.auth.core [None req-581b6f8e-ff18-4086-8848-91fac725c80f None None] MFA Rules not processed for user `0e2d979563d6421995451a0e86bf3022`. Rule list: `[]` (Enabled: `True`). {{(pid=81409) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 13 04:10:20.737786 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.fernet_utils [None req-581b6f8e-ff18-4086-8848-91fac725c80f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81409) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:10:20.803142 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 453/884] 10.222.0.27 () {58 vars in 1024 bytes} [Thu Oct 13 04:10:20 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 88 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Oct 13 04:10:21.070221 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0e45df75-61c5-47f5-8461-2c9f2ecc5b61 None None] Authenticating user token {{(pid=81408) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:10:21.166959 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0e45df75-61c5-47f5-8461-2c9f2ecc5b61 None None] Validating token access rules against request {{(pid=81408) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:10:21.199146 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6d69820f-5faa-46b5-af31-8157d25c9f08 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81408) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:10:21.199754 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d69820f-5faa-46b5-af31-8157d25c9f08 None admin] REQUEST_METHOD: `GET` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:10:21.200025 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d69820f-5faa-46b5-af31-8157d25c9f08 None admin] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:10:21.200290 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d69820f-5faa-46b5-af31-8157d25c9f08 None admin] PATH_INFO: `/v3/domains/9deaa2891ca74c2091775d7d80644ea6` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:10:21.210050 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6d69820f-5faa-46b5-af31-8157d25c9f08 None admin] RBAC: Authorizing `identity:get_domain(domain_id=9deaa2891ca74c2091775d7d80644ea6)` {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:10:21.212072 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6d69820f-5faa-46b5-af31-8157d25c9f08 None admin] RBAC: Authorization granted {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:10:21.213678 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 432/885] 10.222.0.27 () {56 vars in 1183 bytes} [Thu Oct 13 04:10:21 2022] GET /identity/v3/domains/9deaa2891ca74c2091775d7d80644ea6 => generated 267 bytes in 145 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 13 04:10:21.219344 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-581b6f8e-ff18-4086-8848-91fac725c80f None None] Authenticating user token {{(pid=81409) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:10:21.275884 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG dbcounter [-] [81409] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81409) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:10:21.316598 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-581b6f8e-ff18-4086-8848-91fac725c80f None None] Validating token access rules against request {{(pid=81409) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:10:21.349305 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dde046e0-0101-4183-a9ec-7ee89d498e5c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81409) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:10:21.349842 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dde046e0-0101-4183-a9ec-7ee89d498e5c None admin] REQUEST_METHOD: `POST` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:10:21.350116 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dde046e0-0101-4183-a9ec-7ee89d498e5c None admin] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:10:21.350409 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dde046e0-0101-4183-a9ec-7ee89d498e5c None admin] PATH_INFO: `/v3/projects` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:10:21.351950 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dde046e0-0101-4183-a9ec-7ee89d498e5c None admin] RBAC: Authorizing `identity:create_project()` {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:10:21.353367 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dde046e0-0101-4183-a9ec-7ee89d498e5c None admin] RBAC: Authorization granted {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:10:21.378962 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 454/886] 10.222.0.27 () {60 vars in 1141 bytes} [Thu Oct 13 04:10:21 2022] POST /identity/v3/projects => generated 361 bytes in 160 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 13 04:10:23.555536 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ba19c6b-0318-4130-9e5f-665f4d2ef9c5 None None] REQUEST_METHOD: `GET` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:10:23.556076 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ba19c6b-0318-4130-9e5f-665f4d2ef9c5 None None] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:10:23.556076 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ba19c6b-0318-4130-9e5f-665f4d2ef9c5 None None] PATH_INFO: `/` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:10:23.557514 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 433/887] 10.222.0.27 () {54 vars in 927 bytes} [Thu Oct 13 04:10:23 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 13 04:10:23.562671 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fac979e0-2203-4041-bcf6-29a8a4f895d4 None None] REQUEST_METHOD: `POST` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:10:23.563108 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fac979e0-2203-4041-bcf6-29a8a4f895d4 None None] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:10:23.563207 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fac979e0-2203-4041-bcf6-29a8a4f895d4 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:10:23.581094 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.auth.core [None req-fac979e0-2203-4041-bcf6-29a8a4f895d4 None None] MFA Rules not processed for user `0e2d979563d6421995451a0e86bf3022`. Rule list: `[]` (Enabled: `True`). {{(pid=81409) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 13 04:10:23.582338 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.fernet_utils [None req-fac979e0-2203-4041-bcf6-29a8a4f895d4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81409) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:10:23.649776 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 455/888] 10.222.0.27 () {58 vars in 1024 bytes} [Thu Oct 13 04:10:23 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 88 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Oct 13 04:10:23.919183 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6ba19c6b-0318-4130-9e5f-665f4d2ef9c5 None None] Authenticating user token {{(pid=81408) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:10:23.928374 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG dbcounter [-] [81408] Writing DB stats keystone:SELECT=100 {{(pid=81408) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:10:24.024824 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6ba19c6b-0318-4130-9e5f-665f4d2ef9c5 None None] Validating token access rules against request {{(pid=81408) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:10:24.057110 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9a0da13c-b94c-4ffe-84c6-659290ea5bef None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81408) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:10:24.057717 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a0da13c-b94c-4ffe-84c6-659290ea5bef None admin] REQUEST_METHOD: `GET` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:10:24.057990 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a0da13c-b94c-4ffe-84c6-659290ea5bef None admin] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:10:24.058283 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a0da13c-b94c-4ffe-84c6-659290ea5bef None admin] PATH_INFO: `/v3/domains/9deaa2891ca74c2091775d7d80644ea6` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:10:24.059382 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9a0da13c-b94c-4ffe-84c6-659290ea5bef None admin] RBAC: Authorizing `identity:get_domain(domain_id=9deaa2891ca74c2091775d7d80644ea6)` {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:10:24.060802 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9a0da13c-b94c-4ffe-84c6-659290ea5bef None admin] RBAC: Authorization granted {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:10:24.062362 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 434/889] 10.222.0.27 () {56 vars in 1183 bytes} [Thu Oct 13 04:10:23 2022] GET /identity/v3/domains/9deaa2891ca74c2091775d7d80644ea6 => generated 267 bytes in 144 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 13 04:10:24.068148 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fac979e0-2203-4041-bcf6-29a8a4f895d4 None None] Authenticating user token {{(pid=81409) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:10:24.109738 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG dbcounter [-] [81409] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81409) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:10:24.166675 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fac979e0-2203-4041-bcf6-29a8a4f895d4 None None] Validating token access rules against request {{(pid=81409) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:10:24.199971 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8125d79f-2818-4556-9af2-ef10bcfd8c63 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81409) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:10:24.200509 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8125d79f-2818-4556-9af2-ef10bcfd8c63 None admin] REQUEST_METHOD: `POST` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:10:24.200783 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8125d79f-2818-4556-9af2-ef10bcfd8c63 None admin] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:10:24.201050 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8125d79f-2818-4556-9af2-ef10bcfd8c63 None admin] PATH_INFO: `/v3/users` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:10:24.201705 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8125d79f-2818-4556-9af2-ef10bcfd8c63 None admin] RBAC: Authorizing `identity:create_user()` {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:10:24.203153 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8125d79f-2818-4556-9af2-ef10bcfd8c63 None admin] RBAC: Authorization granted {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:10:24.223499 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 456/890] 10.222.0.27 () {60 vars in 1132 bytes} [Thu Oct 13 04:10:24 2022] POST /identity/v3/users => generated 311 bytes in 156 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 13 04:10:26.372934 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0b6ecc7-dcdc-495f-be58-a8a8adfa2190 None None] REQUEST_METHOD: `GET` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:10:26.373486 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0b6ecc7-dcdc-495f-be58-a8a8adfa2190 None None] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:10:26.373486 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0b6ecc7-dcdc-495f-be58-a8a8adfa2190 None None] PATH_INFO: `/` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:10:26.374869 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 435/891] 10.222.0.27 () {54 vars in 927 bytes} [Thu Oct 13 04:10:26 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 13 04:10:26.379951 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2dea562-c013-475a-86c5-49966145a250 None None] REQUEST_METHOD: `POST` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:10:26.380337 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2dea562-c013-475a-86c5-49966145a250 None None] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:10:26.380499 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2dea562-c013-475a-86c5-49966145a250 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:10:26.399039 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.auth.core [None req-f2dea562-c013-475a-86c5-49966145a250 None None] MFA Rules not processed for user `0e2d979563d6421995451a0e86bf3022`. Rule list: `[]` (Enabled: `True`). {{(pid=81409) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 13 04:10:26.400232 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.fernet_utils [None req-f2dea562-c013-475a-86c5-49966145a250 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81409) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:10:26.465682 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 457/892] 10.222.0.27 () {58 vars in 1024 bytes} [Thu Oct 13 04:10:26 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 87 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Oct 13 04:10:26.735306 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d0b6ecc7-dcdc-495f-be58-a8a8adfa2190 None None] Authenticating user token {{(pid=81408) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:10:26.808873 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG dbcounter [-] [81408] Writing DB stats keystone:SELECT=100 {{(pid=81408) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:10:26.839751 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d0b6ecc7-dcdc-495f-be58-a8a8adfa2190 None None] Validating token access rules against request {{(pid=81408) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:10:26.876988 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8255acd6-0382-4fc7-b403-d94a0af466c8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81408) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:10:26.877602 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8255acd6-0382-4fc7-b403-d94a0af466c8 None admin] REQUEST_METHOD: `GET` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:10:26.877880 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8255acd6-0382-4fc7-b403-d94a0af466c8 None admin] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:10:26.878147 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8255acd6-0382-4fc7-b403-d94a0af466c8 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:10:26.888659 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8255acd6-0382-4fc7-b403-d94a0af466c8 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 13 04:10:26.893379 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application [None req-8255acd6-0382-4fc7-b403-d94a0af466c8 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Oct 13 04:10:26.893379 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 13 04:10:26.893379 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 13 04:10:26.893379 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 13 04:10:26.893379 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 13 04:10:26.893379 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 13 04:10:26.893379 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 13 04:10:26.893379 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 13 04:10:26.893379 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 13 04:10:26.893379 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 13 04:10:26.893379 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 13 04:10:26.893379 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 13 04:10:26.893379 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 13 04:10:26.893379 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 13 04:10:26.893379 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 13 04:10:26.893379 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 13 04:10:26.894589 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 13 04:10:26.894589 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 13 04:10:26.894589 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 13 04:10:26.894589 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 13 04:10:26.894589 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 13 04:10:26.894589 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 13 04:10:26.894589 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 13 04:10:26.894589 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 13 04:10:26.894589 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 13 04:10:26.894589 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application return self.get_or_create( Oct 13 04:10:26.894589 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 13 04:10:26.894589 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application with Lock( Oct 13 04:10:26.894589 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 13 04:10:26.894589 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application return self._enter() Oct 13 04:10:26.894589 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 13 04:10:26.894589 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 13 04:10:26.895700 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 13 04:10:26.895700 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application return self.creator() Oct 13 04:10:26.895700 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 13 04:10:26.895700 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application created_value = creator( Oct 13 04:10:26.895700 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 13 04:10:26.895700 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 13 04:10:26.895700 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 13 04:10:26.895700 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 13 04:10:26.895700 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 13 04:10:26.895700 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 13 04:10:26.895700 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Oct 13 04:10:26.895700 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application Oct 13 04:10:26.895700 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 436/893] 10.222.0.27 () {56 vars in 1096 bytes} [Thu Oct 13 04:10:26 2022] GET /identity/v3/roles/admin => generated 83 bytes in 160 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 13 04:10:26.902056 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f2dea562-c013-475a-86c5-49966145a250 None None] Authenticating user token {{(pid=81409) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:10:26.927838 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG dbcounter [-] [81409] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81409) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:10:26.999968 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f2dea562-c013-475a-86c5-49966145a250 None None] Validating token access rules against request {{(pid=81409) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:10:27.032294 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8488c53b-a9de-474d-be48-1e7ec7ebe848 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81409) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:10:27.032881 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8488c53b-a9de-474d-be48-1e7ec7ebe848 None admin] REQUEST_METHOD: `GET` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:10:27.033156 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8488c53b-a9de-474d-be48-1e7ec7ebe848 None admin] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:10:27.033421 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8488c53b-a9de-474d-be48-1e7ec7ebe848 None admin] PATH_INFO: `/v3/roles` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:10:27.034067 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8488c53b-a9de-474d-be48-1e7ec7ebe848 None admin] RBAC: Adding query filter params (name=admin) {{(pid=81409) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 13 04:10:27.034390 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8488c53b-a9de-474d-be48-1e7ec7ebe848 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:10:27.035741 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8488c53b-a9de-474d-be48-1e7ec7ebe848 None admin] RBAC: Authorization granted {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:10:27.044206 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 458/894] 10.222.0.27 () {56 vars in 1099 bytes} [Thu Oct 13 04:10:26 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 143 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 13 04:10:27.050059 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8255acd6-0382-4fc7-b403-d94a0af466c8 None admin] Authenticating user token {{(pid=81408) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:10:27.148963 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8255acd6-0382-4fc7-b403-d94a0af466c8 None admin] Validating token access rules against request {{(pid=81408) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:10:27.181774 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-96f638e5-8e4c-4118-af33-fe631220f4c4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81408) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:10:27.182410 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96f638e5-8e4c-4118-af33-fe631220f4c4 None admin] REQUEST_METHOD: `GET` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:10:27.182689 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96f638e5-8e4c-4118-af33-fe631220f4c4 None admin] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:10:27.182961 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96f638e5-8e4c-4118-af33-fe631220f4c4 None admin] PATH_INFO: `/v3/users/e971aca6124641e78f3774d7d558b0d6` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:10:27.184802 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG dbcounter [-] [81408] Writing DB stats keystone:SELECT=100 {{(pid=81408) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:10:27.196712 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-96f638e5-8e4c-4118-af33-fe631220f4c4 None admin] RBAC: Authorizing `identity:get_user(user_id=e971aca6124641e78f3774d7d558b0d6)` {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:10:27.198159 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-96f638e5-8e4c-4118-af33-fe631220f4c4 None admin] RBAC: Authorization granted {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:10:27.199770 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 437/895] 10.222.0.27 () {56 vars in 1177 bytes} [Thu Oct 13 04:10:27 2022] GET /identity/v3/users/e971aca6124641e78f3774d7d558b0d6 => generated 311 bytes in 150 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 13 04:10:27.205597 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8488c53b-a9de-474d-be48-1e7ec7ebe848 None admin] Authenticating user token {{(pid=81409) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:10:27.292814 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG dbcounter [-] [81409] Writing DB stats keystone:SELECT=100 {{(pid=81409) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:10:27.304612 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8488c53b-a9de-474d-be48-1e7ec7ebe848 None admin] Validating token access rules against request {{(pid=81409) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:10:27.336738 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fd1640fd-e53f-481c-af35-559f44e3952d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81409) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:10:27.337348 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd1640fd-e53f-481c-af35-559f44e3952d None admin] REQUEST_METHOD: `GET` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:10:27.337620 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd1640fd-e53f-481c-af35-559f44e3952d None admin] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:10:27.337885 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd1640fd-e53f-481c-af35-559f44e3952d None admin] PATH_INFO: `/v3/projects/b5d025396dfc4e2384b6eb960d1d9cb8` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:10:27.339021 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fd1640fd-e53f-481c-af35-559f44e3952d None admin] RBAC: Authorizing `identity:get_project(project_id=b5d025396dfc4e2384b6eb960d1d9cb8)` {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:10:27.340424 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fd1640fd-e53f-481c-af35-559f44e3952d None admin] RBAC: Authorization granted {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:10:27.342020 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 459/896] 10.222.0.27 () {56 vars in 1186 bytes} [Thu Oct 13 04:10:27 2022] GET /identity/v3/projects/b5d025396dfc4e2384b6eb960d1d9cb8 => generated 361 bytes in 137 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 13 04:10:27.347968 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-96f638e5-8e4c-4118-af33-fe631220f4c4 None admin] Authenticating user token {{(pid=81408) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:10:27.446329 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-96f638e5-8e4c-4118-af33-fe631220f4c4 None admin] Validating token access rules against request {{(pid=81408) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:10:27.478408 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9055e518-8a0f-42ef-9cfd-45d36c092bc0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81408) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:10:27.479044 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9055e518-8a0f-42ef-9cfd-45d36c092bc0 None admin] REQUEST_METHOD: `PUT` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:10:27.479316 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9055e518-8a0f-42ef-9cfd-45d36c092bc0 None admin] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:10:27.479585 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9055e518-8a0f-42ef-9cfd-45d36c092bc0 None admin] PATH_INFO: `/v3/projects/b5d025396dfc4e2384b6eb960d1d9cb8/users/e971aca6124641e78f3774d7d558b0d6/roles/ef55fbd94e1040be83695754075a7ed5` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:10:27.481426 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9055e518-8a0f-42ef-9cfd-45d36c092bc0 None admin] RBAC: Authorizing `identity:create_grant(project_id=b5d025396dfc4e2384b6eb960d1d9cb8, user_id=e971aca6124641e78f3774d7d558b0d6, role_id=ef55fbd94e1040be83695754075a7ed5)` {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:10:27.482878 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9055e518-8a0f-42ef-9cfd-45d36c092bc0 None admin] RBAC: Authorization granted {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:10:27.496315 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 438/897] 10.222.0.27 () {58 vars in 1439 bytes} [Thu Oct 13 04:10:27 2022] PUT /identity/v3/projects/b5d025396dfc4e2384b6eb960d1d9cb8/users/e971aca6124641e78f3774d7d558b0d6/roles/ef55fbd94e1040be83695754075a7ed5 => generated 0 bytes in 149 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 13 04:10:29.606790 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f91e1a9-80a5-4867-aa00-512317753210 None None] REQUEST_METHOD: `GET` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:10:29.607331 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f91e1a9-80a5-4867-aa00-512317753210 None None] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:10:29.607331 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f91e1a9-80a5-4867-aa00-512317753210 None None] PATH_INFO: `/` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:10:29.608656 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 460/898] 10.222.0.27 () {54 vars in 927 bytes} [Thu Oct 13 04:10:29 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 13 04:10:29.613940 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-500360f1-65f8-4218-a074-c2196020e015 None None] REQUEST_METHOD: `POST` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:10:29.614382 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-500360f1-65f8-4218-a074-c2196020e015 None None] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:10:29.614503 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-500360f1-65f8-4218-a074-c2196020e015 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:10:29.633811 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.auth.core [None req-500360f1-65f8-4218-a074-c2196020e015 None None] MFA Rules not processed for user `0e2d979563d6421995451a0e86bf3022`. Rule list: `[]` (Enabled: `True`). {{(pid=81408) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 13 04:10:29.635019 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.fernet_utils [None req-500360f1-65f8-4218-a074-c2196020e015 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81408) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:10:29.682158 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG dbcounter [-] [81408] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81408) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:10:29.702240 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 439/899] 10.222.0.27 () {58 vars in 1024 bytes} [Thu Oct 13 04:10:29 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 89 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Oct 13 04:10:29.974692 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3f91e1a9-80a5-4867-aa00-512317753210 None None] Authenticating user token {{(pid=81409) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:10:30.075053 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3f91e1a9-80a5-4867-aa00-512317753210 None None] Validating token access rules against request {{(pid=81409) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:10:30.108030 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bf5acaac-67e8-4111-97cb-cab608cd1cb8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81409) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:10:30.108641 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf5acaac-67e8-4111-97cb-cab608cd1cb8 None admin] REQUEST_METHOD: `GET` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:10:30.108916 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf5acaac-67e8-4111-97cb-cab608cd1cb8 None admin] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:10:30.109185 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf5acaac-67e8-4111-97cb-cab608cd1cb8 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:10:30.119816 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bf5acaac-67e8-4111-97cb-cab608cd1cb8 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 13 04:10:30.124575 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application [None req-bf5acaac-67e8-4111-97cb-cab608cd1cb8 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Oct 13 04:10:30.124575 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 13 04:10:30.124575 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 13 04:10:30.124575 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 13 04:10:30.124575 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 13 04:10:30.124575 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 13 04:10:30.124575 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 13 04:10:30.124575 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 13 04:10:30.124575 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 13 04:10:30.124575 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 13 04:10:30.124575 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 13 04:10:30.124575 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 13 04:10:30.124575 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 13 04:10:30.124575 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 13 04:10:30.124575 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 13 04:10:30.124575 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 13 04:10:30.125625 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 13 04:10:30.125625 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 13 04:10:30.125625 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 13 04:10:30.125625 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 13 04:10:30.125625 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 13 04:10:30.125625 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 13 04:10:30.125625 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 13 04:10:30.125625 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 13 04:10:30.125625 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 13 04:10:30.125625 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application return self.get_or_create( Oct 13 04:10:30.125625 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 13 04:10:30.125625 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application with Lock( Oct 13 04:10:30.125625 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 13 04:10:30.125625 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application return self._enter() Oct 13 04:10:30.125625 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 13 04:10:30.125625 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 13 04:10:30.126681 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 13 04:10:30.126681 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application return self.creator() Oct 13 04:10:30.126681 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 13 04:10:30.126681 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application created_value = creator( Oct 13 04:10:30.126681 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 13 04:10:30.126681 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 13 04:10:30.126681 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 13 04:10:30.126681 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 13 04:10:30.126681 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 13 04:10:30.126681 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 13 04:10:30.126681 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Oct 13 04:10:30.126681 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application Oct 13 04:10:30.126681 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 461/900] 10.222.0.27 () {56 vars in 1096 bytes} [Thu Oct 13 04:10:29 2022] GET /identity/v3/roles/admin => generated 83 bytes in 152 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 13 04:10:30.131940 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-500360f1-65f8-4218-a074-c2196020e015 None None] Authenticating user token {{(pid=81408) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:10:30.227394 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-500360f1-65f8-4218-a074-c2196020e015 None None] Validating token access rules against request {{(pid=81408) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:10:30.259287 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5ca288c7-3370-4f3a-8a20-a3633e53f3bd None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81408) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:10:30.259875 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ca288c7-3370-4f3a-8a20-a3633e53f3bd None admin] REQUEST_METHOD: `GET` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:10:30.260149 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ca288c7-3370-4f3a-8a20-a3633e53f3bd None admin] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:10:30.260418 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ca288c7-3370-4f3a-8a20-a3633e53f3bd None admin] PATH_INFO: `/v3/roles` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:10:30.261073 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5ca288c7-3370-4f3a-8a20-a3633e53f3bd None admin] RBAC: Adding query filter params (name=admin) {{(pid=81408) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 13 04:10:30.261372 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5ca288c7-3370-4f3a-8a20-a3633e53f3bd None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:10:30.262769 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5ca288c7-3370-4f3a-8a20-a3633e53f3bd None admin] RBAC: Authorization granted {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:10:30.270433 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 440/901] 10.222.0.27 () {56 vars in 1099 bytes} [Thu Oct 13 04:10:30 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 139 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 13 04:10:30.276858 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bf5acaac-67e8-4111-97cb-cab608cd1cb8 None admin] Authenticating user token {{(pid=81409) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:10:30.295465 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG dbcounter [-] [81409] Writing DB stats keystone:SELECT=100 {{(pid=81409) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:10:30.374789 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bf5acaac-67e8-4111-97cb-cab608cd1cb8 None admin] Validating token access rules against request {{(pid=81409) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:10:30.407081 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7991b46d-0f97-4333-a000-c59f5b7ff3b7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81409) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:10:30.407695 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7991b46d-0f97-4333-a000-c59f5b7ff3b7 None admin] REQUEST_METHOD: `GET` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:10:30.407970 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7991b46d-0f97-4333-a000-c59f5b7ff3b7 None admin] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:10:30.408239 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7991b46d-0f97-4333-a000-c59f5b7ff3b7 None admin] PATH_INFO: `/v3/users/e971aca6124641e78f3774d7d558b0d6` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:10:30.409435 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7991b46d-0f97-4333-a000-c59f5b7ff3b7 None admin] RBAC: Authorizing `identity:get_user(user_id=e971aca6124641e78f3774d7d558b0d6)` {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:10:30.411047 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7991b46d-0f97-4333-a000-c59f5b7ff3b7 None admin] RBAC: Authorization granted {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:10:30.412686 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 462/902] 10.222.0.27 () {56 vars in 1177 bytes} [Thu Oct 13 04:10:30 2022] GET /identity/v3/users/e971aca6124641e78f3774d7d558b0d6 => generated 311 bytes in 136 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 13 04:10:30.418707 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5ca288c7-3370-4f3a-8a20-a3633e53f3bd None admin] Authenticating user token {{(pid=81408) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:10:30.464548 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG dbcounter [-] [81408] Writing DB stats keystone:SELECT=100 {{(pid=81408) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:10:30.517474 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5ca288c7-3370-4f3a-8a20-a3633e53f3bd None admin] Validating token access rules against request {{(pid=81408) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:10:30.549320 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ecbf7f84-ed1b-4e4c-906d-2d50baf24812 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81408) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:10:30.549905 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ecbf7f84-ed1b-4e4c-906d-2d50baf24812 None admin] REQUEST_METHOD: `GET` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:10:30.550183 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ecbf7f84-ed1b-4e4c-906d-2d50baf24812 None admin] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:10:30.550478 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ecbf7f84-ed1b-4e4c-906d-2d50baf24812 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:10:30.614742 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ecbf7f84-ed1b-4e4c-906d-2d50baf24812 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:10:30.616181 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ecbf7f84-ed1b-4e4c-906d-2d50baf24812 None admin] RBAC: Authorization granted {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:10:30.659503 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG dbcounter [-] [81408] Writing DB stats keystone:SELECT=100 {{(pid=81408) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:10:30.711851 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 441/903] 10.222.0.27 () {58 vars in 1282 bytes} [Thu Oct 13 04:10:30 2022] GET /identity/v3/auth/tokens => generated 2118 bytes in 294 msecs (HTTP/1.1 200) 6 headers in 359 bytes (1 switches on core 0) Oct 13 04:10:30.717825 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7991b46d-0f97-4333-a000-c59f5b7ff3b7 None admin] Authenticating user token {{(pid=81409) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:10:30.798961 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG dbcounter [-] [81409] Writing DB stats keystone:SELECT=100 {{(pid=81409) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:10:30.814297 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7991b46d-0f97-4333-a000-c59f5b7ff3b7 None admin] Validating token access rules against request {{(pid=81409) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:10:30.846656 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8567784d-93da-4722-80a5-c68dcc77298d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81409) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:10:30.847259 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8567784d-93da-4722-80a5-c68dcc77298d None admin] REQUEST_METHOD: `GET` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:10:30.847529 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8567784d-93da-4722-80a5-c68dcc77298d None admin] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:10:30.847795 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8567784d-93da-4722-80a5-c68dcc77298d None admin] PATH_INFO: `/v3/projects/b5d025396dfc4e2384b6eb960d1d9cb8` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:10:30.848879 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8567784d-93da-4722-80a5-c68dcc77298d None admin] RBAC: Authorizing `identity:get_project(project_id=b5d025396dfc4e2384b6eb960d1d9cb8)` {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:10:30.850300 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8567784d-93da-4722-80a5-c68dcc77298d None admin] RBAC: Authorization granted {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:10:30.852335 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 463/904] 10.222.0.27 () {56 vars in 1186 bytes} [Thu Oct 13 04:10:30 2022] GET /identity/v3/projects/b5d025396dfc4e2384b6eb960d1d9cb8 => generated 361 bytes in 135 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 13 04:10:30.858415 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ecbf7f84-ed1b-4e4c-906d-2d50baf24812 None admin] Authenticating user token {{(pid=81408) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:10:30.962791 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ecbf7f84-ed1b-4e4c-906d-2d50baf24812 None admin] Validating token access rules against request {{(pid=81408) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:10:30.995743 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1bf35a15-ae12-487d-a0b4-36ea0cf50dae None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81408) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:10:30.996328 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1bf35a15-ae12-487d-a0b4-36ea0cf50dae None admin] REQUEST_METHOD: `GET` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:10:30.996603 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1bf35a15-ae12-487d-a0b4-36ea0cf50dae None admin] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:10:30.996870 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1bf35a15-ae12-487d-a0b4-36ea0cf50dae None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:10:30.997578 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1bf35a15-ae12-487d-a0b4-36ea0cf50dae None admin] RBAC: Adding query filter params (role.id=ef55fbd94e1040be83695754075a7ed5, scope.project.id=b5d025396dfc4e2384b6eb960d1d9cb8, user.id=e971aca6124641e78f3774d7d558b0d6) {{(pid=81408) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 13 04:10:30.997881 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1bf35a15-ae12-487d-a0b4-36ea0cf50dae None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:10:30.999333 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1bf35a15-ae12-487d-a0b4-36ea0cf50dae None admin] RBAC: Authorization granted {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:10:31.004862 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 442/905] 10.222.0.27 () {56 vars in 1374 bytes} [Thu Oct 13 04:10:30 2022] GET /identity/v3/role_assignments?user.id=e971aca6124641e78f3774d7d558b0d6&scope.project.id=b5d025396dfc4e2384b6eb960d1d9cb8&role.id=ef55fbd94e1040be83695754075a7ed5 => generated 608 bytes in 147 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 13 04:10:40.842643 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG dbcounter [-] [81409] Writing DB stats keystone:SELECT=24 {{(pid=81409) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:10:41.002375 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG dbcounter [-] [81408] Writing DB stats keystone:SELECT=96 {{(pid=81408) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:11:26.223482 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f390d66f-f51a-42d0-af5c-ab53df7234af None None] REQUEST_METHOD: `GET` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:11:26.224590 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f390d66f-f51a-42d0-af5c-ab53df7234af None None] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:11:26.224978 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f390d66f-f51a-42d0-af5c-ab53df7234af None None] PATH_INFO: `/` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:11:26.227048 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 464/906] 10.222.0.27 () {54 vars in 927 bytes} [Thu Oct 13 04:11:26 2022] GET /identity => generated 270 bytes in 5 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 13 04:11:26.233589 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a85d7f67-10c7-4303-9b19-dbfbf7697623 None None] REQUEST_METHOD: `POST` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:11:26.234375 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a85d7f67-10c7-4303-9b19-dbfbf7697623 None None] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:11:26.234375 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a85d7f67-10c7-4303-9b19-dbfbf7697623 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:11:26.256659 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.auth.core [None req-a85d7f67-10c7-4303-9b19-dbfbf7697623 None None] MFA Rules not processed for user `0e2d979563d6421995451a0e86bf3022`. Rule list: `[]` (Enabled: `True`). {{(pid=81408) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 13 04:11:26.258272 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.fernet_utils [None req-a85d7f67-10c7-4303-9b19-dbfbf7697623 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81408) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:11:26.346285 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 443/907] 10.222.0.27 () {58 vars in 1024 bytes} [Thu Oct 13 04:11:26 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 113 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Oct 13 04:11:26.636770 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f390d66f-f51a-42d0-af5c-ab53df7234af None None] Authenticating user token {{(pid=81409) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:11:26.779512 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f390d66f-f51a-42d0-af5c-ab53df7234af None None] Validating token access rules against request {{(pid=81409) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:11:26.824862 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-849c222d-30c1-49bb-93c7-d8779c9facfe None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81409) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:11:26.825503 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-849c222d-30c1-49bb-93c7-d8779c9facfe None admin] REQUEST_METHOD: `GET` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:11:26.825781 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-849c222d-30c1-49bb-93c7-d8779c9facfe None admin] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:11:26.826052 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-849c222d-30c1-49bb-93c7-d8779c9facfe None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:11:26.844230 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-849c222d-30c1-49bb-93c7-d8779c9facfe None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:11:26.845686 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-849c222d-30c1-49bb-93c7-d8779c9facfe None admin] RBAC: Authorization granted {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:11:26.866989 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application [None req-849c222d-30c1-49bb-93c7-d8779c9facfe None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 13 04:11:26.866989 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 13 04:11:26.866989 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 13 04:11:26.866989 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 13 04:11:26.866989 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 13 04:11:26.866989 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 13 04:11:26.866989 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 13 04:11:26.866989 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 13 04:11:26.866989 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 13 04:11:26.866989 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 13 04:11:26.866989 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 13 04:11:26.866989 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 13 04:11:26.866989 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 13 04:11:26.866989 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 13 04:11:26.866989 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 13 04:11:26.866989 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 13 04:11:26.868121 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 13 04:11:26.868121 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 13 04:11:26.868121 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 13 04:11:26.868121 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 13 04:11:26.868121 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 13 04:11:26.868121 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application return self.get_or_create( Oct 13 04:11:26.868121 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 13 04:11:26.868121 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application with Lock( Oct 13 04:11:26.868121 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 13 04:11:26.868121 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application return self._enter() Oct 13 04:11:26.868121 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 13 04:11:26.868121 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 13 04:11:26.868121 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 13 04:11:26.868121 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application return self.creator() Oct 13 04:11:26.868121 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 13 04:11:26.868121 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application created_value = creator( Oct 13 04:11:26.868121 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 13 04:11:26.871449 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 13 04:11:26.871449 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 13 04:11:26.871449 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application Oct 13 04:11:26.871449 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 465/908] 10.222.0.27 () {56 vars in 1108 bytes} [Thu Oct 13 04:11:26 2022] GET /identity/v3/domains/Default => generated 87 bytes in 232 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 13 04:11:26.875184 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a85d7f67-10c7-4303-9b19-dbfbf7697623 None None] Authenticating user token {{(pid=81408) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:11:26.994867 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a85d7f67-10c7-4303-9b19-dbfbf7697623 None None] Validating token access rules against request {{(pid=81408) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:11:27.035404 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG dbcounter [-] [81408] Writing DB stats keystone:SELECT=100 {{(pid=81408) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:11:27.044009 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d5fe9edb-e23b-4e66-87b5-53dace32df18 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81408) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:11:27.044891 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5fe9edb-e23b-4e66-87b5-53dace32df18 None admin] REQUEST_METHOD: `GET` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:11:27.045340 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5fe9edb-e23b-4e66-87b5-53dace32df18 None admin] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:11:27.045772 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5fe9edb-e23b-4e66-87b5-53dace32df18 None admin] PATH_INFO: `/v3/domains` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:11:27.046615 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d5fe9edb-e23b-4e66-87b5-53dace32df18 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81408) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 13 04:11:27.047099 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d5fe9edb-e23b-4e66-87b5-53dace32df18 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:11:27.048810 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d5fe9edb-e23b-4e66-87b5-53dace32df18 None admin] RBAC: Authorization granted {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:11:27.071341 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 444/909] 10.222.0.27 () {56 vars in 1109 bytes} [Thu Oct 13 04:11:26 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 196 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 13 04:11:27.072512 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-849c222d-30c1-49bb-93c7-d8779c9facfe None admin] Authenticating user token {{(pid=81409) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:11:27.132066 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG dbcounter [-] [81409] Writing DB stats keystone:SELECT=100 {{(pid=81409) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:11:27.198532 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-849c222d-30c1-49bb-93c7-d8779c9facfe None admin] Validating token access rules against request {{(pid=81409) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:11:27.236741 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0414676f-961d-4421-8929-3d15bb718662 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81409) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:11:27.238039 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0414676f-961d-4421-8929-3d15bb718662 None admin] REQUEST_METHOD: `POST` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:11:27.241604 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0414676f-961d-4421-8929-3d15bb718662 None admin] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:11:27.242073 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0414676f-961d-4421-8929-3d15bb718662 None admin] PATH_INFO: `/v3/users` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:11:27.242940 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0414676f-961d-4421-8929-3d15bb718662 None admin] RBAC: Authorizing `identity:create_user()` {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:11:27.244996 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0414676f-961d-4421-8929-3d15bb718662 None admin] RBAC: Authorization granted {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:11:27.273736 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 466/910] 10.222.0.27 () {60 vars in 1132 bytes} [Thu Oct 13 04:11:27 2022] POST /identity/v3/users => generated 251 bytes in 202 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 13 04:11:29.594113 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b852bf7b-c641-4c63-859e-ba941d428c28 None None] REQUEST_METHOD: `GET` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:11:29.594842 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b852bf7b-c641-4c63-859e-ba941d428c28 None None] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:11:29.594842 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b852bf7b-c641-4c63-859e-ba941d428c28 None None] PATH_INFO: `/` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:11:29.596385 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 445/911] 10.222.0.27 () {54 vars in 927 bytes} [Thu Oct 13 04:11:29 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 13 04:11:29.602054 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44b7f4b9-4635-4e8f-b4c7-764431d754e0 None None] REQUEST_METHOD: `POST` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:11:29.602830 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44b7f4b9-4635-4e8f-b4c7-764431d754e0 None None] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:11:29.602944 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44b7f4b9-4635-4e8f-b4c7-764431d754e0 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:11:29.623715 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.auth.core [None req-44b7f4b9-4635-4e8f-b4c7-764431d754e0 None None] MFA Rules not processed for user `0e2d979563d6421995451a0e86bf3022`. Rule list: `[]` (Enabled: `True`). {{(pid=81409) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 13 04:11:29.624934 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.fernet_utils [None req-44b7f4b9-4635-4e8f-b4c7-764431d754e0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81409) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:11:29.704683 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 467/912] 10.222.0.27 () {58 vars in 1024 bytes} [Thu Oct 13 04:11:29 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 104 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Oct 13 04:11:29.990628 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b852bf7b-c641-4c63-859e-ba941d428c28 None None] Authenticating user token {{(pid=81408) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:11:30.130959 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b852bf7b-c641-4c63-859e-ba941d428c28 None None] Validating token access rules against request {{(pid=81408) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:11:30.173095 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8e746836-dc11-4d44-9890-3f657c1761a5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81408) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:11:30.173749 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e746836-dc11-4d44-9890-3f657c1761a5 None admin] REQUEST_METHOD: `GET` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:11:30.174028 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e746836-dc11-4d44-9890-3f657c1761a5 None admin] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:11:30.174332 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e746836-dc11-4d44-9890-3f657c1761a5 None admin] PATH_INFO: `/v3/roles/service` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:11:30.187188 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8e746836-dc11-4d44-9890-3f657c1761a5 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 13 04:11:30.192461 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application [None req-8e746836-dc11-4d44-9890-3f657c1761a5 None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Oct 13 04:11:30.192461 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 13 04:11:30.192461 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 13 04:11:30.192461 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 13 04:11:30.192461 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 13 04:11:30.192461 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 13 04:11:30.192461 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 13 04:11:30.192461 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 13 04:11:30.192461 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 13 04:11:30.192461 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 13 04:11:30.192461 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 13 04:11:30.192461 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 13 04:11:30.192461 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 13 04:11:30.192461 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 13 04:11:30.192461 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 13 04:11:30.192461 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 13 04:11:30.193975 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 13 04:11:30.193975 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 13 04:11:30.193975 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 13 04:11:30.193975 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 13 04:11:30.193975 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 13 04:11:30.193975 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 13 04:11:30.193975 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 13 04:11:30.193975 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 13 04:11:30.193975 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 13 04:11:30.193975 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application return self.get_or_create( Oct 13 04:11:30.193975 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 13 04:11:30.193975 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application with Lock( Oct 13 04:11:30.193975 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 13 04:11:30.193975 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application return self._enter() Oct 13 04:11:30.193975 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 13 04:11:30.193975 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 13 04:11:30.194848 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 13 04:11:30.194848 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application return self.creator() Oct 13 04:11:30.194848 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 13 04:11:30.194848 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application created_value = creator( Oct 13 04:11:30.194848 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 13 04:11:30.194848 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 13 04:11:30.194848 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 13 04:11:30.194848 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 13 04:11:30.194848 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 13 04:11:30.194848 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 13 04:11:30.194848 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Oct 13 04:11:30.194848 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application Oct 13 04:11:30.199145 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 446/913] 10.222.0.27 () {56 vars in 1102 bytes} [Thu Oct 13 04:11:29 2022] GET /identity/v3/roles/service => generated 85 bytes in 209 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 13 04:11:30.202818 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-44b7f4b9-4635-4e8f-b4c7-764431d754e0 None None] Authenticating user token {{(pid=81409) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:11:30.246971 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG dbcounter [-] [81409] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81409) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:11:30.320402 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-44b7f4b9-4635-4e8f-b4c7-764431d754e0 None None] Validating token access rules against request {{(pid=81409) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:11:30.361606 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fd5efc53-5de2-4be0-92fe-0056660d9e83 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81409) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:11:30.362334 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd5efc53-5de2-4be0-92fe-0056660d9e83 None admin] REQUEST_METHOD: `GET` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:11:30.362603 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd5efc53-5de2-4be0-92fe-0056660d9e83 None admin] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:11:30.362930 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd5efc53-5de2-4be0-92fe-0056660d9e83 None admin] PATH_INFO: `/v3/roles` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:11:30.363745 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fd5efc53-5de2-4be0-92fe-0056660d9e83 None admin] RBAC: Adding query filter params (name=service) {{(pid=81409) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 13 04:11:30.364297 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fd5efc53-5de2-4be0-92fe-0056660d9e83 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:11:30.366541 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fd5efc53-5de2-4be0-92fe-0056660d9e83 None admin] RBAC: Authorization granted {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:11:30.376695 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 468/914] 10.222.0.27 () {56 vars in 1103 bytes} [Thu Oct 13 04:11:30 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 176 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 13 04:11:30.382938 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8e746836-dc11-4d44-9890-3f657c1761a5 None admin] Authenticating user token {{(pid=81408) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:11:30.444458 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG dbcounter [-] [81408] Writing DB stats keystone:SELECT=100 {{(pid=81408) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:11:30.502293 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8e746836-dc11-4d44-9890-3f657c1761a5 None admin] Validating token access rules against request {{(pid=81408) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:11:30.542187 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7e262533-a17e-4046-8e76-d0f44c3b61eb None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81408) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:11:30.542869 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e262533-a17e-4046-8e76-d0f44c3b61eb None admin] REQUEST_METHOD: `GET` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:11:30.543166 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e262533-a17e-4046-8e76-d0f44c3b61eb None admin] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:11:30.543474 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e262533-a17e-4046-8e76-d0f44c3b61eb None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:11:30.554853 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7e262533-a17e-4046-8e76-d0f44c3b61eb None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:11:30.556322 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7e262533-a17e-4046-8e76-d0f44c3b61eb None admin] RBAC: Authorization granted {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:11:30.571998 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application [None req-7e262533-a17e-4046-8e76-d0f44c3b61eb None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 13 04:11:30.571998 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 13 04:11:30.571998 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 13 04:11:30.571998 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 13 04:11:30.571998 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 13 04:11:30.571998 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 13 04:11:30.571998 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 13 04:11:30.571998 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 13 04:11:30.571998 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 13 04:11:30.571998 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 13 04:11:30.571998 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 13 04:11:30.571998 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 13 04:11:30.571998 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 13 04:11:30.571998 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 13 04:11:30.571998 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 13 04:11:30.571998 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 13 04:11:30.573165 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 13 04:11:30.573165 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 13 04:11:30.573165 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 13 04:11:30.573165 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 13 04:11:30.573165 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 13 04:11:30.573165 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application return self.get_or_create( Oct 13 04:11:30.573165 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 13 04:11:30.573165 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application with Lock( Oct 13 04:11:30.573165 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 13 04:11:30.573165 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application return self._enter() Oct 13 04:11:30.573165 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 13 04:11:30.573165 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 13 04:11:30.573165 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 13 04:11:30.573165 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application return self.creator() Oct 13 04:11:30.573165 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 13 04:11:30.573165 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application created_value = creator( Oct 13 04:11:30.573165 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 13 04:11:30.574045 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 13 04:11:30.574045 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 13 04:11:30.574045 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application Oct 13 04:11:30.578077 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 447/915] 10.222.0.27 () {56 vars in 1108 bytes} [Thu Oct 13 04:11:30 2022] GET /identity/v3/domains/Default => generated 87 bytes in 195 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Oct 13 04:11:30.579313 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fd5efc53-5de2-4be0-92fe-0056660d9e83 None admin] Authenticating user token {{(pid=81409) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:11:30.704139 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG dbcounter [-] [81409] Writing DB stats keystone:SELECT=100 {{(pid=81409) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:11:30.708147 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fd5efc53-5de2-4be0-92fe-0056660d9e83 None admin] Validating token access rules against request {{(pid=81409) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:11:30.749242 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-40b860f9-e14a-4493-afaa-6947bfb9902e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81409) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:11:30.750126 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40b860f9-e14a-4493-afaa-6947bfb9902e None admin] REQUEST_METHOD: `GET` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:11:30.750499 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40b860f9-e14a-4493-afaa-6947bfb9902e None admin] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:11:30.750782 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40b860f9-e14a-4493-afaa-6947bfb9902e None admin] PATH_INFO: `/v3/domains` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:11:30.751609 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-40b860f9-e14a-4493-afaa-6947bfb9902e None admin] RBAC: Adding query filter params (name=Default) {{(pid=81409) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 13 04:11:30.752025 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-40b860f9-e14a-4493-afaa-6947bfb9902e None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:11:30.753752 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-40b860f9-e14a-4493-afaa-6947bfb9902e None admin] RBAC: Authorization granted {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:11:30.767620 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 469/916] 10.222.0.27 () {56 vars in 1109 bytes} [Thu Oct 13 04:11:30 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 189 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 13 04:11:30.775374 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7e262533-a17e-4046-8e76-d0f44c3b61eb None admin] Authenticating user token {{(pid=81408) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:11:30.897782 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7e262533-a17e-4046-8e76-d0f44c3b61eb None admin] Validating token access rules against request {{(pid=81408) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:11:30.899416 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG dbcounter [-] [81408] Writing DB stats keystone:SELECT=100 {{(pid=81408) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:11:30.938865 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d3754925-bae3-4b91-8350-a74a13a3e6a7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81408) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:11:30.939536 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3754925-bae3-4b91-8350-a74a13a3e6a7 None admin] REQUEST_METHOD: `GET` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:11:30.939870 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3754925-bae3-4b91-8350-a74a13a3e6a7 None admin] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:11:30.940267 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3754925-bae3-4b91-8350-a74a13a3e6a7 None admin] PATH_INFO: `/v3/users/placement` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:11:30.950481 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d3754925-bae3-4b91-8350-a74a13a3e6a7 None admin] RBAC: Authorizing `identity:get_user(user_id=placement)` {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:11:30.952402 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d3754925-bae3-4b91-8350-a74a13a3e6a7 None admin] RBAC: Authorization granted {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:11:30.967174 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application [None req-d3754925-bae3-4b91-8350-a74a13a3e6a7 None admin] Could not find user: placement.: keystone.exception.UserNotFound: Could not find user: placement. Oct 13 04:11:30.967174 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 13 04:11:30.967174 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 13 04:11:30.967174 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 13 04:11:30.967174 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 13 04:11:30.967174 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 13 04:11:30.967174 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 13 04:11:30.967174 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 13 04:11:30.967174 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 13 04:11:30.967174 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 13 04:11:30.967174 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 13 04:11:30.967174 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 13 04:11:30.967174 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Oct 13 04:11:30.967174 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application return self._get_user(user_id) Oct 13 04:11:30.967174 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Oct 13 04:11:30.967174 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Oct 13 04:11:30.968376 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 13 04:11:30.968376 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 13 04:11:30.968376 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Oct 13 04:11:30.968376 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 13 04:11:30.968376 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Oct 13 04:11:30.968376 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 13 04:11:30.968376 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 13 04:11:30.968376 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 13 04:11:30.968376 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 13 04:11:30.968376 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application return self.get_or_create( Oct 13 04:11:30.968376 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 13 04:11:30.968376 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application with Lock( Oct 13 04:11:30.968376 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 13 04:11:30.968376 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application return self._enter() Oct 13 04:11:30.968376 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 13 04:11:30.968376 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 13 04:11:30.968376 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 13 04:11:30.968376 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application return self.creator() Oct 13 04:11:30.969335 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 13 04:11:30.969335 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application created_value = creator( Oct 13 04:11:30.969335 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Oct 13 04:11:30.969335 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Oct 13 04:11:30.969335 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Oct 13 04:11:30.969335 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Oct 13 04:11:30.969335 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Oct 13 04:11:30.969335 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Oct 13 04:11:30.969335 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: placement. Oct 13 04:11:30.969335 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application Oct 13 04:11:30.969335 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 448/917] 10.222.0.27 () {56 vars in 1108 bytes} [Thu Oct 13 04:11:30 2022] GET /identity/v3/users/placement => generated 87 bytes in 193 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 13 04:11:30.975338 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-40b860f9-e14a-4493-afaa-6947bfb9902e None admin] Authenticating user token {{(pid=81409) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:11:31.096244 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-40b860f9-e14a-4493-afaa-6947bfb9902e None admin] Validating token access rules against request {{(pid=81409) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:11:31.134433 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-76c30ad9-a642-4a2e-a3d1-4019e78950f8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81409) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:11:31.135096 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76c30ad9-a642-4a2e-a3d1-4019e78950f8 None admin] REQUEST_METHOD: `GET` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:11:31.135381 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76c30ad9-a642-4a2e-a3d1-4019e78950f8 None admin] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:11:31.135653 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76c30ad9-a642-4a2e-a3d1-4019e78950f8 None admin] PATH_INFO: `/v3/users` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:11:31.136376 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-76c30ad9-a642-4a2e-a3d1-4019e78950f8 None admin] RBAC: Adding query filter params (domain_id=default, name=placement) {{(pid=81409) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 13 04:11:31.136683 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-76c30ad9-a642-4a2e-a3d1-4019e78950f8 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:11:31.138100 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-76c30ad9-a642-4a2e-a3d1-4019e78950f8 None admin] RBAC: Authorization granted {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:11:31.164047 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 470/918] 10.222.0.27 () {56 vars in 1143 bytes} [Thu Oct 13 04:11:30 2022] GET /identity/v3/users?domain_id=default&name=placement => generated 378 bytes in 185 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 13 04:11:31.165842 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d3754925-bae3-4b91-8350-a74a13a3e6a7 None admin] Authenticating user token {{(pid=81408) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:11:31.299609 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d3754925-bae3-4b91-8350-a74a13a3e6a7 None admin] Validating token access rules against request {{(pid=81408) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:11:31.338293 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4846d9f4-2db8-4852-8f0c-0dcb92ae1ed6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81408) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:11:31.338929 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4846d9f4-2db8-4852-8f0c-0dcb92ae1ed6 None admin] REQUEST_METHOD: `GET` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:11:31.339204 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4846d9f4-2db8-4852-8f0c-0dcb92ae1ed6 None admin] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:11:31.339453 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4846d9f4-2db8-4852-8f0c-0dcb92ae1ed6 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:11:31.351659 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4846d9f4-2db8-4852-8f0c-0dcb92ae1ed6 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:11:31.353320 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4846d9f4-2db8-4852-8f0c-0dcb92ae1ed6 None admin] RBAC: Authorization granted {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:11:31.368936 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application [None req-4846d9f4-2db8-4852-8f0c-0dcb92ae1ed6 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 13 04:11:31.368936 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 13 04:11:31.368936 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 13 04:11:31.368936 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 13 04:11:31.368936 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 13 04:11:31.368936 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 13 04:11:31.368936 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 13 04:11:31.368936 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 13 04:11:31.368936 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 13 04:11:31.368936 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 13 04:11:31.368936 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 13 04:11:31.368936 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 13 04:11:31.368936 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 13 04:11:31.368936 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 13 04:11:31.368936 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 13 04:11:31.368936 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 13 04:11:31.369995 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 13 04:11:31.369995 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 13 04:11:31.369995 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 13 04:11:31.369995 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 13 04:11:31.369995 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 13 04:11:31.369995 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application return self.get_or_create( Oct 13 04:11:31.369995 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 13 04:11:31.369995 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application with Lock( Oct 13 04:11:31.369995 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 13 04:11:31.369995 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application return self._enter() Oct 13 04:11:31.369995 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 13 04:11:31.369995 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 13 04:11:31.369995 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 13 04:11:31.369995 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application return self.creator() Oct 13 04:11:31.369995 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 13 04:11:31.369995 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application created_value = creator( Oct 13 04:11:31.369995 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 13 04:11:31.370958 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 13 04:11:31.370958 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 13 04:11:31.370958 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application Oct 13 04:11:31.370958 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 449/919] 10.222.0.27 () {56 vars in 1108 bytes} [Thu Oct 13 04:11:31 2022] GET /identity/v3/domains/Default => generated 87 bytes in 205 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 13 04:11:31.376104 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-76c30ad9-a642-4a2e-a3d1-4019e78950f8 None admin] Authenticating user token {{(pid=81409) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:11:31.395559 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG dbcounter [-] [81409] Writing DB stats keystone:SELECT=100 {{(pid=81409) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:11:31.493699 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-76c30ad9-a642-4a2e-a3d1-4019e78950f8 None admin] Validating token access rules against request {{(pid=81409) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:11:31.531286 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-727263ee-a9ea-480b-8d46-0825ad176775 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81409) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:11:31.532135 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-727263ee-a9ea-480b-8d46-0825ad176775 None admin] REQUEST_METHOD: `GET` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:11:31.532562 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-727263ee-a9ea-480b-8d46-0825ad176775 None admin] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:11:31.532998 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-727263ee-a9ea-480b-8d46-0825ad176775 None admin] PATH_INFO: `/v3/domains` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:11:31.533804 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-727263ee-a9ea-480b-8d46-0825ad176775 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81409) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 13 04:11:31.534347 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-727263ee-a9ea-480b-8d46-0825ad176775 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:11:31.535868 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-727263ee-a9ea-480b-8d46-0825ad176775 None admin] RBAC: Authorization granted {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:11:31.550373 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 471/920] 10.222.0.27 () {56 vars in 1109 bytes} [Thu Oct 13 04:11:31 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 175 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 13 04:11:31.557139 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4846d9f4-2db8-4852-8f0c-0dcb92ae1ed6 None admin] Authenticating user token {{(pid=81408) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:11:31.575596 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG dbcounter [-] [81408] Writing DB stats keystone:SELECT=100 {{(pid=81408) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:11:31.674964 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4846d9f4-2db8-4852-8f0c-0dcb92ae1ed6 None admin] Validating token access rules against request {{(pid=81408) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:11:31.712710 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c70613bd-6617-4ac9-86ca-8cf6af0dab01 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81408) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:11:31.713365 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c70613bd-6617-4ac9-86ca-8cf6af0dab01 None admin] REQUEST_METHOD: `GET` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:11:31.713735 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c70613bd-6617-4ac9-86ca-8cf6af0dab01 None admin] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:11:31.714014 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c70613bd-6617-4ac9-86ca-8cf6af0dab01 None admin] PATH_INFO: `/v3/projects/service` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:11:31.720905 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c70613bd-6617-4ac9-86ca-8cf6af0dab01 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:11:31.722380 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c70613bd-6617-4ac9-86ca-8cf6af0dab01 None admin] RBAC: Authorization granted {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:11:31.734068 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application [None req-c70613bd-6617-4ac9-86ca-8cf6af0dab01 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Oct 13 04:11:31.734068 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 13 04:11:31.734068 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 13 04:11:31.734068 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 13 04:11:31.734068 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 13 04:11:31.734068 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 13 04:11:31.734068 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 13 04:11:31.734068 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 13 04:11:31.734068 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 13 04:11:31.734068 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 13 04:11:31.734068 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 13 04:11:31.734068 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 13 04:11:31.734068 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Oct 13 04:11:31.734068 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application return self._get_project(project_id) Oct 13 04:11:31.734068 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Oct 13 04:11:31.735190 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Oct 13 04:11:31.735190 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 13 04:11:31.735190 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 13 04:11:31.735190 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 13 04:11:31.735190 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 13 04:11:31.735190 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 13 04:11:31.735190 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application return self.get_or_create( Oct 13 04:11:31.735190 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 13 04:11:31.735190 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application with Lock( Oct 13 04:11:31.735190 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 13 04:11:31.735190 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application return self._enter() Oct 13 04:11:31.735190 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 13 04:11:31.735190 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 13 04:11:31.735190 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 13 04:11:31.735190 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application return self.creator() Oct 13 04:11:31.735190 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 13 04:11:31.735190 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application created_value = creator( Oct 13 04:11:31.736086 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Oct 13 04:11:31.736086 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Oct 13 04:11:31.736086 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Oct 13 04:11:31.736086 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Oct 13 04:11:31.736086 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Oct 13 04:11:31.736086 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Oct 13 04:11:31.736086 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Oct 13 04:11:31.736086 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application Oct 13 04:11:31.736086 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 450/921] 10.222.0.27 () {56 vars in 1111 bytes} [Thu Oct 13 04:11:31 2022] GET /identity/v3/projects/service => generated 88 bytes in 179 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 13 04:11:31.741599 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-727263ee-a9ea-480b-8d46-0825ad176775 None admin] Authenticating user token {{(pid=81409) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:11:31.833329 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG dbcounter [-] [81409] Writing DB stats keystone:SELECT=100 {{(pid=81409) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:11:31.860490 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-727263ee-a9ea-480b-8d46-0825ad176775 None admin] Validating token access rules against request {{(pid=81409) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:11:31.901097 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-51b941d7-c7cf-4c6a-9bba-17b88d0196f7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81409) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:11:31.901715 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51b941d7-c7cf-4c6a-9bba-17b88d0196f7 None admin] REQUEST_METHOD: `GET` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:11:31.902116 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51b941d7-c7cf-4c6a-9bba-17b88d0196f7 None admin] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:11:31.902421 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51b941d7-c7cf-4c6a-9bba-17b88d0196f7 None admin] PATH_INFO: `/v3/projects` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:11:31.903124 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-51b941d7-c7cf-4c6a-9bba-17b88d0196f7 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81409) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 13 04:11:31.903424 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-51b941d7-c7cf-4c6a-9bba-17b88d0196f7 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:11:31.905021 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-51b941d7-c7cf-4c6a-9bba-17b88d0196f7 None admin] RBAC: Authorization granted {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:11:31.919960 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 472/922] 10.222.0.27 () {56 vars in 1148 bytes} [Thu Oct 13 04:11:31 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 179 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 13 04:11:31.926259 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c70613bd-6617-4ac9-86ca-8cf6af0dab01 None admin] Authenticating user token {{(pid=81408) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:11:32.019079 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG dbcounter [-] [81408] Writing DB stats keystone:SELECT=100 {{(pid=81408) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:11:32.052752 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c70613bd-6617-4ac9-86ca-8cf6af0dab01 None admin] Validating token access rules against request {{(pid=81408) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:11:32.095272 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-04014989-aabb-4282-a335-fc221ed08f58 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81408) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:11:32.095974 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04014989-aabb-4282-a335-fc221ed08f58 None admin] REQUEST_METHOD: `PUT` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:11:32.096311 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04014989-aabb-4282-a335-fc221ed08f58 None admin] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:11:32.096610 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04014989-aabb-4282-a335-fc221ed08f58 None admin] PATH_INFO: `/v3/projects/dcec0431fb6849499d6aec35a234daa6/users/ba1b6e5868a549a99760ebe6068fd6b9/roles/4cfff1d3d68e49c9b04231a19cf76ede` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:11:32.114307 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-04014989-aabb-4282-a335-fc221ed08f58 None admin] RBAC: Authorizing `identity:create_grant(project_id=dcec0431fb6849499d6aec35a234daa6, user_id=ba1b6e5868a549a99760ebe6068fd6b9, role_id=4cfff1d3d68e49c9b04231a19cf76ede)` {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:11:32.116837 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-04014989-aabb-4282-a335-fc221ed08f58 None admin] RBAC: Authorization granted {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:11:32.134248 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 451/923] 10.222.0.27 () {58 vars in 1439 bytes} [Thu Oct 13 04:11:31 2022] PUT /identity/v3/projects/dcec0431fb6849499d6aec35a234daa6/users/ba1b6e5868a549a99760ebe6068fd6b9/roles/4cfff1d3d68e49c9b04231a19cf76ede => generated 0 bytes in 206 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 13 04:11:34.554068 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21050357-94cc-442b-9133-01e2320c4cf5 None None] REQUEST_METHOD: `GET` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:11:34.554651 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21050357-94cc-442b-9133-01e2320c4cf5 None None] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:11:34.554651 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21050357-94cc-442b-9133-01e2320c4cf5 None None] PATH_INFO: `/` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:11:34.556270 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 473/924] 10.222.0.27 () {54 vars in 927 bytes} [Thu Oct 13 04:11:34 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 13 04:11:34.562395 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6a3a482-923f-441e-abb7-3168ddf84d85 None None] REQUEST_METHOD: `POST` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:11:34.562880 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6a3a482-923f-441e-abb7-3168ddf84d85 None None] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:11:34.563090 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6a3a482-923f-441e-abb7-3168ddf84d85 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:11:34.582029 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.auth.core [None req-b6a3a482-923f-441e-abb7-3168ddf84d85 None None] MFA Rules not processed for user `0e2d979563d6421995451a0e86bf3022`. Rule list: `[]` (Enabled: `True`). {{(pid=81408) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 13 04:11:34.583825 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.fernet_utils [None req-b6a3a482-923f-441e-abb7-3168ddf84d85 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81408) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:11:34.651438 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 452/925] 10.222.0.27 () {58 vars in 1024 bytes} [Thu Oct 13 04:11:34 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 90 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Oct 13 04:11:34.922227 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-21050357-94cc-442b-9133-01e2320c4cf5 None None] Authenticating user token {{(pid=81409) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:11:35.044287 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-21050357-94cc-442b-9133-01e2320c4cf5 None None] Validating token access rules against request {{(pid=81409) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:11:35.088875 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f05f714f-9a25-42d4-8fed-aa1fdfee2698 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81409) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:11:35.089512 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f05f714f-9a25-42d4-8fed-aa1fdfee2698 None admin] REQUEST_METHOD: `GET` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:11:35.089790 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f05f714f-9a25-42d4-8fed-aa1fdfee2698 None admin] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:11:35.090056 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f05f714f-9a25-42d4-8fed-aa1fdfee2698 None admin] PATH_INFO: `/v3/roles/service` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:11:35.098029 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG dbcounter [-] [81409] Writing DB stats keystone:SELECT=100 {{(pid=81409) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:11:35.102699 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f05f714f-9a25-42d4-8fed-aa1fdfee2698 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 13 04:11:35.108716 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application [None req-f05f714f-9a25-42d4-8fed-aa1fdfee2698 None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Oct 13 04:11:35.108716 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 13 04:11:35.108716 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 13 04:11:35.108716 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 13 04:11:35.108716 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 13 04:11:35.108716 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 13 04:11:35.108716 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 13 04:11:35.108716 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 13 04:11:35.108716 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 13 04:11:35.108716 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 13 04:11:35.108716 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 13 04:11:35.108716 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 13 04:11:35.108716 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 13 04:11:35.108716 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 13 04:11:35.108716 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 13 04:11:35.108716 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 13 04:11:35.109829 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 13 04:11:35.109829 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 13 04:11:35.109829 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 13 04:11:35.109829 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 13 04:11:35.109829 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 13 04:11:35.109829 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 13 04:11:35.109829 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 13 04:11:35.109829 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 13 04:11:35.109829 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 13 04:11:35.109829 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application return self.get_or_create( Oct 13 04:11:35.109829 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 13 04:11:35.109829 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application with Lock( Oct 13 04:11:35.109829 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 13 04:11:35.109829 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application return self._enter() Oct 13 04:11:35.109829 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 13 04:11:35.109829 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 13 04:11:35.110910 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 13 04:11:35.110910 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application return self.creator() Oct 13 04:11:35.110910 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 13 04:11:35.110910 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application created_value = creator( Oct 13 04:11:35.110910 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 13 04:11:35.110910 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 13 04:11:35.110910 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 13 04:11:35.110910 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 13 04:11:35.110910 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 13 04:11:35.110910 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 13 04:11:35.110910 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Oct 13 04:11:35.110910 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application Oct 13 04:11:35.110910 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 474/926] 10.222.0.27 () {56 vars in 1102 bytes} [Thu Oct 13 04:11:34 2022] GET /identity/v3/roles/service => generated 85 bytes in 188 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 13 04:11:35.116517 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b6a3a482-923f-441e-abb7-3168ddf84d85 None None] Authenticating user token {{(pid=81408) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:11:35.182029 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG dbcounter [-] [81408] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81408) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:11:35.230346 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b6a3a482-923f-441e-abb7-3168ddf84d85 None None] Validating token access rules against request {{(pid=81408) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:11:35.266429 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1a2f2da6-7a6f-4474-b066-fadcd2b5e26c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81408) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:11:35.267050 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a2f2da6-7a6f-4474-b066-fadcd2b5e26c None admin] REQUEST_METHOD: `GET` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:11:35.267362 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a2f2da6-7a6f-4474-b066-fadcd2b5e26c None admin] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:11:35.267594 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a2f2da6-7a6f-4474-b066-fadcd2b5e26c None admin] PATH_INFO: `/v3/roles` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:11:35.268254 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1a2f2da6-7a6f-4474-b066-fadcd2b5e26c None admin] RBAC: Adding query filter params (name=service) {{(pid=81408) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 13 04:11:35.268551 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1a2f2da6-7a6f-4474-b066-fadcd2b5e26c None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:11:35.269922 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1a2f2da6-7a6f-4474-b066-fadcd2b5e26c None admin] RBAC: Authorization granted {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:11:35.279211 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 453/927] 10.222.0.27 () {56 vars in 1103 bytes} [Thu Oct 13 04:11:35 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 163 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 13 04:11:35.285479 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f05f714f-9a25-42d4-8fed-aa1fdfee2698 None admin] Authenticating user token {{(pid=81409) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:11:35.399040 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f05f714f-9a25-42d4-8fed-aa1fdfee2698 None admin] Validating token access rules against request {{(pid=81409) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:11:35.438956 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bc6c4a35-92b1-4b82-8c9e-deee43b8e170 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81409) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:11:35.439645 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc6c4a35-92b1-4b82-8c9e-deee43b8e170 None admin] REQUEST_METHOD: `GET` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:11:35.439958 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc6c4a35-92b1-4b82-8c9e-deee43b8e170 None admin] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:11:35.440329 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc6c4a35-92b1-4b82-8c9e-deee43b8e170 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:11:35.451740 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bc6c4a35-92b1-4b82-8c9e-deee43b8e170 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:11:35.453469 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bc6c4a35-92b1-4b82-8c9e-deee43b8e170 None admin] RBAC: Authorization granted {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:11:35.470944 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application [None req-bc6c4a35-92b1-4b82-8c9e-deee43b8e170 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 13 04:11:35.470944 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 13 04:11:35.470944 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 13 04:11:35.470944 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 13 04:11:35.470944 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 13 04:11:35.470944 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 13 04:11:35.470944 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 13 04:11:35.470944 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 13 04:11:35.470944 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 13 04:11:35.470944 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 13 04:11:35.470944 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 13 04:11:35.470944 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 13 04:11:35.470944 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 13 04:11:35.470944 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 13 04:11:35.470944 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 13 04:11:35.470944 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 13 04:11:35.472033 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 13 04:11:35.472033 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 13 04:11:35.472033 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 13 04:11:35.472033 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 13 04:11:35.472033 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 13 04:11:35.472033 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application return self.get_or_create( Oct 13 04:11:35.472033 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 13 04:11:35.472033 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application with Lock( Oct 13 04:11:35.472033 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 13 04:11:35.472033 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application return self._enter() Oct 13 04:11:35.472033 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 13 04:11:35.472033 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 13 04:11:35.472033 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 13 04:11:35.472033 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application return self.creator() Oct 13 04:11:35.472033 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 13 04:11:35.472033 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application created_value = creator( Oct 13 04:11:35.472033 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 13 04:11:35.472918 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 13 04:11:35.472918 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 13 04:11:35.472918 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application Oct 13 04:11:35.472918 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 475/928] 10.222.0.27 () {56 vars in 1108 bytes} [Thu Oct 13 04:11:35 2022] GET /identity/v3/domains/Default => generated 87 bytes in 187 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 13 04:11:35.478734 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1a2f2da6-7a6f-4474-b066-fadcd2b5e26c None admin] Authenticating user token {{(pid=81408) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:11:35.582914 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1a2f2da6-7a6f-4474-b066-fadcd2b5e26c None admin] Validating token access rules against request {{(pid=81408) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:11:35.600137 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG dbcounter [-] [81408] Writing DB stats keystone:SELECT=100 {{(pid=81408) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:11:35.618367 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-831a177f-7b78-4754-8a53-525c3744ef26 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81408) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:11:35.618964 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-831a177f-7b78-4754-8a53-525c3744ef26 None admin] REQUEST_METHOD: `GET` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:11:35.619240 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-831a177f-7b78-4754-8a53-525c3744ef26 None admin] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:11:35.619511 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-831a177f-7b78-4754-8a53-525c3744ef26 None admin] PATH_INFO: `/v3/domains` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:11:35.620148 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-831a177f-7b78-4754-8a53-525c3744ef26 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81408) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 13 04:11:35.620450 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-831a177f-7b78-4754-8a53-525c3744ef26 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:11:35.621806 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-831a177f-7b78-4754-8a53-525c3744ef26 None admin] RBAC: Authorization granted {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:11:35.634893 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 454/929] 10.222.0.27 () {56 vars in 1109 bytes} [Thu Oct 13 04:11:35 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 157 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 13 04:11:35.640802 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bc6c4a35-92b1-4b82-8c9e-deee43b8e170 None admin] Authenticating user token {{(pid=81409) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:11:35.695136 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG dbcounter [-] [81409] Writing DB stats keystone:SELECT=100 {{(pid=81409) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:11:35.752467 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bc6c4a35-92b1-4b82-8c9e-deee43b8e170 None admin] Validating token access rules against request {{(pid=81409) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:11:35.790981 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f8a771df-bf7d-4248-ab44-8449fd932a7d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81409) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:11:35.791709 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8a771df-bf7d-4248-ab44-8449fd932a7d None admin] REQUEST_METHOD: `GET` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:11:35.792010 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8a771df-bf7d-4248-ab44-8449fd932a7d None admin] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:11:35.792312 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8a771df-bf7d-4248-ab44-8449fd932a7d None admin] PATH_INFO: `/v3/users/placement` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:11:35.800940 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f8a771df-bf7d-4248-ab44-8449fd932a7d None admin] RBAC: Authorizing `identity:get_user(user_id=placement)` {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:11:35.802696 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f8a771df-bf7d-4248-ab44-8449fd932a7d None admin] RBAC: Authorization granted {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:11:35.815666 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application [None req-f8a771df-bf7d-4248-ab44-8449fd932a7d None admin] Could not find user: placement.: keystone.exception.UserNotFound: Could not find user: placement. Oct 13 04:11:35.815666 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 13 04:11:35.815666 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 13 04:11:35.815666 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 13 04:11:35.815666 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 13 04:11:35.815666 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 13 04:11:35.815666 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 13 04:11:35.815666 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 13 04:11:35.815666 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 13 04:11:35.815666 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 13 04:11:35.815666 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 13 04:11:35.815666 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 13 04:11:35.815666 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Oct 13 04:11:35.815666 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application return self._get_user(user_id) Oct 13 04:11:35.815666 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Oct 13 04:11:35.815666 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Oct 13 04:11:35.816746 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 13 04:11:35.816746 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 13 04:11:35.816746 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Oct 13 04:11:35.816746 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 13 04:11:35.816746 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Oct 13 04:11:35.816746 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 13 04:11:35.816746 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 13 04:11:35.816746 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 13 04:11:35.816746 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 13 04:11:35.816746 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application return self.get_or_create( Oct 13 04:11:35.816746 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 13 04:11:35.816746 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application with Lock( Oct 13 04:11:35.816746 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 13 04:11:35.816746 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application return self._enter() Oct 13 04:11:35.816746 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 13 04:11:35.816746 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 13 04:11:35.816746 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 13 04:11:35.816746 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application return self.creator() Oct 13 04:11:35.819072 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 13 04:11:35.819072 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application created_value = creator( Oct 13 04:11:35.819072 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Oct 13 04:11:35.819072 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Oct 13 04:11:35.819072 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Oct 13 04:11:35.819072 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Oct 13 04:11:35.819072 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Oct 13 04:11:35.819072 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Oct 13 04:11:35.819072 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: placement. Oct 13 04:11:35.819072 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application Oct 13 04:11:35.819072 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 476/930] 10.222.0.27 () {56 vars in 1108 bytes} [Thu Oct 13 04:11:35 2022] GET /identity/v3/users/placement => generated 87 bytes in 177 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 13 04:11:35.823451 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-831a177f-7b78-4754-8a53-525c3744ef26 None admin] Authenticating user token {{(pid=81408) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:11:35.933792 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-831a177f-7b78-4754-8a53-525c3744ef26 None admin] Validating token access rules against request {{(pid=81408) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:11:35.969672 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9bd238d3-5862-420f-864e-143d8b3c2436 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81408) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:11:35.970090 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9bd238d3-5862-420f-864e-143d8b3c2436 None admin] REQUEST_METHOD: `GET` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:11:35.970398 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9bd238d3-5862-420f-864e-143d8b3c2436 None admin] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:11:35.970671 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9bd238d3-5862-420f-864e-143d8b3c2436 None admin] PATH_INFO: `/v3/users` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:11:35.971397 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9bd238d3-5862-420f-864e-143d8b3c2436 None admin] RBAC: Adding query filter params (domain_id=default, name=placement) {{(pid=81408) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 13 04:11:35.971704 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9bd238d3-5862-420f-864e-143d8b3c2436 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:11:35.973141 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9bd238d3-5862-420f-864e-143d8b3c2436 None admin] RBAC: Authorization granted {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:11:35.993534 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 455/931] 10.222.0.27 () {56 vars in 1143 bytes} [Thu Oct 13 04:11:35 2022] GET /identity/v3/users?domain_id=default&name=placement => generated 378 bytes in 171 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 13 04:11:35.999830 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f8a771df-bf7d-4248-ab44-8449fd932a7d None admin] Authenticating user token {{(pid=81409) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:11:36.109387 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f8a771df-bf7d-4248-ab44-8449fd932a7d None admin] Validating token access rules against request {{(pid=81409) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:11:36.121210 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG dbcounter [-] [81409] Writing DB stats keystone:SELECT=100 {{(pid=81409) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:11:36.145831 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-28e59356-ed3e-4f8e-bea5-2a492f512aee None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81409) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:11:36.146692 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28e59356-ed3e-4f8e-bea5-2a492f512aee None admin] REQUEST_METHOD: `GET` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:11:36.146961 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28e59356-ed3e-4f8e-bea5-2a492f512aee None admin] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:11:36.147220 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28e59356-ed3e-4f8e-bea5-2a492f512aee None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:11:36.224416 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-28e59356-ed3e-4f8e-bea5-2a492f512aee None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:11:36.225779 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-28e59356-ed3e-4f8e-bea5-2a492f512aee None admin] RBAC: Authorization granted {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:11:36.335561 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 477/932] 10.222.0.27 () {58 vars in 1282 bytes} [Thu Oct 13 04:11:35 2022] GET /identity/v3/auth/tokens => generated 2118 bytes in 336 msecs (HTTP/1.1 200) 6 headers in 359 bytes (1 switches on core 0) Oct 13 04:11:36.341613 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9bd238d3-5862-420f-864e-143d8b3c2436 None admin] Authenticating user token {{(pid=81408) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:11:36.377067 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG dbcounter [-] [81408] Writing DB stats keystone:SELECT=100 {{(pid=81408) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:11:36.463396 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9bd238d3-5862-420f-864e-143d8b3c2436 None admin] Validating token access rules against request {{(pid=81408) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:11:36.499987 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-26dd58de-3143-4213-aae0-72130f9bd71d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81408) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:11:36.500719 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26dd58de-3143-4213-aae0-72130f9bd71d None admin] REQUEST_METHOD: `GET` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:11:36.500995 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26dd58de-3143-4213-aae0-72130f9bd71d None admin] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:11:36.501318 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26dd58de-3143-4213-aae0-72130f9bd71d None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:11:36.512756 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-26dd58de-3143-4213-aae0-72130f9bd71d None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:11:36.514526 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-26dd58de-3143-4213-aae0-72130f9bd71d None admin] RBAC: Authorization granted {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:11:36.532649 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application [None req-26dd58de-3143-4213-aae0-72130f9bd71d None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 13 04:11:36.532649 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 13 04:11:36.532649 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 13 04:11:36.532649 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 13 04:11:36.532649 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 13 04:11:36.532649 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 13 04:11:36.532649 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 13 04:11:36.532649 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 13 04:11:36.532649 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 13 04:11:36.532649 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 13 04:11:36.532649 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 13 04:11:36.532649 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 13 04:11:36.532649 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 13 04:11:36.532649 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 13 04:11:36.532649 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 13 04:11:36.532649 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 13 04:11:36.534020 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 13 04:11:36.534020 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 13 04:11:36.534020 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 13 04:11:36.534020 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 13 04:11:36.534020 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 13 04:11:36.534020 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application return self.get_or_create( Oct 13 04:11:36.534020 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 13 04:11:36.534020 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application with Lock( Oct 13 04:11:36.534020 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 13 04:11:36.534020 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application return self._enter() Oct 13 04:11:36.534020 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 13 04:11:36.534020 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 13 04:11:36.534020 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 13 04:11:36.534020 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application return self.creator() Oct 13 04:11:36.534020 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 13 04:11:36.534020 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application created_value = creator( Oct 13 04:11:36.534020 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 13 04:11:36.540336 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 13 04:11:36.540336 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 13 04:11:36.540336 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application Oct 13 04:11:36.540336 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 456/933] 10.222.0.27 () {56 vars in 1108 bytes} [Thu Oct 13 04:11:36 2022] GET /identity/v3/domains/Default => generated 87 bytes in 193 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 13 04:11:36.540586 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-28e59356-ed3e-4f8e-bea5-2a492f512aee None admin] Authenticating user token {{(pid=81409) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:11:36.553105 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG dbcounter [-] [81409] Writing DB stats keystone:SELECT=100 {{(pid=81409) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:11:36.647090 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-28e59356-ed3e-4f8e-bea5-2a492f512aee None admin] Validating token access rules against request {{(pid=81409) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:11:36.684786 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-28d34805-1437-4e8e-ac50-a570ac537175 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81409) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:11:36.685391 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28d34805-1437-4e8e-ac50-a570ac537175 None admin] REQUEST_METHOD: `GET` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:11:36.685685 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28d34805-1437-4e8e-ac50-a570ac537175 None admin] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:11:36.685935 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28d34805-1437-4e8e-ac50-a570ac537175 None admin] PATH_INFO: `/v3/domains` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:11:36.686602 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-28d34805-1437-4e8e-ac50-a570ac537175 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81409) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 13 04:11:36.686889 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-28d34805-1437-4e8e-ac50-a570ac537175 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:11:36.688247 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-28d34805-1437-4e8e-ac50-a570ac537175 None admin] RBAC: Authorization granted {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:11:36.700265 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 478/934] 10.222.0.27 () {56 vars in 1109 bytes} [Thu Oct 13 04:11:36 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 160 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 13 04:11:36.706136 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-26dd58de-3143-4213-aae0-72130f9bd71d None admin] Authenticating user token {{(pid=81408) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:11:36.801213 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG dbcounter [-] [81408] Writing DB stats keystone:SELECT=100 {{(pid=81408) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:11:36.820176 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-26dd58de-3143-4213-aae0-72130f9bd71d None admin] Validating token access rules against request {{(pid=81408) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:11:36.861442 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5381adbd-9806-4511-ab94-aff54c39b2ca None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81408) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:11:36.862092 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5381adbd-9806-4511-ab94-aff54c39b2ca None admin] REQUEST_METHOD: `GET` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:11:36.862398 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5381adbd-9806-4511-ab94-aff54c39b2ca None admin] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:11:36.862718 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5381adbd-9806-4511-ab94-aff54c39b2ca None admin] PATH_INFO: `/v3/projects/service` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:11:36.871028 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5381adbd-9806-4511-ab94-aff54c39b2ca None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:11:36.872804 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5381adbd-9806-4511-ab94-aff54c39b2ca None admin] RBAC: Authorization granted {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:11:36.886654 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application [None req-5381adbd-9806-4511-ab94-aff54c39b2ca None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Oct 13 04:11:36.886654 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 13 04:11:36.886654 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 13 04:11:36.886654 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 13 04:11:36.886654 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 13 04:11:36.886654 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 13 04:11:36.886654 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 13 04:11:36.886654 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 13 04:11:36.886654 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 13 04:11:36.886654 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 13 04:11:36.886654 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 13 04:11:36.886654 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 13 04:11:36.886654 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Oct 13 04:11:36.886654 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application return self._get_project(project_id) Oct 13 04:11:36.886654 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Oct 13 04:11:36.887696 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Oct 13 04:11:36.887696 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 13 04:11:36.887696 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 13 04:11:36.887696 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 13 04:11:36.887696 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 13 04:11:36.887696 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 13 04:11:36.887696 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application return self.get_or_create( Oct 13 04:11:36.887696 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 13 04:11:36.887696 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application with Lock( Oct 13 04:11:36.887696 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 13 04:11:36.887696 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application return self._enter() Oct 13 04:11:36.887696 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 13 04:11:36.887696 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 13 04:11:36.887696 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 13 04:11:36.887696 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application return self.creator() Oct 13 04:11:36.887696 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 13 04:11:36.887696 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application created_value = creator( Oct 13 04:11:36.888773 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Oct 13 04:11:36.888773 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Oct 13 04:11:36.888773 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Oct 13 04:11:36.888773 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Oct 13 04:11:36.888773 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Oct 13 04:11:36.888773 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Oct 13 04:11:36.888773 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Oct 13 04:11:36.888773 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application Oct 13 04:11:36.888773 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 457/935] 10.222.0.27 () {56 vars in 1111 bytes} [Thu Oct 13 04:11:36 2022] GET /identity/v3/projects/service => generated 88 bytes in 182 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 13 04:11:36.893943 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-28d34805-1437-4e8e-ac50-a570ac537175 None admin] Authenticating user token {{(pid=81409) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:11:36.973095 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG dbcounter [-] [81409] Writing DB stats keystone:SELECT=100 {{(pid=81409) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:11:37.011480 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-28d34805-1437-4e8e-ac50-a570ac537175 None admin] Validating token access rules against request {{(pid=81409) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:11:37.053671 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a2912188-cec9-462c-abed-e7ca877a3ab4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81409) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:11:37.054504 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2912188-cec9-462c-abed-e7ca877a3ab4 None admin] REQUEST_METHOD: `GET` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:11:37.054912 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2912188-cec9-462c-abed-e7ca877a3ab4 None admin] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:11:37.055347 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2912188-cec9-462c-abed-e7ca877a3ab4 None admin] PATH_INFO: `/v3/projects` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:11:37.056314 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a2912188-cec9-462c-abed-e7ca877a3ab4 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81409) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 13 04:11:37.056712 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a2912188-cec9-462c-abed-e7ca877a3ab4 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:11:37.058908 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a2912188-cec9-462c-abed-e7ca877a3ab4 None admin] RBAC: Authorization granted {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:11:37.072821 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 479/936] 10.222.0.27 () {56 vars in 1148 bytes} [Thu Oct 13 04:11:36 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 179 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 13 04:11:37.079042 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5381adbd-9806-4511-ab94-aff54c39b2ca None admin] Authenticating user token {{(pid=81408) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:11:37.194411 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5381adbd-9806-4511-ab94-aff54c39b2ca None admin] Validating token access rules against request {{(pid=81408) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:11:37.230805 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-197916f3-0dea-4851-ac50-53ab3ffaa8c1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81408) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:11:37.231429 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-197916f3-0dea-4851-ac50-53ab3ffaa8c1 None admin] REQUEST_METHOD: `GET` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:11:37.231738 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-197916f3-0dea-4851-ac50-53ab3ffaa8c1 None admin] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:11:37.231963 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-197916f3-0dea-4851-ac50-53ab3ffaa8c1 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:11:37.232688 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-197916f3-0dea-4851-ac50-53ab3ffaa8c1 None admin] RBAC: Adding query filter params (role.id=4cfff1d3d68e49c9b04231a19cf76ede, scope.project.id=dcec0431fb6849499d6aec35a234daa6, user.id=ba1b6e5868a549a99760ebe6068fd6b9) {{(pid=81408) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 13 04:11:37.232995 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-197916f3-0dea-4851-ac50-53ab3ffaa8c1 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:11:37.234450 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-197916f3-0dea-4851-ac50-53ab3ffaa8c1 None admin] RBAC: Authorization granted {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:11:37.241348 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 458/937] 10.222.0.27 () {56 vars in 1374 bytes} [Thu Oct 13 04:11:37 2022] GET /identity/v3/role_assignments?user.id=ba1b6e5868a549a99760ebe6068fd6b9&scope.project.id=dcec0431fb6849499d6aec35a234daa6&role.id=4cfff1d3d68e49c9b04231a19cf76ede => generated 608 bytes in 163 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 13 04:11:39.442042 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62484c2d-bb28-424b-9f82-aef918430319 None None] REQUEST_METHOD: `GET` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:11:39.442998 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62484c2d-bb28-424b-9f82-aef918430319 None None] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:11:39.442998 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62484c2d-bb28-424b-9f82-aef918430319 None None] PATH_INFO: `/` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:11:39.444832 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 480/938] 10.222.0.27 () {54 vars in 927 bytes} [Thu Oct 13 04:11:39 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 13 04:11:39.450851 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2033c296-2e24-4ba0-97f8-b30846d002c3 None None] REQUEST_METHOD: `POST` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:11:39.451357 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2033c296-2e24-4ba0-97f8-b30846d002c3 None None] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:11:39.451357 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2033c296-2e24-4ba0-97f8-b30846d002c3 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:11:39.470521 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.auth.core [None req-2033c296-2e24-4ba0-97f8-b30846d002c3 None None] MFA Rules not processed for user `0e2d979563d6421995451a0e86bf3022`. Rule list: `[]` (Enabled: `True`). {{(pid=81408) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 13 04:11:39.471733 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.fernet_utils [None req-2033c296-2e24-4ba0-97f8-b30846d002c3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81408) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:11:39.475381 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG dbcounter [-] [81408] Writing DB stats keystone:SELECT=100 {{(pid=81408) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:11:39.551018 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 459/939] 10.222.0.27 () {58 vars in 1024 bytes} [Thu Oct 13 04:11:39 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 101 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Oct 13 04:11:39.821667 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-62484c2d-bb28-424b-9f82-aef918430319 None None] Authenticating user token {{(pid=81409) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:11:39.933536 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-62484c2d-bb28-424b-9f82-aef918430319 None None] Validating token access rules against request {{(pid=81409) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:11:39.967629 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG dbcounter [-] [81409] Writing DB stats keystone:SELECT=100 {{(pid=81409) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:11:39.973621 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8b053695-3be4-4bfa-baca-d85bc1658251 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81409) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:11:39.974344 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b053695-3be4-4bfa-baca-d85bc1658251 None admin] REQUEST_METHOD: `GET` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:11:39.974622 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b053695-3be4-4bfa-baca-d85bc1658251 None admin] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:11:39.974918 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b053695-3be4-4bfa-baca-d85bc1658251 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:11:39.986393 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8b053695-3be4-4bfa-baca-d85bc1658251 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 13 04:11:39.993718 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application [None req-8b053695-3be4-4bfa-baca-d85bc1658251 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Oct 13 04:11:39.993718 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 13 04:11:39.993718 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 13 04:11:39.993718 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 13 04:11:39.993718 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 13 04:11:39.993718 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 13 04:11:39.993718 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 13 04:11:39.993718 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 13 04:11:39.993718 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 13 04:11:39.993718 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 13 04:11:39.993718 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 13 04:11:39.993718 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 13 04:11:39.993718 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 13 04:11:39.993718 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 13 04:11:39.993718 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 13 04:11:39.993718 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 13 04:11:39.994926 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 13 04:11:39.994926 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 13 04:11:39.994926 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 13 04:11:39.994926 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 13 04:11:39.994926 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 13 04:11:39.994926 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 13 04:11:39.994926 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 13 04:11:39.994926 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 13 04:11:39.994926 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 13 04:11:39.994926 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application return self.get_or_create( Oct 13 04:11:39.994926 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 13 04:11:39.994926 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application with Lock( Oct 13 04:11:39.994926 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 13 04:11:39.994926 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application return self._enter() Oct 13 04:11:39.994926 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 13 04:11:39.994926 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 13 04:11:39.995838 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 13 04:11:39.995838 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application return self.creator() Oct 13 04:11:39.995838 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 13 04:11:39.995838 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application created_value = creator( Oct 13 04:11:39.995838 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 13 04:11:39.995838 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 13 04:11:39.995838 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 13 04:11:39.995838 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 13 04:11:39.995838 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 13 04:11:39.995838 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 13 04:11:39.995838 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Oct 13 04:11:39.995838 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application Oct 13 04:11:39.995838 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 481/940] 10.222.0.27 () {56 vars in 1096 bytes} [Thu Oct 13 04:11:39 2022] GET /identity/v3/roles/admin => generated 83 bytes in 174 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 13 04:11:40.001268 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2033c296-2e24-4ba0-97f8-b30846d002c3 None None] Authenticating user token {{(pid=81408) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:11:40.125901 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2033c296-2e24-4ba0-97f8-b30846d002c3 None None] Validating token access rules against request {{(pid=81408) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:11:40.166506 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-32536eb7-7b09-4a33-9116-3f0f43c59f64 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81408) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:11:40.167194 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32536eb7-7b09-4a33-9116-3f0f43c59f64 None admin] REQUEST_METHOD: `GET` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:11:40.167537 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32536eb7-7b09-4a33-9116-3f0f43c59f64 None admin] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:11:40.167869 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32536eb7-7b09-4a33-9116-3f0f43c59f64 None admin] PATH_INFO: `/v3/roles` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:11:40.168596 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-32536eb7-7b09-4a33-9116-3f0f43c59f64 None admin] RBAC: Adding query filter params (name=admin) {{(pid=81408) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 13 04:11:40.168962 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-32536eb7-7b09-4a33-9116-3f0f43c59f64 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:11:40.170416 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-32536eb7-7b09-4a33-9116-3f0f43c59f64 None admin] RBAC: Authorization granted {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:11:40.178750 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 460/941] 10.222.0.27 () {56 vars in 1099 bytes} [Thu Oct 13 04:11:40 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 178 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 13 04:11:40.184516 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8b053695-3be4-4bfa-baca-d85bc1658251 None admin] Authenticating user token {{(pid=81409) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:11:40.299990 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8b053695-3be4-4bfa-baca-d85bc1658251 None admin] Validating token access rules against request {{(pid=81409) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:11:40.335264 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f8987dee-5fc3-426f-8be5-4814fdb30c76 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81409) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:11:40.335884 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8987dee-5fc3-426f-8be5-4814fdb30c76 None admin] REQUEST_METHOD: `GET` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:11:40.336160 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8987dee-5fc3-426f-8be5-4814fdb30c76 None admin] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:11:40.336428 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8987dee-5fc3-426f-8be5-4814fdb30c76 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:11:40.346259 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f8987dee-5fc3-426f-8be5-4814fdb30c76 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:11:40.347690 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f8987dee-5fc3-426f-8be5-4814fdb30c76 None admin] RBAC: Authorization granted {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:11:40.363669 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application [None req-f8987dee-5fc3-426f-8be5-4814fdb30c76 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 13 04:11:40.363669 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 13 04:11:40.363669 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 13 04:11:40.363669 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 13 04:11:40.363669 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 13 04:11:40.363669 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 13 04:11:40.363669 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 13 04:11:40.363669 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 13 04:11:40.363669 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 13 04:11:40.363669 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 13 04:11:40.363669 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 13 04:11:40.363669 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 13 04:11:40.363669 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 13 04:11:40.363669 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 13 04:11:40.363669 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 13 04:11:40.363669 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 13 04:11:40.365766 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 13 04:11:40.365766 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 13 04:11:40.365766 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 13 04:11:40.365766 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 13 04:11:40.365766 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 13 04:11:40.365766 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application return self.get_or_create( Oct 13 04:11:40.365766 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 13 04:11:40.365766 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application with Lock( Oct 13 04:11:40.365766 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 13 04:11:40.365766 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application return self._enter() Oct 13 04:11:40.365766 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 13 04:11:40.365766 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 13 04:11:40.365766 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 13 04:11:40.365766 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application return self.creator() Oct 13 04:11:40.365766 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 13 04:11:40.365766 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application created_value = creator( Oct 13 04:11:40.365766 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 13 04:11:40.366871 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 13 04:11:40.366871 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 13 04:11:40.366871 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application Oct 13 04:11:40.366871 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 482/942] 10.222.0.27 () {56 vars in 1108 bytes} [Thu Oct 13 04:11:40 2022] GET /identity/v3/domains/Default => generated 87 bytes in 182 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Oct 13 04:11:40.371536 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-32536eb7-7b09-4a33-9116-3f0f43c59f64 None admin] Authenticating user token {{(pid=81408) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:11:40.374771 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG dbcounter [-] [81408] Writing DB stats keystone:SELECT=100 {{(pid=81408) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:11:40.486924 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-32536eb7-7b09-4a33-9116-3f0f43c59f64 None admin] Validating token access rules against request {{(pid=81408) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:11:40.523475 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ca163241-1a85-40bc-9ca1-463f474ad795 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81408) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:11:40.524149 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca163241-1a85-40bc-9ca1-463f474ad795 None admin] REQUEST_METHOD: `GET` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:11:40.524429 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca163241-1a85-40bc-9ca1-463f474ad795 None admin] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:11:40.524698 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca163241-1a85-40bc-9ca1-463f474ad795 None admin] PATH_INFO: `/v3/domains` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:11:40.525435 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ca163241-1a85-40bc-9ca1-463f474ad795 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81408) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 13 04:11:40.525774 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ca163241-1a85-40bc-9ca1-463f474ad795 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:11:40.527244 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ca163241-1a85-40bc-9ca1-463f474ad795 None admin] RBAC: Authorization granted {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:11:40.540492 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 461/943] 10.222.0.27 () {56 vars in 1109 bytes} [Thu Oct 13 04:11:40 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 170 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 13 04:11:40.546334 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f8987dee-5fc3-426f-8be5-4814fdb30c76 None admin] Authenticating user token {{(pid=81409) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:11:40.594239 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG dbcounter [-] [81409] Writing DB stats keystone:SELECT=100 {{(pid=81409) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:11:40.658756 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f8987dee-5fc3-426f-8be5-4814fdb30c76 None admin] Validating token access rules against request {{(pid=81409) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:11:40.698363 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a5593884-1226-4a54-bcc6-b94984d684b6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81409) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:11:40.698919 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5593884-1226-4a54-bcc6-b94984d684b6 None admin] REQUEST_METHOD: `GET` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:11:40.699193 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5593884-1226-4a54-bcc6-b94984d684b6 None admin] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:11:40.699459 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5593884-1226-4a54-bcc6-b94984d684b6 None admin] PATH_INFO: `/v3/users/placement` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:11:40.709088 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a5593884-1226-4a54-bcc6-b94984d684b6 None admin] RBAC: Authorizing `identity:get_user(user_id=placement)` {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:11:40.710638 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a5593884-1226-4a54-bcc6-b94984d684b6 None admin] RBAC: Authorization granted {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:11:40.722828 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application [None req-a5593884-1226-4a54-bcc6-b94984d684b6 None admin] Could not find user: placement.: keystone.exception.UserNotFound: Could not find user: placement. Oct 13 04:11:40.722828 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 13 04:11:40.722828 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 13 04:11:40.722828 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 13 04:11:40.722828 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 13 04:11:40.722828 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 13 04:11:40.722828 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 13 04:11:40.722828 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 13 04:11:40.722828 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 13 04:11:40.722828 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 13 04:11:40.722828 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 13 04:11:40.722828 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 13 04:11:40.722828 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Oct 13 04:11:40.722828 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application return self._get_user(user_id) Oct 13 04:11:40.722828 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Oct 13 04:11:40.722828 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Oct 13 04:11:40.724585 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 13 04:11:40.724585 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 13 04:11:40.724585 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Oct 13 04:11:40.724585 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 13 04:11:40.724585 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Oct 13 04:11:40.724585 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 13 04:11:40.724585 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 13 04:11:40.724585 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 13 04:11:40.724585 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 13 04:11:40.724585 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application return self.get_or_create( Oct 13 04:11:40.724585 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 13 04:11:40.724585 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application with Lock( Oct 13 04:11:40.724585 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 13 04:11:40.724585 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application return self._enter() Oct 13 04:11:40.724585 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 13 04:11:40.724585 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 13 04:11:40.724585 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 13 04:11:40.724585 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application return self.creator() Oct 13 04:11:40.725552 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 13 04:11:40.725552 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application created_value = creator( Oct 13 04:11:40.725552 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Oct 13 04:11:40.725552 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Oct 13 04:11:40.725552 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Oct 13 04:11:40.725552 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Oct 13 04:11:40.725552 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Oct 13 04:11:40.725552 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Oct 13 04:11:40.725552 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: placement. Oct 13 04:11:40.725552 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application Oct 13 04:11:40.725552 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 483/944] 10.222.0.27 () {56 vars in 1108 bytes} [Thu Oct 13 04:11:40 2022] GET /identity/v3/users/placement => generated 87 bytes in 179 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 13 04:11:40.731352 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ca163241-1a85-40bc-9ca1-463f474ad795 None admin] Authenticating user token {{(pid=81408) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:11:40.798743 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG dbcounter [-] [81408] Writing DB stats keystone:SELECT=100 {{(pid=81408) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:11:40.844418 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ca163241-1a85-40bc-9ca1-463f474ad795 None admin] Validating token access rules against request {{(pid=81408) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:11:40.882681 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0bf69a8c-096a-4849-a2c1-b95f64d9fb8d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81408) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:11:40.883288 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0bf69a8c-096a-4849-a2c1-b95f64d9fb8d None admin] REQUEST_METHOD: `GET` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:11:40.883562 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0bf69a8c-096a-4849-a2c1-b95f64d9fb8d None admin] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:11:40.883827 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0bf69a8c-096a-4849-a2c1-b95f64d9fb8d None admin] PATH_INFO: `/v3/users` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:11:40.884558 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0bf69a8c-096a-4849-a2c1-b95f64d9fb8d None admin] RBAC: Adding query filter params (domain_id=default, name=placement) {{(pid=81408) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 13 04:11:40.884862 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0bf69a8c-096a-4849-a2c1-b95f64d9fb8d None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:11:40.886314 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0bf69a8c-096a-4849-a2c1-b95f64d9fb8d None admin] RBAC: Authorization granted {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:11:40.905407 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 462/945] 10.222.0.27 () {56 vars in 1143 bytes} [Thu Oct 13 04:11:40 2022] GET /identity/v3/users?domain_id=default&name=placement => generated 378 bytes in 175 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 13 04:11:40.911443 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a5593884-1226-4a54-bcc6-b94984d684b6 None admin] Authenticating user token {{(pid=81409) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:11:41.054577 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a5593884-1226-4a54-bcc6-b94984d684b6 None admin] Validating token access rules against request {{(pid=81409) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:11:41.056285 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG dbcounter [-] [81409] Writing DB stats keystone:SELECT=100 {{(pid=81409) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:11:41.093032 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-53fd2ef6-ead3-4848-b358-d6472313e114 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81409) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:11:41.093664 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53fd2ef6-ead3-4848-b358-d6472313e114 None admin] REQUEST_METHOD: `GET` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:11:41.093934 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53fd2ef6-ead3-4848-b358-d6472313e114 None admin] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:11:41.094217 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53fd2ef6-ead3-4848-b358-d6472313e114 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:11:41.105449 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-53fd2ef6-ead3-4848-b358-d6472313e114 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:11:41.106915 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-53fd2ef6-ead3-4848-b358-d6472313e114 None admin] RBAC: Authorization granted {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:11:41.120697 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application [None req-53fd2ef6-ead3-4848-b358-d6472313e114 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 13 04:11:41.120697 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 13 04:11:41.120697 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 13 04:11:41.120697 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 13 04:11:41.120697 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 13 04:11:41.120697 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 13 04:11:41.120697 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 13 04:11:41.120697 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 13 04:11:41.120697 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 13 04:11:41.120697 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 13 04:11:41.120697 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 13 04:11:41.120697 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 13 04:11:41.120697 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 13 04:11:41.120697 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 13 04:11:41.120697 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 13 04:11:41.120697 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 13 04:11:41.126378 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 13 04:11:41.126378 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 13 04:11:41.126378 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 13 04:11:41.126378 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 13 04:11:41.126378 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 13 04:11:41.126378 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application return self.get_or_create( Oct 13 04:11:41.126378 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 13 04:11:41.126378 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application with Lock( Oct 13 04:11:41.126378 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 13 04:11:41.126378 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application return self._enter() Oct 13 04:11:41.126378 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 13 04:11:41.126378 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 13 04:11:41.126378 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 13 04:11:41.126378 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application return self.creator() Oct 13 04:11:41.126378 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 13 04:11:41.126378 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application created_value = creator( Oct 13 04:11:41.126378 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 13 04:11:41.127543 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 13 04:11:41.127543 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 13 04:11:41.127543 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application Oct 13 04:11:41.127543 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 484/946] 10.222.0.27 () {56 vars in 1108 bytes} [Thu Oct 13 04:11:40 2022] GET /identity/v3/domains/Default => generated 87 bytes in 211 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Oct 13 04:11:41.127813 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0bf69a8c-096a-4849-a2c1-b95f64d9fb8d None admin] Authenticating user token {{(pid=81408) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:11:41.231534 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0bf69a8c-096a-4849-a2c1-b95f64d9fb8d None admin] Validating token access rules against request {{(pid=81408) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:11:41.253303 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG dbcounter [-] [81408] Writing DB stats keystone:SELECT=100 {{(pid=81408) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:11:41.269793 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7c02285d-3df6-4c7c-afb9-b73933c9ef22 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81408) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:11:41.270451 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c02285d-3df6-4c7c-afb9-b73933c9ef22 None admin] REQUEST_METHOD: `GET` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:11:41.271280 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c02285d-3df6-4c7c-afb9-b73933c9ef22 None admin] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:11:41.271554 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c02285d-3df6-4c7c-afb9-b73933c9ef22 None admin] PATH_INFO: `/v3/domains` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:11:41.272245 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7c02285d-3df6-4c7c-afb9-b73933c9ef22 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81408) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 13 04:11:41.272544 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7c02285d-3df6-4c7c-afb9-b73933c9ef22 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:11:41.273914 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7c02285d-3df6-4c7c-afb9-b73933c9ef22 None admin] RBAC: Authorization granted {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:11:41.286532 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 463/947] 10.222.0.27 () {56 vars in 1109 bytes} [Thu Oct 13 04:11:41 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 159 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 13 04:11:41.292149 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-53fd2ef6-ead3-4848-b358-d6472313e114 None admin] Authenticating user token {{(pid=81409) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:11:41.422440 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-53fd2ef6-ead3-4848-b358-d6472313e114 None admin] Validating token access rules against request {{(pid=81409) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:11:41.459085 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-81f7beee-82cb-476a-b65b-1f94f4ee6aff None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81409) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:11:41.459697 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81f7beee-82cb-476a-b65b-1f94f4ee6aff None admin] REQUEST_METHOD: `GET` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:11:41.459968 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81f7beee-82cb-476a-b65b-1f94f4ee6aff None admin] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:11:41.460276 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81f7beee-82cb-476a-b65b-1f94f4ee6aff None admin] PATH_INFO: `/v3/projects/service` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:11:41.467709 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-81f7beee-82cb-476a-b65b-1f94f4ee6aff None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:11:41.469174 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-81f7beee-82cb-476a-b65b-1f94f4ee6aff None admin] RBAC: Authorization granted {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:11:41.482331 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application [None req-81f7beee-82cb-476a-b65b-1f94f4ee6aff None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Oct 13 04:11:41.482331 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 13 04:11:41.482331 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 13 04:11:41.482331 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 13 04:11:41.482331 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 13 04:11:41.482331 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 13 04:11:41.482331 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 13 04:11:41.482331 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 13 04:11:41.482331 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 13 04:11:41.482331 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 13 04:11:41.482331 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 13 04:11:41.482331 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 13 04:11:41.482331 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Oct 13 04:11:41.482331 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application return self._get_project(project_id) Oct 13 04:11:41.482331 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Oct 13 04:11:41.488130 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Oct 13 04:11:41.488130 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 13 04:11:41.488130 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 13 04:11:41.488130 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 13 04:11:41.488130 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 13 04:11:41.488130 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 13 04:11:41.488130 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application return self.get_or_create( Oct 13 04:11:41.488130 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 13 04:11:41.488130 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application with Lock( Oct 13 04:11:41.488130 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 13 04:11:41.488130 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application return self._enter() Oct 13 04:11:41.488130 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 13 04:11:41.488130 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 13 04:11:41.488130 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 13 04:11:41.488130 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application return self.creator() Oct 13 04:11:41.488130 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 13 04:11:41.488130 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application created_value = creator( Oct 13 04:11:41.489083 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Oct 13 04:11:41.489083 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Oct 13 04:11:41.489083 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Oct 13 04:11:41.489083 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Oct 13 04:11:41.489083 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Oct 13 04:11:41.489083 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Oct 13 04:11:41.489083 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Oct 13 04:11:41.489083 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application Oct 13 04:11:41.489083 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 485/948] 10.222.0.27 () {56 vars in 1111 bytes} [Thu Oct 13 04:11:41 2022] GET /identity/v3/projects/service => generated 88 bytes in 192 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 13 04:11:41.490716 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7c02285d-3df6-4c7c-afb9-b73933c9ef22 None admin] Authenticating user token {{(pid=81408) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:11:41.606821 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7c02285d-3df6-4c7c-afb9-b73933c9ef22 None admin] Validating token access rules against request {{(pid=81408) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:11:41.644413 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4de456c3-b294-48f9-a397-8c16638c2d5a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81408) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:11:41.645095 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4de456c3-b294-48f9-a397-8c16638c2d5a None admin] REQUEST_METHOD: `GET` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:11:41.645529 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4de456c3-b294-48f9-a397-8c16638c2d5a None admin] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:11:41.645821 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4de456c3-b294-48f9-a397-8c16638c2d5a None admin] PATH_INFO: `/v3/projects` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:11:41.646669 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4de456c3-b294-48f9-a397-8c16638c2d5a None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81408) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 13 04:11:41.647103 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4de456c3-b294-48f9-a397-8c16638c2d5a None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:11:41.648895 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4de456c3-b294-48f9-a397-8c16638c2d5a None admin] RBAC: Authorization granted {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:11:41.662910 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 464/949] 10.222.0.27 () {56 vars in 1148 bytes} [Thu Oct 13 04:11:41 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 174 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 13 04:11:41.668942 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-81f7beee-82cb-476a-b65b-1f94f4ee6aff None admin] Authenticating user token {{(pid=81409) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:11:41.687078 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG dbcounter [-] [81409] Writing DB stats keystone:SELECT=100 {{(pid=81409) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:11:41.790552 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-81f7beee-82cb-476a-b65b-1f94f4ee6aff None admin] Validating token access rules against request {{(pid=81409) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:11:41.828403 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-27c26f3d-598d-4083-8eed-27d2f39c8847 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81409) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:11:41.829080 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27c26f3d-598d-4083-8eed-27d2f39c8847 None admin] REQUEST_METHOD: `PUT` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:11:41.829397 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27c26f3d-598d-4083-8eed-27d2f39c8847 None admin] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:11:41.830906 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27c26f3d-598d-4083-8eed-27d2f39c8847 None admin] PATH_INFO: `/v3/projects/dcec0431fb6849499d6aec35a234daa6/users/ba1b6e5868a549a99760ebe6068fd6b9/roles/ef55fbd94e1040be83695754075a7ed5` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:11:41.832334 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-27c26f3d-598d-4083-8eed-27d2f39c8847 None admin] RBAC: Authorizing `identity:create_grant(project_id=dcec0431fb6849499d6aec35a234daa6, user_id=ba1b6e5868a549a99760ebe6068fd6b9, role_id=ef55fbd94e1040be83695754075a7ed5)` {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:11:41.834137 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-27c26f3d-598d-4083-8eed-27d2f39c8847 None admin] RBAC: Authorization granted {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:11:41.847854 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 486/950] 10.222.0.27 () {58 vars in 1439 bytes} [Thu Oct 13 04:11:41 2022] PUT /identity/v3/projects/dcec0431fb6849499d6aec35a234daa6/users/ba1b6e5868a549a99760ebe6068fd6b9/roles/ef55fbd94e1040be83695754075a7ed5 => generated 0 bytes in 179 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 13 04:11:44.078863 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62c25691-c78c-444b-8530-a6aad63e3276 None None] REQUEST_METHOD: `GET` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:11:44.079410 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62c25691-c78c-444b-8530-a6aad63e3276 None None] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:11:44.079410 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62c25691-c78c-444b-8530-a6aad63e3276 None None] PATH_INFO: `/` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:11:44.080751 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 465/951] 10.222.0.27 () {54 vars in 927 bytes} [Thu Oct 13 04:11:44 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 13 04:11:44.086685 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d25f004c-3fb2-4d70-b4b4-d372abc7d3c6 None None] REQUEST_METHOD: `POST` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:11:44.087103 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d25f004c-3fb2-4d70-b4b4-d372abc7d3c6 None None] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:11:44.087223 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d25f004c-3fb2-4d70-b4b4-d372abc7d3c6 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:11:44.107585 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.auth.core [None req-d25f004c-3fb2-4d70-b4b4-d372abc7d3c6 None None] MFA Rules not processed for user `0e2d979563d6421995451a0e86bf3022`. Rule list: `[]` (Enabled: `True`). {{(pid=81409) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 13 04:11:44.108789 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.fernet_utils [None req-d25f004c-3fb2-4d70-b4b4-d372abc7d3c6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81409) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:11:44.187419 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 487/952] 10.222.0.27 () {58 vars in 1024 bytes} [Thu Oct 13 04:11:44 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 102 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Oct 13 04:11:44.471280 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-62c25691-c78c-444b-8530-a6aad63e3276 None None] Authenticating user token {{(pid=81408) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:11:44.519011 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG dbcounter [-] [81408] Writing DB stats keystone:SELECT=100 {{(pid=81408) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:11:44.583638 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-62c25691-c78c-444b-8530-a6aad63e3276 None None] Validating token access rules against request {{(pid=81408) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:11:44.623165 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-abdd8a77-4d64-45cd-af88-4b1c07d240e9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81408) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:11:44.623811 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-abdd8a77-4d64-45cd-af88-4b1c07d240e9 None admin] REQUEST_METHOD: `GET` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:11:44.624096 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-abdd8a77-4d64-45cd-af88-4b1c07d240e9 None admin] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:11:44.624350 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-abdd8a77-4d64-45cd-af88-4b1c07d240e9 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:11:44.635985 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-abdd8a77-4d64-45cd-af88-4b1c07d240e9 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 13 04:11:44.640692 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application [None req-abdd8a77-4d64-45cd-af88-4b1c07d240e9 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Oct 13 04:11:44.640692 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 13 04:11:44.640692 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 13 04:11:44.640692 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 13 04:11:44.640692 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 13 04:11:44.640692 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 13 04:11:44.640692 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 13 04:11:44.640692 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 13 04:11:44.640692 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 13 04:11:44.640692 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 13 04:11:44.640692 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 13 04:11:44.640692 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 13 04:11:44.640692 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 13 04:11:44.640692 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 13 04:11:44.640692 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 13 04:11:44.640692 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 13 04:11:44.641736 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 13 04:11:44.641736 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 13 04:11:44.641736 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 13 04:11:44.641736 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 13 04:11:44.641736 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 13 04:11:44.641736 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 13 04:11:44.641736 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 13 04:11:44.641736 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 13 04:11:44.641736 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 13 04:11:44.641736 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application return self.get_or_create( Oct 13 04:11:44.641736 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 13 04:11:44.641736 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application with Lock( Oct 13 04:11:44.641736 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 13 04:11:44.641736 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application return self._enter() Oct 13 04:11:44.641736 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 13 04:11:44.641736 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 13 04:11:44.642637 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 13 04:11:44.642637 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application return self.creator() Oct 13 04:11:44.642637 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 13 04:11:44.642637 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application created_value = creator( Oct 13 04:11:44.642637 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 13 04:11:44.642637 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 13 04:11:44.642637 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 13 04:11:44.642637 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 13 04:11:44.642637 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 13 04:11:44.642637 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 13 04:11:44.642637 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Oct 13 04:11:44.642637 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application Oct 13 04:11:44.642637 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 466/953] 10.222.0.27 () {56 vars in 1096 bytes} [Thu Oct 13 04:11:44 2022] GET /identity/v3/roles/admin => generated 83 bytes in 171 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 13 04:11:44.648702 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d25f004c-3fb2-4d70-b4b4-d372abc7d3c6 None None] Authenticating user token {{(pid=81409) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:11:44.651809 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG dbcounter [-] [81409] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81409) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:11:44.760659 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d25f004c-3fb2-4d70-b4b4-d372abc7d3c6 None None] Validating token access rules against request {{(pid=81409) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:11:44.796836 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-89b22e36-a955-48ea-8165-6be24f1679b8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81409) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:11:44.797436 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89b22e36-a955-48ea-8165-6be24f1679b8 None admin] REQUEST_METHOD: `GET` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:11:44.797707 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89b22e36-a955-48ea-8165-6be24f1679b8 None admin] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:11:44.797995 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89b22e36-a955-48ea-8165-6be24f1679b8 None admin] PATH_INFO: `/v3/roles` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:11:44.798656 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-89b22e36-a955-48ea-8165-6be24f1679b8 None admin] RBAC: Adding query filter params (name=admin) {{(pid=81409) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 13 04:11:44.798952 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-89b22e36-a955-48ea-8165-6be24f1679b8 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:11:44.800299 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-89b22e36-a955-48ea-8165-6be24f1679b8 None admin] RBAC: Authorization granted {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:11:44.809866 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 488/954] 10.222.0.27 () {56 vars in 1099 bytes} [Thu Oct 13 04:11:44 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 162 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 13 04:11:44.815837 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-abdd8a77-4d64-45cd-af88-4b1c07d240e9 None admin] Authenticating user token {{(pid=81408) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:11:44.924905 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG dbcounter [-] [81408] Writing DB stats keystone:SELECT=100 {{(pid=81408) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:11:44.928525 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-abdd8a77-4d64-45cd-af88-4b1c07d240e9 None admin] Validating token access rules against request {{(pid=81408) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:11:44.965532 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e075e76c-4244-4cdb-8dd5-4bbdefd68be2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81408) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:11:44.966233 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e075e76c-4244-4cdb-8dd5-4bbdefd68be2 None admin] REQUEST_METHOD: `GET` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:11:44.966518 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e075e76c-4244-4cdb-8dd5-4bbdefd68be2 None admin] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:11:44.966845 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e075e76c-4244-4cdb-8dd5-4bbdefd68be2 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:11:44.978281 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e075e76c-4244-4cdb-8dd5-4bbdefd68be2 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:11:44.980026 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e075e76c-4244-4cdb-8dd5-4bbdefd68be2 None admin] RBAC: Authorization granted {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:11:44.995477 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application [None req-e075e76c-4244-4cdb-8dd5-4bbdefd68be2 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 13 04:11:44.995477 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 13 04:11:44.995477 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 13 04:11:44.995477 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 13 04:11:44.995477 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 13 04:11:44.995477 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 13 04:11:44.995477 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 13 04:11:44.995477 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 13 04:11:44.995477 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 13 04:11:44.995477 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 13 04:11:44.995477 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 13 04:11:44.995477 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 13 04:11:44.995477 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 13 04:11:44.995477 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 13 04:11:44.995477 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 13 04:11:44.995477 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 13 04:11:44.996585 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 13 04:11:44.996585 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 13 04:11:44.996585 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 13 04:11:44.996585 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 13 04:11:44.996585 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 13 04:11:44.996585 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application return self.get_or_create( Oct 13 04:11:44.996585 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 13 04:11:44.996585 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application with Lock( Oct 13 04:11:44.996585 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 13 04:11:44.996585 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application return self._enter() Oct 13 04:11:44.996585 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 13 04:11:44.996585 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 13 04:11:44.996585 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 13 04:11:44.996585 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application return self.creator() Oct 13 04:11:44.996585 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 13 04:11:44.996585 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application created_value = creator( Oct 13 04:11:44.996585 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 13 04:11:44.997682 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 13 04:11:44.997682 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 13 04:11:44.997682 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application Oct 13 04:11:44.997682 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 467/955] 10.222.0.27 () {56 vars in 1108 bytes} [Thu Oct 13 04:11:44 2022] GET /identity/v3/domains/Default => generated 87 bytes in 181 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 13 04:11:45.004348 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-89b22e36-a955-48ea-8165-6be24f1679b8 None admin] Authenticating user token {{(pid=81409) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:11:45.077892 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG dbcounter [-] [81409] Writing DB stats keystone:SELECT=100 {{(pid=81409) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:11:45.115642 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-89b22e36-a955-48ea-8165-6be24f1679b8 None admin] Validating token access rules against request {{(pid=81409) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:11:45.154879 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c1a041ac-8285-4b3a-a2bf-6e1a16aa27ee None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81409) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:11:45.155612 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1a041ac-8285-4b3a-a2bf-6e1a16aa27ee None admin] REQUEST_METHOD: `GET` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:11:45.155963 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1a041ac-8285-4b3a-a2bf-6e1a16aa27ee None admin] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:11:45.156208 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1a041ac-8285-4b3a-a2bf-6e1a16aa27ee None admin] PATH_INFO: `/v3/domains` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:11:45.156958 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c1a041ac-8285-4b3a-a2bf-6e1a16aa27ee None admin] RBAC: Adding query filter params (name=Default) {{(pid=81409) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 13 04:11:45.157264 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c1a041ac-8285-4b3a-a2bf-6e1a16aa27ee None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:11:45.158650 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c1a041ac-8285-4b3a-a2bf-6e1a16aa27ee None admin] RBAC: Authorization granted {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:11:45.171180 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 489/956] 10.222.0.27 () {56 vars in 1109 bytes} [Thu Oct 13 04:11:45 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 167 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 13 04:11:45.177539 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e075e76c-4244-4cdb-8dd5-4bbdefd68be2 None admin] Authenticating user token {{(pid=81408) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:11:45.289552 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e075e76c-4244-4cdb-8dd5-4bbdefd68be2 None admin] Validating token access rules against request {{(pid=81408) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:11:45.326275 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-41b35e70-96ed-419b-86b9-9654f64b9cf5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81408) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:11:45.327111 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41b35e70-96ed-419b-86b9-9654f64b9cf5 None admin] REQUEST_METHOD: `GET` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:11:45.327489 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41b35e70-96ed-419b-86b9-9654f64b9cf5 None admin] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:11:45.327768 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41b35e70-96ed-419b-86b9-9654f64b9cf5 None admin] PATH_INFO: `/v3/users/placement` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:11:45.337668 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-41b35e70-96ed-419b-86b9-9654f64b9cf5 None admin] RBAC: Authorizing `identity:get_user(user_id=placement)` {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:11:45.339485 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-41b35e70-96ed-419b-86b9-9654f64b9cf5 None admin] RBAC: Authorization granted {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:11:45.353980 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application [None req-41b35e70-96ed-419b-86b9-9654f64b9cf5 None admin] Could not find user: placement.: keystone.exception.UserNotFound: Could not find user: placement. Oct 13 04:11:45.353980 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 13 04:11:45.353980 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 13 04:11:45.353980 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 13 04:11:45.353980 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 13 04:11:45.353980 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 13 04:11:45.353980 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 13 04:11:45.353980 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 13 04:11:45.353980 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 13 04:11:45.353980 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 13 04:11:45.353980 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 13 04:11:45.353980 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 13 04:11:45.353980 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Oct 13 04:11:45.353980 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application return self._get_user(user_id) Oct 13 04:11:45.353980 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Oct 13 04:11:45.353980 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Oct 13 04:11:45.355176 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 13 04:11:45.355176 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 13 04:11:45.355176 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Oct 13 04:11:45.355176 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 13 04:11:45.355176 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Oct 13 04:11:45.355176 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 13 04:11:45.355176 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 13 04:11:45.355176 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 13 04:11:45.355176 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 13 04:11:45.355176 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application return self.get_or_create( Oct 13 04:11:45.355176 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 13 04:11:45.355176 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application with Lock( Oct 13 04:11:45.355176 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 13 04:11:45.355176 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application return self._enter() Oct 13 04:11:45.355176 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 13 04:11:45.355176 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 13 04:11:45.355176 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 13 04:11:45.355176 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application return self.creator() Oct 13 04:11:45.356378 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 13 04:11:45.356378 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application created_value = creator( Oct 13 04:11:45.356378 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Oct 13 04:11:45.356378 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Oct 13 04:11:45.356378 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Oct 13 04:11:45.356378 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Oct 13 04:11:45.356378 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Oct 13 04:11:45.356378 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Oct 13 04:11:45.356378 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: placement. Oct 13 04:11:45.356378 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application Oct 13 04:11:45.356378 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 468/957] 10.222.0.27 () {56 vars in 1108 bytes} [Thu Oct 13 04:11:45 2022] GET /identity/v3/users/placement => generated 87 bytes in 179 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 13 04:11:45.361239 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c1a041ac-8285-4b3a-a2bf-6e1a16aa27ee None admin] Authenticating user token {{(pid=81409) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:11:45.484528 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c1a041ac-8285-4b3a-a2bf-6e1a16aa27ee None admin] Validating token access rules against request {{(pid=81409) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:11:45.511193 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG dbcounter [-] [81409] Writing DB stats keystone:SELECT=100 {{(pid=81409) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:11:45.521920 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8100cd86-3622-42d9-b402-29be144d6afc None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81409) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:11:45.522653 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8100cd86-3622-42d9-b402-29be144d6afc None admin] REQUEST_METHOD: `GET` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:11:45.522969 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8100cd86-3622-42d9-b402-29be144d6afc None admin] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:11:45.523346 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8100cd86-3622-42d9-b402-29be144d6afc None admin] PATH_INFO: `/v3/users` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:11:45.524151 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8100cd86-3622-42d9-b402-29be144d6afc None admin] RBAC: Adding query filter params (domain_id=default, name=placement) {{(pid=81409) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 13 04:11:45.524455 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8100cd86-3622-42d9-b402-29be144d6afc None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:11:45.525908 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8100cd86-3622-42d9-b402-29be144d6afc None admin] RBAC: Authorization granted {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:11:45.545993 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 490/958] 10.222.0.27 () {56 vars in 1143 bytes} [Thu Oct 13 04:11:45 2022] GET /identity/v3/users?domain_id=default&name=placement => generated 378 bytes in 185 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 13 04:11:45.552210 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-41b35e70-96ed-419b-86b9-9654f64b9cf5 None admin] Authenticating user token {{(pid=81408) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:11:45.569109 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG dbcounter [-] [81408] Writing DB stats keystone:SELECT=100 {{(pid=81408) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:11:45.663095 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-41b35e70-96ed-419b-86b9-9654f64b9cf5 None admin] Validating token access rules against request {{(pid=81408) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:11:45.699863 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5430326c-e822-417f-9f80-d7785cdba90e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81408) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:11:45.700483 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5430326c-e822-417f-9f80-d7785cdba90e None admin] REQUEST_METHOD: `GET` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:11:45.700756 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5430326c-e822-417f-9f80-d7785cdba90e None admin] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:11:45.701020 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5430326c-e822-417f-9f80-d7785cdba90e None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:11:45.774652 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5430326c-e822-417f-9f80-d7785cdba90e None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:11:45.776119 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5430326c-e822-417f-9f80-d7785cdba90e None admin] RBAC: Authorization granted {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:11:45.792548 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG dbcounter [-] [81408] Writing DB stats keystone:SELECT=100 {{(pid=81408) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:11:45.883224 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 469/959] 10.222.0.27 () {58 vars in 1282 bytes} [Thu Oct 13 04:11:45 2022] GET /identity/v3/auth/tokens => generated 2118 bytes in 332 msecs (HTTP/1.1 200) 6 headers in 359 bytes (1 switches on core 0) Oct 13 04:11:45.889283 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8100cd86-3622-42d9-b402-29be144d6afc None admin] Authenticating user token {{(pid=81409) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:11:45.991718 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8100cd86-3622-42d9-b402-29be144d6afc None admin] Validating token access rules against request {{(pid=81409) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:11:46.030526 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-532fe2ca-b965-42bb-b0c0-475b8c0441ba None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81409) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:11:46.031241 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-532fe2ca-b965-42bb-b0c0-475b8c0441ba None admin] REQUEST_METHOD: `GET` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:11:46.031521 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-532fe2ca-b965-42bb-b0c0-475b8c0441ba None admin] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:11:46.031793 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-532fe2ca-b965-42bb-b0c0-475b8c0441ba None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:11:46.045711 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-532fe2ca-b965-42bb-b0c0-475b8c0441ba None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:11:46.047627 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-532fe2ca-b965-42bb-b0c0-475b8c0441ba None admin] RBAC: Authorization granted {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:11:46.063787 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application [None req-532fe2ca-b965-42bb-b0c0-475b8c0441ba None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 13 04:11:46.063787 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 13 04:11:46.063787 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 13 04:11:46.063787 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 13 04:11:46.063787 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 13 04:11:46.063787 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 13 04:11:46.063787 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 13 04:11:46.063787 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 13 04:11:46.063787 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 13 04:11:46.063787 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 13 04:11:46.063787 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 13 04:11:46.063787 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 13 04:11:46.063787 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 13 04:11:46.063787 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 13 04:11:46.063787 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 13 04:11:46.063787 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 13 04:11:46.064978 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 13 04:11:46.064978 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 13 04:11:46.064978 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 13 04:11:46.064978 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 13 04:11:46.064978 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 13 04:11:46.064978 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application return self.get_or_create( Oct 13 04:11:46.064978 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 13 04:11:46.064978 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application with Lock( Oct 13 04:11:46.064978 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 13 04:11:46.064978 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application return self._enter() Oct 13 04:11:46.064978 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 13 04:11:46.064978 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 13 04:11:46.064978 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 13 04:11:46.064978 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application return self.creator() Oct 13 04:11:46.064978 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 13 04:11:46.064978 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application created_value = creator( Oct 13 04:11:46.064978 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 13 04:11:46.066061 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 13 04:11:46.066061 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 13 04:11:46.066061 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application Oct 13 04:11:46.067244 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 491/960] 10.222.0.27 () {56 vars in 1108 bytes} [Thu Oct 13 04:11:45 2022] GET /identity/v3/domains/Default => generated 87 bytes in 178 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 13 04:11:46.073308 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5430326c-e822-417f-9f80-d7785cdba90e None admin] Authenticating user token {{(pid=81408) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:11:46.182772 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5430326c-e822-417f-9f80-d7785cdba90e None admin] Validating token access rules against request {{(pid=81408) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:11:46.197281 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG dbcounter [-] [81408] Writing DB stats keystone:SELECT=100 {{(pid=81408) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:11:46.219034 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-92e07cf4-3266-4568-9668-77a1622f7ef1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81408) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:11:46.219642 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92e07cf4-3266-4568-9668-77a1622f7ef1 None admin] REQUEST_METHOD: `GET` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:11:46.219945 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92e07cf4-3266-4568-9668-77a1622f7ef1 None admin] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:11:46.220217 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92e07cf4-3266-4568-9668-77a1622f7ef1 None admin] PATH_INFO: `/v3/domains` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:11:46.220854 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-92e07cf4-3266-4568-9668-77a1622f7ef1 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81408) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 13 04:11:46.221154 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-92e07cf4-3266-4568-9668-77a1622f7ef1 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:11:46.222547 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-92e07cf4-3266-4568-9668-77a1622f7ef1 None admin] RBAC: Authorization granted {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:11:46.234627 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 470/961] 10.222.0.27 () {56 vars in 1109 bytes} [Thu Oct 13 04:11:46 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 162 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 13 04:11:46.240317 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-532fe2ca-b965-42bb-b0c0-475b8c0441ba None admin] Authenticating user token {{(pid=81409) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:11:46.274950 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG dbcounter [-] [81409] Writing DB stats keystone:SELECT=100 {{(pid=81409) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:11:46.337851 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-532fe2ca-b965-42bb-b0c0-475b8c0441ba None admin] Validating token access rules against request {{(pid=81409) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:11:46.370087 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4de16195-0d35-4af9-ac4b-861c717195d7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81409) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:11:46.370716 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4de16195-0d35-4af9-ac4b-861c717195d7 None admin] REQUEST_METHOD: `GET` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:11:46.370991 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4de16195-0d35-4af9-ac4b-861c717195d7 None admin] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:11:46.371255 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4de16195-0d35-4af9-ac4b-861c717195d7 None admin] PATH_INFO: `/v3/projects/service` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:11:46.377515 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4de16195-0d35-4af9-ac4b-861c717195d7 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:11:46.378961 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4de16195-0d35-4af9-ac4b-861c717195d7 None admin] RBAC: Authorization granted {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:11:46.390629 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application [None req-4de16195-0d35-4af9-ac4b-861c717195d7 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Oct 13 04:11:46.390629 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 13 04:11:46.390629 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 13 04:11:46.390629 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 13 04:11:46.390629 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 13 04:11:46.390629 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 13 04:11:46.390629 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 13 04:11:46.390629 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 13 04:11:46.390629 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 13 04:11:46.390629 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 13 04:11:46.390629 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 13 04:11:46.390629 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 13 04:11:46.390629 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Oct 13 04:11:46.390629 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application return self._get_project(project_id) Oct 13 04:11:46.390629 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Oct 13 04:11:46.391665 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Oct 13 04:11:46.391665 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 13 04:11:46.391665 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 13 04:11:46.391665 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 13 04:11:46.391665 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 13 04:11:46.391665 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 13 04:11:46.391665 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application return self.get_or_create( Oct 13 04:11:46.391665 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 13 04:11:46.391665 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application with Lock( Oct 13 04:11:46.391665 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 13 04:11:46.391665 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application return self._enter() Oct 13 04:11:46.391665 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 13 04:11:46.391665 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 13 04:11:46.391665 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 13 04:11:46.391665 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application return self.creator() Oct 13 04:11:46.391665 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 13 04:11:46.391665 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application created_value = creator( Oct 13 04:11:46.392778 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Oct 13 04:11:46.392778 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Oct 13 04:11:46.392778 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Oct 13 04:11:46.392778 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Oct 13 04:11:46.392778 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Oct 13 04:11:46.392778 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Oct 13 04:11:46.392778 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Oct 13 04:11:46.392778 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: ERROR keystone.server.flask.application Oct 13 04:11:46.392778 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 492/962] 10.222.0.27 () {56 vars in 1111 bytes} [Thu Oct 13 04:11:46 2022] GET /identity/v3/projects/service => generated 88 bytes in 152 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 13 04:11:46.398160 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-92e07cf4-3266-4568-9668-77a1622f7ef1 None admin] Authenticating user token {{(pid=81408) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:11:46.495521 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-92e07cf4-3266-4568-9668-77a1622f7ef1 None admin] Validating token access rules against request {{(pid=81408) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:11:46.532069 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eaca8c7e-1ef8-4258-9375-5af654e2f32f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81408) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:11:46.532594 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eaca8c7e-1ef8-4258-9375-5af654e2f32f None admin] REQUEST_METHOD: `GET` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:11:46.532867 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eaca8c7e-1ef8-4258-9375-5af654e2f32f None admin] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:11:46.533158 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eaca8c7e-1ef8-4258-9375-5af654e2f32f None admin] PATH_INFO: `/v3/projects` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:11:46.534428 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eaca8c7e-1ef8-4258-9375-5af654e2f32f None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81408) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 13 04:11:46.534959 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eaca8c7e-1ef8-4258-9375-5af654e2f32f None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:11:46.538124 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eaca8c7e-1ef8-4258-9375-5af654e2f32f None admin] RBAC: Authorization granted {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:11:46.553342 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 471/963] 10.222.0.27 () {56 vars in 1148 bytes} [Thu Oct 13 04:11:46 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 156 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 13 04:11:46.559455 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4de16195-0d35-4af9-ac4b-861c717195d7 None admin] Authenticating user token {{(pid=81409) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:11:46.653074 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG dbcounter [-] [81409] Writing DB stats keystone:SELECT=100 {{(pid=81409) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:11:46.658882 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4de16195-0d35-4af9-ac4b-861c717195d7 None admin] Validating token access rules against request {{(pid=81409) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:11:46.692623 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9888546c-8b03-4b7a-8393-e8b22d3f0c64 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81409) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:11:46.693222 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9888546c-8b03-4b7a-8393-e8b22d3f0c64 None admin] REQUEST_METHOD: `GET` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:11:46.693496 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9888546c-8b03-4b7a-8393-e8b22d3f0c64 None admin] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:11:46.694319 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9888546c-8b03-4b7a-8393-e8b22d3f0c64 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:11:46.695046 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9888546c-8b03-4b7a-8393-e8b22d3f0c64 None admin] RBAC: Adding query filter params (role.id=ef55fbd94e1040be83695754075a7ed5, scope.project.id=dcec0431fb6849499d6aec35a234daa6, user.id=ba1b6e5868a549a99760ebe6068fd6b9) {{(pid=81409) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 13 04:11:46.695352 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9888546c-8b03-4b7a-8393-e8b22d3f0c64 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:11:46.696771 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9888546c-8b03-4b7a-8393-e8b22d3f0c64 None admin] RBAC: Authorization granted {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:11:46.702601 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 493/964] 10.222.0.27 () {56 vars in 1374 bytes} [Thu Oct 13 04:11:46 2022] GET /identity/v3/role_assignments?user.id=ba1b6e5868a549a99760ebe6068fd6b9&scope.project.id=dcec0431fb6849499d6aec35a234daa6&role.id=ef55fbd94e1040be83695754075a7ed5 => generated 608 bytes in 144 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 13 04:11:48.884977 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a92c6dc-1616-4dd8-8f5a-39c9bd00ade0 None None] REQUEST_METHOD: `GET` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:11:48.885504 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a92c6dc-1616-4dd8-8f5a-39c9bd00ade0 None None] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:11:48.885504 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a92c6dc-1616-4dd8-8f5a-39c9bd00ade0 None None] PATH_INFO: `/` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:11:48.886889 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 472/965] 10.222.0.27 () {54 vars in 927 bytes} [Thu Oct 13 04:11:48 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 13 04:11:48.892071 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9396cbd1-b75d-4965-b591-ecfbaf01a9c4 None None] REQUEST_METHOD: `POST` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:11:48.892468 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9396cbd1-b75d-4965-b591-ecfbaf01a9c4 None None] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:11:48.892584 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9396cbd1-b75d-4965-b591-ecfbaf01a9c4 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:11:48.910524 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.auth.core [None req-9396cbd1-b75d-4965-b591-ecfbaf01a9c4 None None] MFA Rules not processed for user `0e2d979563d6421995451a0e86bf3022`. Rule list: `[]` (Enabled: `True`). {{(pid=81409) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 13 04:11:48.912060 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.fernet_utils [None req-9396cbd1-b75d-4965-b591-ecfbaf01a9c4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81409) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:11:48.976618 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 494/966] 10.222.0.27 () {58 vars in 1024 bytes} [Thu Oct 13 04:11:48 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 85 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Oct 13 04:11:49.246673 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5a92c6dc-1616-4dd8-8f5a-39c9bd00ade0 None None] Authenticating user token {{(pid=81408) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:11:49.277603 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG dbcounter [-] [81408] Writing DB stats keystone:SELECT=100 {{(pid=81408) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:11:49.346040 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5a92c6dc-1616-4dd8-8f5a-39c9bd00ade0 None None] Validating token access rules against request {{(pid=81408) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:11:49.379285 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9b63689b-c527-4299-9a4e-e7faa485491b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81408) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:11:49.379921 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b63689b-c527-4299-9a4e-e7faa485491b None admin] REQUEST_METHOD: `GET` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:11:49.380197 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b63689b-c527-4299-9a4e-e7faa485491b None admin] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:11:49.380470 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b63689b-c527-4299-9a4e-e7faa485491b None admin] PATH_INFO: `/v3/services/placement` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:11:49.381049 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9b63689b-c527-4299-9a4e-e7faa485491b None admin] RBAC: Authorizing `identity:get_service(service_id=placement)` {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:11:49.382439 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9b63689b-c527-4299-9a4e-e7faa485491b None admin] RBAC: Authorization granted {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:11:49.391062 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application [None req-9b63689b-c527-4299-9a4e-e7faa485491b None admin] Could not find service: placement.: keystone.exception.ServiceNotFound: Could not find service: placement. Oct 13 04:11:49.391062 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 13 04:11:49.391062 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Oct 13 04:11:49.391062 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Oct 13 04:11:49.391062 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Oct 13 04:11:49.391062 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Oct 13 04:11:49.391062 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Oct 13 04:11:49.391062 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 13 04:11:49.391062 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: placement. Oct 13 04:11:49.391062 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application Oct 13 04:11:49.391062 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Oct 13 04:11:49.391062 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application Oct 13 04:11:49.391062 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 13 04:11:49.391062 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 13 04:11:49.391062 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 13 04:11:49.391062 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 13 04:11:49.391062 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 13 04:11:49.392124 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 13 04:11:49.392124 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 13 04:11:49.392124 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 13 04:11:49.392124 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 13 04:11:49.392124 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 13 04:11:49.392124 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 13 04:11:49.392124 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Oct 13 04:11:49.392124 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application return self._get_service(service_id) Oct 13 04:11:49.392124 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Oct 13 04:11:49.392124 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Oct 13 04:11:49.392124 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 13 04:11:49.392124 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 13 04:11:49.392124 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 13 04:11:49.392124 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 13 04:11:49.392124 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 13 04:11:49.392124 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application return self.get_or_create( Oct 13 04:11:49.393054 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 13 04:11:49.393054 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application with Lock( Oct 13 04:11:49.393054 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 13 04:11:49.393054 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application return self._enter() Oct 13 04:11:49.393054 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 13 04:11:49.393054 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 13 04:11:49.393054 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 13 04:11:49.393054 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application return self.creator() Oct 13 04:11:49.393054 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 13 04:11:49.393054 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application created_value = creator( Oct 13 04:11:49.393054 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Oct 13 04:11:49.393054 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 13 04:11:49.393054 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: placement. Oct 13 04:11:49.393054 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application Oct 13 04:11:49.393054 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 473/967] 10.222.0.27 () {56 vars in 1117 bytes} [Thu Oct 13 04:11:49 2022] GET /identity/v3/services/placement => generated 90 bytes in 146 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 13 04:11:49.398145 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9396cbd1-b75d-4965-b591-ecfbaf01a9c4 None None] Authenticating user token {{(pid=81409) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:11:49.487929 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG dbcounter [-] [81409] Writing DB stats keystone:SELECT=100 {{(pid=81409) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:11:49.498506 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9396cbd1-b75d-4965-b591-ecfbaf01a9c4 None None] Validating token access rules against request {{(pid=81409) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:11:49.531247 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-55ab04a4-ecb0-46a1-886a-d8d5b85fdbc0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81409) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:11:49.531843 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55ab04a4-ecb0-46a1-886a-d8d5b85fdbc0 None admin] REQUEST_METHOD: `GET` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:11:49.532117 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55ab04a4-ecb0-46a1-886a-d8d5b85fdbc0 None admin] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:11:49.532385 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55ab04a4-ecb0-46a1-886a-d8d5b85fdbc0 None admin] PATH_INFO: `/v3/services` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:11:49.533002 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-55ab04a4-ecb0-46a1-886a-d8d5b85fdbc0 None admin] RBAC: Adding query filter params (name=placement) {{(pid=81409) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 13 04:11:49.533302 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-55ab04a4-ecb0-46a1-886a-d8d5b85fdbc0 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:11:49.534687 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-55ab04a4-ecb0-46a1-886a-d8d5b85fdbc0 None admin] RBAC: Authorization granted {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:11:49.540434 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 495/968] 10.222.0.27 () {56 vars in 1116 bytes} [Thu Oct 13 04:11:49 2022] GET /identity/v3/services?name=placement => generated 126 bytes in 143 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 13 04:11:49.546327 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9b63689b-c527-4299-9a4e-e7faa485491b None admin] Authenticating user token {{(pid=81408) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:11:49.637727 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG dbcounter [-] [81408] Writing DB stats keystone:SELECT=100 {{(pid=81408) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:11:49.643932 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9b63689b-c527-4299-9a4e-e7faa485491b None admin] Validating token access rules against request {{(pid=81408) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:11:49.676778 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-19b70299-c6fd-4dee-bce2-df11dc292c91 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81408) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:11:49.677348 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19b70299-c6fd-4dee-bce2-df11dc292c91 None admin] REQUEST_METHOD: `GET` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:11:49.677619 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19b70299-c6fd-4dee-bce2-df11dc292c91 None admin] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:11:49.677884 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19b70299-c6fd-4dee-bce2-df11dc292c91 None admin] PATH_INFO: `/v3/services` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:11:49.678527 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-19b70299-c6fd-4dee-bce2-df11dc292c91 None admin] RBAC: Adding query filter params (type=placement) {{(pid=81408) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 13 04:11:49.678824 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-19b70299-c6fd-4dee-bce2-df11dc292c91 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:11:49.680171 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-19b70299-c6fd-4dee-bce2-df11dc292c91 None admin] RBAC: Authorization granted {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:11:49.684375 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 474/969] 10.222.0.27 () {56 vars in 1116 bytes} [Thu Oct 13 04:11:49 2022] GET /identity/v3/services?type=placement => generated 126 bytes in 139 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 13 04:11:51.799856 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7dcb252-c2cb-47c3-a3f2-484df793ceb3 None None] REQUEST_METHOD: `GET` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:11:51.800389 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7dcb252-c2cb-47c3-a3f2-484df793ceb3 None None] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:11:51.800389 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7dcb252-c2cb-47c3-a3f2-484df793ceb3 None None] PATH_INFO: `/` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:11:51.802079 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 496/970] 10.222.0.27 () {54 vars in 927 bytes} [Thu Oct 13 04:11:51 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Oct 13 04:11:51.807244 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7dd5f08c-605e-4ff4-b177-aaaa78c71c96 None None] REQUEST_METHOD: `POST` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:11:51.807647 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7dd5f08c-605e-4ff4-b177-aaaa78c71c96 None None] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:11:51.807783 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7dd5f08c-605e-4ff4-b177-aaaa78c71c96 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:11:51.826030 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.auth.core [None req-7dd5f08c-605e-4ff4-b177-aaaa78c71c96 None None] MFA Rules not processed for user `0e2d979563d6421995451a0e86bf3022`. Rule list: `[]` (Enabled: `True`). {{(pid=81408) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 13 04:11:51.827250 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.fernet_utils [None req-7dd5f08c-605e-4ff4-b177-aaaa78c71c96 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81408) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:11:51.893572 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 475/971] 10.222.0.27 () {58 vars in 1024 bytes} [Thu Oct 13 04:11:51 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 87 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Oct 13 04:11:52.166335 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e7dcb252-c2cb-47c3-a3f2-484df793ceb3 None None] Authenticating user token {{(pid=81409) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:11:52.265251 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e7dcb252-c2cb-47c3-a3f2-484df793ceb3 None None] Validating token access rules against request {{(pid=81409) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:11:52.298754 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c19043e6-1c4f-4cc9-a4a8-14d48ef941ce None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81409) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:11:52.299346 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c19043e6-1c4f-4cc9-a4a8-14d48ef941ce None admin] REQUEST_METHOD: `POST` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:11:52.299624 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c19043e6-1c4f-4cc9-a4a8-14d48ef941ce None admin] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:11:52.299895 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c19043e6-1c4f-4cc9-a4a8-14d48ef941ce None admin] PATH_INFO: `/v3/services` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:11:52.300535 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c19043e6-1c4f-4cc9-a4a8-14d48ef941ce None admin] RBAC: Authorizing `identity:create_service()` {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:11:52.302015 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c19043e6-1c4f-4cc9-a4a8-14d48ef941ce None admin] RBAC: Authorization granted {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:11:52.315682 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 497/972] 10.222.0.27 () {60 vars in 1141 bytes} [Thu Oct 13 04:11:52 2022] POST /identity/v3/services => generated 246 bytes in 150 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 13 04:11:54.493744 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4881aaf2-46c0-402a-933c-8c50a470216e None None] REQUEST_METHOD: `GET` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:11:54.494366 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4881aaf2-46c0-402a-933c-8c50a470216e None None] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:11:54.494366 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4881aaf2-46c0-402a-933c-8c50a470216e None None] PATH_INFO: `/` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:11:54.495687 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 476/973] 10.222.0.27 () {54 vars in 927 bytes} [Thu Oct 13 04:11:54 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 13 04:11:54.500985 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c985d175-dc58-42ac-89f3-86bcf21c09dc None None] REQUEST_METHOD: `POST` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:11:54.501400 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c985d175-dc58-42ac-89f3-86bcf21c09dc None None] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:11:54.501522 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c985d175-dc58-42ac-89f3-86bcf21c09dc None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:11:54.519890 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.auth.core [None req-c985d175-dc58-42ac-89f3-86bcf21c09dc None None] MFA Rules not processed for user `0e2d979563d6421995451a0e86bf3022`. Rule list: `[]` (Enabled: `True`). {{(pid=81409) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 13 04:11:54.521372 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.fernet_utils [None req-c985d175-dc58-42ac-89f3-86bcf21c09dc None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81409) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:11:54.530335 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG dbcounter [-] [81409] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81409) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:11:54.592696 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 498/974] 10.222.0.27 () {58 vars in 1024 bytes} [Thu Oct 13 04:11:54 2022] POST /identity/v3/auth/tokens => generated 2221 bytes in 93 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Oct 13 04:11:54.867962 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4881aaf2-46c0-402a-933c-8c50a470216e None None] Authenticating user token {{(pid=81408) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:11:54.954342 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG dbcounter [-] [81408] Writing DB stats keystone:SELECT=100 {{(pid=81408) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:11:54.965455 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4881aaf2-46c0-402a-933c-8c50a470216e None None] Validating token access rules against request {{(pid=81408) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:11:54.998326 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3e07b9db-063b-4263-ab38-9c1c59a35e53 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81408) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:11:54.998935 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e07b9db-063b-4263-ab38-9c1c59a35e53 None admin] REQUEST_METHOD: `GET` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:11:54.999210 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e07b9db-063b-4263-ab38-9c1c59a35e53 None admin] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:11:54.999476 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e07b9db-063b-4263-ab38-9c1c59a35e53 None admin] PATH_INFO: `/v3/services/placement` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:11:55.000050 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3e07b9db-063b-4263-ab38-9c1c59a35e53 None admin] RBAC: Authorizing `identity:get_service(service_id=placement)` {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:11:55.001408 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3e07b9db-063b-4263-ab38-9c1c59a35e53 None admin] RBAC: Authorization granted {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:11:55.009885 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application [None req-3e07b9db-063b-4263-ab38-9c1c59a35e53 None admin] Could not find service: placement.: keystone.exception.ServiceNotFound: Could not find service: placement. Oct 13 04:11:55.009885 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 13 04:11:55.009885 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Oct 13 04:11:55.009885 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Oct 13 04:11:55.009885 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Oct 13 04:11:55.009885 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Oct 13 04:11:55.009885 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Oct 13 04:11:55.009885 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 13 04:11:55.009885 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: placement. Oct 13 04:11:55.009885 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application Oct 13 04:11:55.009885 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Oct 13 04:11:55.009885 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application Oct 13 04:11:55.009885 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 13 04:11:55.009885 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 13 04:11:55.009885 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 13 04:11:55.009885 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 13 04:11:55.009885 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 13 04:11:55.011157 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 13 04:11:55.011157 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 13 04:11:55.011157 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 13 04:11:55.011157 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 13 04:11:55.011157 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 13 04:11:55.011157 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 13 04:11:55.011157 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Oct 13 04:11:55.011157 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application return self._get_service(service_id) Oct 13 04:11:55.011157 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Oct 13 04:11:55.011157 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Oct 13 04:11:55.011157 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 13 04:11:55.011157 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 13 04:11:55.011157 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 13 04:11:55.011157 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 13 04:11:55.011157 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 13 04:11:55.011157 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application return self.get_or_create( Oct 13 04:11:55.012272 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 13 04:11:55.012272 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application with Lock( Oct 13 04:11:55.012272 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 13 04:11:55.012272 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application return self._enter() Oct 13 04:11:55.012272 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 13 04:11:55.012272 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 13 04:11:55.012272 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 13 04:11:55.012272 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application return self.creator() Oct 13 04:11:55.012272 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 13 04:11:55.012272 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application created_value = creator( Oct 13 04:11:55.012272 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Oct 13 04:11:55.012272 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 13 04:11:55.012272 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: placement. Oct 13 04:11:55.012272 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application Oct 13 04:11:55.012272 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 477/975] 10.222.0.27 () {56 vars in 1117 bytes} [Thu Oct 13 04:11:54 2022] GET /identity/v3/services/placement => generated 90 bytes in 144 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 13 04:11:55.017089 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c985d175-dc58-42ac-89f3-86bcf21c09dc None None] Authenticating user token {{(pid=81409) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:11:55.121271 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c985d175-dc58-42ac-89f3-86bcf21c09dc None None] Validating token access rules against request {{(pid=81409) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:11:55.153972 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2ce39a24-a2bf-4807-b3ca-1edec724b894 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81409) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:11:55.154590 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ce39a24-a2bf-4807-b3ca-1edec724b894 None admin] REQUEST_METHOD: `GET` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:11:55.154862 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ce39a24-a2bf-4807-b3ca-1edec724b894 None admin] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:11:55.155127 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ce39a24-a2bf-4807-b3ca-1edec724b894 None admin] PATH_INFO: `/v3/services` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:11:55.155737 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2ce39a24-a2bf-4807-b3ca-1edec724b894 None admin] RBAC: Adding query filter params (name=placement) {{(pid=81409) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 13 04:11:55.156032 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2ce39a24-a2bf-4807-b3ca-1edec724b894 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:11:55.157380 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2ce39a24-a2bf-4807-b3ca-1edec724b894 None admin] RBAC: Authorization granted {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:11:55.163176 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 499/976] 10.222.0.27 () {56 vars in 1116 bytes} [Thu Oct 13 04:11:55 2022] GET /identity/v3/services?name=placement => generated 358 bytes in 147 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 13 04:11:55.169191 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3e07b9db-063b-4263-ab38-9c1c59a35e53 None admin] Authenticating user token {{(pid=81408) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:11:55.265971 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3e07b9db-063b-4263-ab38-9c1c59a35e53 None admin] Validating token access rules against request {{(pid=81408) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:11:55.298405 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-57a240f3-8b1a-47ea-bb42-508a7a9e3948 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81408) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:11:55.298984 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57a240f3-8b1a-47ea-bb42-508a7a9e3948 None admin] REQUEST_METHOD: `GET` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:11:55.299255 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57a240f3-8b1a-47ea-bb42-508a7a9e3948 None admin] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:11:55.299522 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57a240f3-8b1a-47ea-bb42-508a7a9e3948 None admin] PATH_INFO: `/v3/endpoints` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:11:55.300207 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-57a240f3-8b1a-47ea-bb42-508a7a9e3948 None admin] RBAC: Adding query filter params (interface=public, service_id=e6d3bd2a2ac4498eb84ed5f4cc028482, region_id=RegionOne) {{(pid=81408) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 13 04:11:55.300506 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-57a240f3-8b1a-47ea-bb42-508a7a9e3948 None admin] RBAC: Authorizing `identity:list_endpoints()` {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:11:55.301990 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-57a240f3-8b1a-47ea-bb42-508a7a9e3948 None admin] RBAC: Authorization granted {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:11:55.306766 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 478/977] 10.222.0.27 () {56 vars in 1251 bytes} [Thu Oct 13 04:11:55 2022] GET /identity/v3/endpoints?service_id=e6d3bd2a2ac4498eb84ed5f4cc028482&interface=public®ion_id=RegionOne => generated 194 bytes in 138 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 13 04:11:55.312358 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2ce39a24-a2bf-4807-b3ca-1edec724b894 None admin] Authenticating user token {{(pid=81409) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:11:55.316027 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG dbcounter [-] [81409] Writing DB stats keystone:SELECT=100 {{(pid=81409) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:11:55.410157 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2ce39a24-a2bf-4807-b3ca-1edec724b894 None admin] Validating token access rules against request {{(pid=81409) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:11:55.444172 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c026ff75-451c-4282-8dd9-563f0494e08c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81409) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:11:55.444760 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c026ff75-451c-4282-8dd9-563f0494e08c None admin] REQUEST_METHOD: `GET` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:11:55.445034 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c026ff75-451c-4282-8dd9-563f0494e08c None admin] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:11:55.445298 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c026ff75-451c-4282-8dd9-563f0494e08c None admin] PATH_INFO: `/v3/services` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:11:55.445888 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c026ff75-451c-4282-8dd9-563f0494e08c None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:11:55.447261 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c026ff75-451c-4282-8dd9-563f0494e08c None admin] RBAC: Authorization granted {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:11:55.453827 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 500/978] 10.222.0.27 () {56 vars in 1087 bytes} [Thu Oct 13 04:11:55 2022] GET /identity/v3/services => generated 2191 bytes in 142 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 13 04:11:57.595465 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e264dd6f-4738-438b-8ea7-ddb85bac85d9 None None] REQUEST_METHOD: `GET` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:11:57.596025 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e264dd6f-4738-438b-8ea7-ddb85bac85d9 None None] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:11:57.596025 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e264dd6f-4738-438b-8ea7-ddb85bac85d9 None None] PATH_INFO: `/` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:11:57.597330 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 479/979] 10.222.0.27 () {54 vars in 927 bytes} [Thu Oct 13 04:11:57 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 13 04:11:57.602990 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0d4a0d0-a000-4987-b128-54287daf0316 None None] REQUEST_METHOD: `POST` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:11:57.603376 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0d4a0d0-a000-4987-b128-54287daf0316 None None] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:11:57.603516 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0d4a0d0-a000-4987-b128-54287daf0316 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:11:57.621317 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.auth.core [None req-d0d4a0d0-a000-4987-b128-54287daf0316 None None] MFA Rules not processed for user `0e2d979563d6421995451a0e86bf3022`. Rule list: `[]` (Enabled: `True`). {{(pid=81409) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 13 04:11:57.622514 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.fernet_utils [None req-d0d4a0d0-a000-4987-b128-54287daf0316 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81409) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:11:57.683537 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG dbcounter [-] [81409] Writing DB stats keystone:SELECT=100 {{(pid=81409) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:11:57.688967 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 501/980] 10.222.0.27 () {58 vars in 1024 bytes} [Thu Oct 13 04:11:57 2022] POST /identity/v3/auth/tokens => generated 2221 bytes in 87 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Oct 13 04:11:57.957303 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e264dd6f-4738-438b-8ea7-ddb85bac85d9 None None] Authenticating user token {{(pid=81408) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:11:57.977623 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG dbcounter [-] [81408] Writing DB stats keystone:SELECT=100 {{(pid=81408) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:11:58.075952 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e264dd6f-4738-438b-8ea7-ddb85bac85d9 None None] Validating token access rules against request {{(pid=81408) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:11:58.110806 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ec5e2ce4-fbf5-40ee-8051-aecc659bdcec None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81408) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:11:58.111426 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec5e2ce4-fbf5-40ee-8051-aecc659bdcec None admin] REQUEST_METHOD: `GET` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:11:58.111703 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec5e2ce4-fbf5-40ee-8051-aecc659bdcec None admin] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:11:58.111974 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec5e2ce4-fbf5-40ee-8051-aecc659bdcec None admin] PATH_INFO: `/v3/services/placement` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:11:58.112549 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ec5e2ce4-fbf5-40ee-8051-aecc659bdcec None admin] RBAC: Authorizing `identity:get_service(service_id=placement)` {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:11:58.113974 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ec5e2ce4-fbf5-40ee-8051-aecc659bdcec None admin] RBAC: Authorization granted {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:11:58.122696 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application [None req-ec5e2ce4-fbf5-40ee-8051-aecc659bdcec None admin] Could not find service: placement.: keystone.exception.ServiceNotFound: Could not find service: placement. Oct 13 04:11:58.122696 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 13 04:11:58.122696 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Oct 13 04:11:58.122696 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Oct 13 04:11:58.122696 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Oct 13 04:11:58.122696 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Oct 13 04:11:58.122696 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Oct 13 04:11:58.122696 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 13 04:11:58.122696 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: placement. Oct 13 04:11:58.122696 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application Oct 13 04:11:58.122696 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Oct 13 04:11:58.122696 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application Oct 13 04:11:58.122696 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 13 04:11:58.122696 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 13 04:11:58.122696 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 13 04:11:58.122696 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 13 04:11:58.122696 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 13 04:11:58.123872 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 13 04:11:58.123872 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 13 04:11:58.123872 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 13 04:11:58.123872 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 13 04:11:58.123872 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 13 04:11:58.123872 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 13 04:11:58.123872 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Oct 13 04:11:58.123872 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application return self._get_service(service_id) Oct 13 04:11:58.123872 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Oct 13 04:11:58.123872 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Oct 13 04:11:58.123872 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 13 04:11:58.123872 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 13 04:11:58.123872 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 13 04:11:58.123872 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 13 04:11:58.123872 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 13 04:11:58.123872 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application return self.get_or_create( Oct 13 04:11:58.124909 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 13 04:11:58.124909 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application with Lock( Oct 13 04:11:58.124909 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 13 04:11:58.124909 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application return self._enter() Oct 13 04:11:58.124909 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 13 04:11:58.124909 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 13 04:11:58.124909 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 13 04:11:58.124909 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application return self.creator() Oct 13 04:11:58.124909 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 13 04:11:58.124909 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application created_value = creator( Oct 13 04:11:58.124909 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Oct 13 04:11:58.124909 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 13 04:11:58.124909 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: placement. Oct 13 04:11:58.124909 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: ERROR keystone.server.flask.application Oct 13 04:11:58.124909 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 480/981] 10.222.0.27 () {56 vars in 1117 bytes} [Thu Oct 13 04:11:57 2022] GET /identity/v3/services/placement => generated 90 bytes in 167 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 13 04:11:58.129862 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d0d4a0d0-a000-4987-b128-54287daf0316 None None] Authenticating user token {{(pid=81409) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:11:58.226532 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d0d4a0d0-a000-4987-b128-54287daf0316 None None] Validating token access rules against request {{(pid=81409) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:11:58.259539 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e8bbdbdb-7412-4ea2-ac52-d186a7c186d3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81409) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:11:58.260132 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8bbdbdb-7412-4ea2-ac52-d186a7c186d3 None admin] REQUEST_METHOD: `GET` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:11:58.260406 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8bbdbdb-7412-4ea2-ac52-d186a7c186d3 None admin] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:11:58.260676 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8bbdbdb-7412-4ea2-ac52-d186a7c186d3 None admin] PATH_INFO: `/v3/services` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:11:58.261296 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e8bbdbdb-7412-4ea2-ac52-d186a7c186d3 None admin] RBAC: Adding query filter params (name=placement) {{(pid=81409) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 13 04:11:58.261593 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e8bbdbdb-7412-4ea2-ac52-d186a7c186d3 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:11:58.262984 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e8bbdbdb-7412-4ea2-ac52-d186a7c186d3 None admin] RBAC: Authorization granted {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:11:58.267923 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 502/982] 10.222.0.27 () {56 vars in 1116 bytes} [Thu Oct 13 04:11:58 2022] GET /identity/v3/services?name=placement => generated 358 bytes in 139 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 13 04:11:58.273913 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ec5e2ce4-fbf5-40ee-8051-aecc659bdcec None admin] Authenticating user token {{(pid=81408) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:11:58.354308 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG dbcounter [-] [81408] Writing DB stats keystone:SELECT=100 {{(pid=81408) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:11:58.372496 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ec5e2ce4-fbf5-40ee-8051-aecc659bdcec None admin] Validating token access rules against request {{(pid=81408) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:11:58.404519 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2afcb998-c025-4c17-9fd6-686cbb719074 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81408) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:11:58.405107 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2afcb998-c025-4c17-9fd6-686cbb719074 None admin] REQUEST_METHOD: `POST` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:11:58.405381 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2afcb998-c025-4c17-9fd6-686cbb719074 None admin] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:11:58.405649 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2afcb998-c025-4c17-9fd6-686cbb719074 None admin] PATH_INFO: `/v3/endpoints` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:11:58.406352 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2afcb998-c025-4c17-9fd6-686cbb719074 None admin] RBAC: Authorizing `identity:create_endpoint()` {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:11:58.407723 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2afcb998-c025-4c17-9fd6-686cbb719074 None admin] RBAC: Authorization granted {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:11:58.427516 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 481/983] 10.222.0.27 () {60 vars in 1144 bytes} [Thu Oct 13 04:11:58 2022] POST /identity/v3/endpoints => generated 331 bytes in 154 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 13 04:12:08.265054 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG dbcounter [-] [81409] Writing DB stats keystone:SELECT=69 {{(pid=81409) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:12:08.423202 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG dbcounter [-] [81408] Writing DB stats keystone:SELECT=30,keystone:INSERT=1 {{(pid=81408) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:13:02.052995 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd6d0b10-3a6c-42f3-84ae-ceaecd5fdf0e None None] REQUEST_METHOD: `GET` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:13:02.053450 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd6d0b10-3a6c-42f3-84ae-ceaecd5fdf0e None None] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:13:02.053450 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd6d0b10-3a6c-42f3-84ae-ceaecd5fdf0e None None] PATH_INFO: `/` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:13:02.060097 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8883f4a-6c4b-427e-a4e5-a4b9f652204b None None] REQUEST_METHOD: `POST` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:13:02.060501 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8883f4a-6c4b-427e-a4e5-a4b9f652204b None None] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:13:02.060629 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8883f4a-6c4b-427e-a4e5-a4b9f652204b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:13:02.063581 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 503/984] 10.222.0.27 () {54 vars in 927 bytes} [Thu Oct 13 04:13:02 2022] GET /identity => generated 270 bytes in 11 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 13 04:13:02.158834 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.auth.core [None req-a8883f4a-6c4b-427e-a4e5-a4b9f652204b None None] MFA Rules not processed for user `0e2d979563d6421995451a0e86bf3022`. Rule list: `[]` (Enabled: `True`). {{(pid=81408) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 13 04:13:02.160606 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.fernet_utils [None req-a8883f4a-6c4b-427e-a4e5-a4b9f652204b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81408) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:13:02.375196 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 482/985] 10.222.0.27 () {58 vars in 1024 bytes} [Thu Oct 13 04:13:02 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 316 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 13 04:13:02.704096 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e7ce170-ea29-466e-80f7-89c9f06e97c0 None None] REQUEST_METHOD: `GET` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:13:02.704748 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e7ce170-ea29-466e-80f7-89c9f06e97c0 None None] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:13:02.705684 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e7ce170-ea29-466e-80f7-89c9f06e97c0 None None] PATH_INFO: `/` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:13:02.708148 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 504/986] 10.222.0.27 () {54 vars in 964 bytes} [Thu Oct 13 04:13:02 2022] GET /identity => generated 270 bytes in 6 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 13 04:13:02.714014 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66b762f3-e945-486d-8e78-d43a0c097d10 None None] REQUEST_METHOD: `POST` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:13:02.714726 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66b762f3-e945-486d-8e78-d43a0c097d10 None None] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:13:02.715304 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66b762f3-e945-486d-8e78-d43a0c097d10 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:13:02.808852 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.auth.core [None req-66b762f3-e945-486d-8e78-d43a0c097d10 None None] MFA Rules not processed for user `d8812c3974f84871bb2d761cf0016257`. Rule list: `[]` (Enabled: `True`). {{(pid=81408) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 13 04:13:02.810395 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.fernet_utils [None req-66b762f3-e945-486d-8e78-d43a0c097d10 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81408) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:13:02.837643 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG dbcounter [-] [81408] Writing DB stats keystone:SELECT=100 {{(pid=81408) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:13:02.915753 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 483/987] 10.222.0.27 () {58 vars in 1061 bytes} [Thu Oct 13 04:13:02 2022] POST /identity/v3/auth/tokens => generated 3121 bytes in 203 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 13 04:13:02.916460 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e7ce170-ea29-466e-80f7-89c9f06e97c0 None None] Authenticating user token {{(pid=81409) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:13:02.918256 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.fernet_utils [None req-1e7ce170-ea29-466e-80f7-89c9f06e97c0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81409) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:13:03.049021 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e7ce170-ea29-466e-80f7-89c9f06e97c0 None None] Validating token access rules against request {{(pid=81409) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:13:03.140277 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-45d5c806-bdb7-470e-817a-3c634a418d41 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd8812c3974f84871bb2d761cf0016257', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dcec0431fb6849499d6aec35a234daa6', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81409) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:13:03.141590 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45d5c806-bdb7-470e-817a-3c634a418d41 service neutron] REQUEST_METHOD: `GET` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:13:03.142061 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45d5c806-bdb7-470e-817a-3c634a418d41 service neutron] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:13:03.143155 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45d5c806-bdb7-470e-817a-3c634a418d41 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:13:03.145077 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.fernet_utils [None req-45d5c806-bdb7-470e-817a-3c634a418d41 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81409) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:13:03.213948 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG dbcounter [-] [81409] Writing DB stats keystone:SELECT=100 {{(pid=81409) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:13:03.287137 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-45d5c806-bdb7-470e-817a-3c634a418d41 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:13:03.288691 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-45d5c806-bdb7-470e-817a-3c634a418d41 service neutron] RBAC: Authorization granted {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:13:03.311886 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 505/988] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Oct 13 04:13:02 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 396 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 13 04:13:05.787770 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cee443a1-441f-4ab4-b4a2-e267acccf639 None None] REQUEST_METHOD: `GET` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:13:05.788329 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cee443a1-441f-4ab4-b4a2-e267acccf639 None None] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:13:05.788329 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cee443a1-441f-4ab4-b4a2-e267acccf639 None None] PATH_INFO: `/` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:13:05.789676 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 484/989] 10.222.0.27 () {54 vars in 927 bytes} [Thu Oct 13 04:13:05 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 13 04:13:05.794947 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6189f78-2b49-463c-bb26-59f905efeee2 None None] REQUEST_METHOD: `POST` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:13:05.795347 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6189f78-2b49-463c-bb26-59f905efeee2 None None] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:13:05.795607 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6189f78-2b49-463c-bb26-59f905efeee2 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:13:05.881663 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.auth.core [None req-a6189f78-2b49-463c-bb26-59f905efeee2 None None] MFA Rules not processed for user `0e2d979563d6421995451a0e86bf3022`. Rule list: `[]` (Enabled: `True`). {{(pid=81409) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 13 04:13:05.882821 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.fernet_utils [None req-a6189f78-2b49-463c-bb26-59f905efeee2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81409) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:13:05.981852 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 506/990] 10.222.0.27 () {58 vars in 1024 bytes} [Thu Oct 13 04:13:05 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 188 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 13 04:13:06.269485 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cee443a1-441f-4ab4-b4a2-e267acccf639 None None] Authenticating user token {{(pid=81408) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:13:06.271067 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.fernet_utils [None req-cee443a1-441f-4ab4-b4a2-e267acccf639 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81408) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:13:06.373594 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cee443a1-441f-4ab4-b4a2-e267acccf639 None None] Validating token access rules against request {{(pid=81408) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:13:06.445290 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG dbcounter [-] [81408] Writing DB stats keystone:SELECT=100 {{(pid=81408) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:13:06.447994 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cc7a3f89-7148-40b3-84ee-7d32827c8906 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd8812c3974f84871bb2d761cf0016257', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dcec0431fb6849499d6aec35a234daa6', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81408) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:13:06.448820 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc7a3f89-7148-40b3-84ee-7d32827c8906 service neutron] REQUEST_METHOD: `GET` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:13:06.449306 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc7a3f89-7148-40b3-84ee-7d32827c8906 service neutron] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:13:06.449775 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc7a3f89-7148-40b3-84ee-7d32827c8906 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:13:06.451448 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.fernet_utils [None req-cc7a3f89-7148-40b3-84ee-7d32827c8906 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81408) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:13:06.560925 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cc7a3f89-7148-40b3-84ee-7d32827c8906 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:13:06.562299 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cc7a3f89-7148-40b3-84ee-7d32827c8906 service neutron] RBAC: Authorization granted {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:13:06.581364 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 485/991] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Oct 13 04:13:06 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 313 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Oct 13 04:13:09.041693 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-889d6e2e-c773-4c07-96fd-e20660a31538 None None] REQUEST_METHOD: `GET` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:13:09.041693 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-889d6e2e-c773-4c07-96fd-e20660a31538 None None] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:13:09.041693 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-889d6e2e-c773-4c07-96fd-e20660a31538 None None] PATH_INFO: `/` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:13:09.041693 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 507/992] 10.222.0.27 () {54 vars in 927 bytes} [Thu Oct 13 04:13:09 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 13 04:13:09.046984 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9eae5ee9-9c38-42f9-aba3-65051514aa8e None None] REQUEST_METHOD: `POST` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:13:09.047396 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9eae5ee9-9c38-42f9-aba3-65051514aa8e None None] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:13:09.047596 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9eae5ee9-9c38-42f9-aba3-65051514aa8e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:13:09.113894 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.auth.core [None req-9eae5ee9-9c38-42f9-aba3-65051514aa8e None None] MFA Rules not processed for user `761e23aab9234a7c9c88dc41d2ddbe8e`. Rule list: `[]` (Enabled: `True`). {{(pid=81408) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 13 04:13:09.115136 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.fernet_utils [None req-9eae5ee9-9c38-42f9-aba3-65051514aa8e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81408) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:13:09.165779 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG dbcounter [-] [81408] Writing DB stats keystone:SELECT=100 {{(pid=81408) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:13:09.210698 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 486/993] 10.222.0.27 () {58 vars in 1024 bytes} [Thu Oct 13 04:13:09 2022] POST /identity/v3/auth/tokens => generated 3243 bytes in 165 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 13 04:13:09.495933 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-889d6e2e-c773-4c07-96fd-e20660a31538 None None] Authenticating user token {{(pid=81409) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:13:09.497461 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.fernet_utils [None req-889d6e2e-c773-4c07-96fd-e20660a31538 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81409) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:13:09.507775 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG dbcounter [-] [81409] Writing DB stats keystone:SELECT=100 {{(pid=81409) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:13:09.600868 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-889d6e2e-c773-4c07-96fd-e20660a31538 None None] Validating token access rules against request {{(pid=81409) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:13:09.673514 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0de972f8-bed8-4279-b956-abd3f0bb4a18 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd8812c3974f84871bb2d761cf0016257', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dcec0431fb6849499d6aec35a234daa6', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81409) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:13:09.674042 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0de972f8-bed8-4279-b956-abd3f0bb4a18 service neutron] REQUEST_METHOD: `GET` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:13:09.674345 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0de972f8-bed8-4279-b956-abd3f0bb4a18 service neutron] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:13:09.674618 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0de972f8-bed8-4279-b956-abd3f0bb4a18 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:13:09.676210 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.fernet_utils [None req-0de972f8-bed8-4279-b956-abd3f0bb4a18 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81409) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:13:09.738414 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG dbcounter [-] [81409] Writing DB stats keystone:SELECT=100 {{(pid=81409) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:13:09.785148 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0de972f8-bed8-4279-b956-abd3f0bb4a18 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:13:09.786612 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0de972f8-bed8-4279-b956-abd3f0bb4a18 service neutron] RBAC: Authorization granted {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:13:09.806076 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 508/994] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Oct 13 04:13:09 2022] GET /identity/v3/auth/tokens => generated 3243 bytes in 311 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 13 04:13:12.462874 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-537a0291-f614-4b14-9ba2-93925fd150e7 None None] REQUEST_METHOD: `GET` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:13:12.463358 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-537a0291-f614-4b14-9ba2-93925fd150e7 None None] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:13:12.463473 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-537a0291-f614-4b14-9ba2-93925fd150e7 None None] PATH_INFO: `/` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:13:12.464743 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 487/995] 10.222.0.27 () {54 vars in 927 bytes} [Thu Oct 13 04:13:12 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 13 04:13:12.470004 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f8b85b4-fdb1-45d7-a666-8c4bea3d9c37 None None] REQUEST_METHOD: `POST` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:13:12.470629 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f8b85b4-fdb1-45d7-a666-8c4bea3d9c37 None None] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:13:12.470742 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f8b85b4-fdb1-45d7-a666-8c4bea3d9c37 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:13:12.536779 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.auth.core [None req-4f8b85b4-fdb1-45d7-a666-8c4bea3d9c37 None None] MFA Rules not processed for user `0e2d979563d6421995451a0e86bf3022`. Rule list: `[]` (Enabled: `True`). {{(pid=81409) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 13 04:13:12.537982 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.fernet_utils [None req-4f8b85b4-fdb1-45d7-a666-8c4bea3d9c37 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81409) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:13:12.633285 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 509/996] 10.222.0.27 () {58 vars in 1024 bytes} [Thu Oct 13 04:13:12 2022] POST /identity/v3/auth/tokens => generated 3238 bytes in 164 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 13 04:13:12.915590 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-537a0291-f614-4b14-9ba2-93925fd150e7 None None] Authenticating user token {{(pid=81408) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:13:12.917117 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.fernet_utils [None req-537a0291-f614-4b14-9ba2-93925fd150e7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81408) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:13:13.018334 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-537a0291-f614-4b14-9ba2-93925fd150e7 None None] Validating token access rules against request {{(pid=81408) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:13:13.091000 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-39dc1a20-2b38-4be5-a572-981e882de0c9 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd8812c3974f84871bb2d761cf0016257', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dcec0431fb6849499d6aec35a234daa6', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81408) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:13:13.091625 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39dc1a20-2b38-4be5-a572-981e882de0c9 service neutron] REQUEST_METHOD: `GET` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:13:13.091897 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39dc1a20-2b38-4be5-a572-981e882de0c9 service neutron] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:13:13.092162 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39dc1a20-2b38-4be5-a572-981e882de0c9 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:13:13.093795 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.fernet_utils [None req-39dc1a20-2b38-4be5-a572-981e882de0c9 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81408) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:13:13.100622 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG dbcounter [-] [81408] Writing DB stats keystone:SELECT=100 {{(pid=81408) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:13:13.202317 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-39dc1a20-2b38-4be5-a572-981e882de0c9 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:13:13.203657 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-39dc1a20-2b38-4be5-a572-981e882de0c9 service neutron] RBAC: Authorization granted {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:13:13.221907 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 488/997] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Oct 13 04:13:12 2022] GET /identity/v3/auth/tokens => generated 3238 bytes in 307 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 13 04:13:16.014760 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c556e77b-e903-43e8-b41a-783938620b7b None None] REQUEST_METHOD: `GET` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:13:16.015310 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c556e77b-e903-43e8-b41a-783938620b7b None None] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:13:16.015310 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c556e77b-e903-43e8-b41a-783938620b7b None None] PATH_INFO: `/` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:13:16.016792 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 510/998] 10.222.0.27 () {54 vars in 927 bytes} [Thu Oct 13 04:13:16 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 13 04:13:16.021996 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-172648fe-eaf0-4040-978d-af3764e062e8 None None] REQUEST_METHOD: `POST` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:13:16.022412 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-172648fe-eaf0-4040-978d-af3764e062e8 None None] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:13:16.022695 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-172648fe-eaf0-4040-978d-af3764e062e8 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:13:16.095561 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.auth.core [None req-172648fe-eaf0-4040-978d-af3764e062e8 None None] MFA Rules not processed for user `0e2d979563d6421995451a0e86bf3022`. Rule list: `[]` (Enabled: `True`). {{(pid=81408) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 13 04:13:16.096826 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.fernet_utils [None req-172648fe-eaf0-4040-978d-af3764e062e8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81408) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:13:16.161639 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG dbcounter [-] [81408] Writing DB stats keystone:SELECT=100 {{(pid=81408) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:13:16.205387 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 489/999] 10.222.0.27 () {58 vars in 1024 bytes} [Thu Oct 13 04:13:16 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 184 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 13 04:13:16.486740 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c556e77b-e903-43e8-b41a-783938620b7b None None] Authenticating user token {{(pid=81409) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:13:16.488318 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.fernet_utils [None req-c556e77b-e903-43e8-b41a-783938620b7b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81409) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:13:16.508124 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG dbcounter [-] [81409] Writing DB stats keystone:SELECT=100 {{(pid=81409) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:13:16.592580 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c556e77b-e903-43e8-b41a-783938620b7b None None] Validating token access rules against request {{(pid=81409) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:13:16.666854 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-061ae97f-97e3-4d66-8720-b7526fd40a1e service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd8812c3974f84871bb2d761cf0016257', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dcec0431fb6849499d6aec35a234daa6', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81409) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:13:16.667368 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-061ae97f-97e3-4d66-8720-b7526fd40a1e service neutron] REQUEST_METHOD: `GET` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:13:16.667643 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-061ae97f-97e3-4d66-8720-b7526fd40a1e service neutron] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:13:16.667912 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-061ae97f-97e3-4d66-8720-b7526fd40a1e service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:13:16.669505 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.fernet_utils [None req-061ae97f-97e3-4d66-8720-b7526fd40a1e service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81409) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:13:16.741491 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG dbcounter [-] [81409] Writing DB stats keystone:SELECT=100 {{(pid=81409) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:13:16.780262 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-061ae97f-97e3-4d66-8720-b7526fd40a1e service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:13:16.781691 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-061ae97f-97e3-4d66-8720-b7526fd40a1e service neutron] RBAC: Authorization granted {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:13:16.800310 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 511/1000] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Oct 13 04:13:16 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 315 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 13 04:13:19.045169 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07ab7f47-8ddc-4915-a37f-2fbc6f6bd92b None None] REQUEST_METHOD: `GET` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:13:19.045723 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07ab7f47-8ddc-4915-a37f-2fbc6f6bd92b None None] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:13:19.045723 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07ab7f47-8ddc-4915-a37f-2fbc6f6bd92b None None] PATH_INFO: `/` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:13:19.047079 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 490/1001] 10.222.0.27 () {54 vars in 927 bytes} [Thu Oct 13 04:13:19 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 13 04:13:19.053083 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f41a9022-2395-404c-982c-58288c37b595 None None] REQUEST_METHOD: `POST` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:13:19.053521 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f41a9022-2395-404c-982c-58288c37b595 None None] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:13:19.053761 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f41a9022-2395-404c-982c-58288c37b595 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:13:19.124451 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.auth.core [None req-f41a9022-2395-404c-982c-58288c37b595 None None] MFA Rules not processed for user `0e2d979563d6421995451a0e86bf3022`. Rule list: `[]` (Enabled: `True`). {{(pid=81409) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 13 04:13:19.125642 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.fernet_utils [None req-f41a9022-2395-404c-982c-58288c37b595 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81409) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:13:19.233224 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 512/1002] 10.222.0.27 () {58 vars in 1024 bytes} [Thu Oct 13 04:13:19 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 181 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 13 04:13:19.518100 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-07ab7f47-8ddc-4915-a37f-2fbc6f6bd92b None None] Authenticating user token {{(pid=81408) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:13:19.519765 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.fernet_utils [None req-07ab7f47-8ddc-4915-a37f-2fbc6f6bd92b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81408) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:13:19.623508 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-07ab7f47-8ddc-4915-a37f-2fbc6f6bd92b None None] Validating token access rules against request {{(pid=81408) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:13:19.699375 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6f3374b2-c55e-4b8e-96c8-718201d39def service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd8812c3974f84871bb2d761cf0016257', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dcec0431fb6849499d6aec35a234daa6', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81408) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:13:19.699995 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f3374b2-c55e-4b8e-96c8-718201d39def service neutron] REQUEST_METHOD: `GET` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:13:19.700254 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f3374b2-c55e-4b8e-96c8-718201d39def service neutron] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:13:19.700521 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f3374b2-c55e-4b8e-96c8-718201d39def service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:13:19.702120 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.fernet_utils [None req-6f3374b2-c55e-4b8e-96c8-718201d39def service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81408) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:13:19.718816 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG dbcounter [-] [81408] Writing DB stats keystone:SELECT=100 {{(pid=81408) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:13:19.816434 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6f3374b2-c55e-4b8e-96c8-718201d39def service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:13:19.817794 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6f3374b2-c55e-4b8e-96c8-718201d39def service neutron] RBAC: Authorization granted {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:13:19.836284 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 491/1003] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Oct 13 04:13:19 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 319 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Oct 13 04:13:22.061906 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11b872e8-4ec3-414b-abf0-525a4f1d533c None None] REQUEST_METHOD: `GET` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:13:22.062437 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11b872e8-4ec3-414b-abf0-525a4f1d533c None None] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:13:22.062530 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11b872e8-4ec3-414b-abf0-525a4f1d533c None None] PATH_INFO: `/` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:13:22.063887 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 513/1004] 10.222.0.27 () {54 vars in 927 bytes} [Thu Oct 13 04:13:22 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 13 04:13:22.069179 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85a15897-f84d-41ee-8fac-b558941e8543 None None] REQUEST_METHOD: `POST` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:13:22.069626 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85a15897-f84d-41ee-8fac-b558941e8543 None None] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:13:22.069720 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85a15897-f84d-41ee-8fac-b558941e8543 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:13:22.141128 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.auth.core [None req-85a15897-f84d-41ee-8fac-b558941e8543 None None] MFA Rules not processed for user `761e23aab9234a7c9c88dc41d2ddbe8e`. Rule list: `[]` (Enabled: `True`). {{(pid=81408) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 13 04:13:22.143329 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.fernet_utils [None req-85a15897-f84d-41ee-8fac-b558941e8543 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81408) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:13:22.211672 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG dbcounter [-] [81408] Writing DB stats keystone:SELECT=100 {{(pid=81408) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:13:22.247870 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 492/1005] 10.222.0.27 () {58 vars in 1024 bytes} [Thu Oct 13 04:13:22 2022] POST /identity/v3/auth/tokens => generated 3243 bytes in 180 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 13 04:13:22.532974 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-11b872e8-4ec3-414b-abf0-525a4f1d533c None None] Authenticating user token {{(pid=81409) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:13:22.534884 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.fernet_utils [None req-11b872e8-4ec3-414b-abf0-525a4f1d533c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81409) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:13:22.568593 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG dbcounter [-] [81409] Writing DB stats keystone:SELECT=100 {{(pid=81409) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:13:22.644885 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-11b872e8-4ec3-414b-abf0-525a4f1d533c None None] Validating token access rules against request {{(pid=81409) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:13:22.717219 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dba5865d-8b15-4f3e-b1be-5d069495ae1c service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd8812c3974f84871bb2d761cf0016257', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dcec0431fb6849499d6aec35a234daa6', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81409) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:13:22.717738 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dba5865d-8b15-4f3e-b1be-5d069495ae1c service neutron] REQUEST_METHOD: `GET` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:13:22.718010 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dba5865d-8b15-4f3e-b1be-5d069495ae1c service neutron] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:13:22.718304 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dba5865d-8b15-4f3e-b1be-5d069495ae1c service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:13:22.720473 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.fernet_utils [None req-dba5865d-8b15-4f3e-b1be-5d069495ae1c service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81409) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:13:22.797519 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG dbcounter [-] [81409] Writing DB stats keystone:SELECT=100 {{(pid=81409) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:13:22.831437 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dba5865d-8b15-4f3e-b1be-5d069495ae1c service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:13:22.832888 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dba5865d-8b15-4f3e-b1be-5d069495ae1c service neutron] RBAC: Authorization granted {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:13:22.850574 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 514/1006] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Oct 13 04:13:22 2022] GET /identity/v3/auth/tokens => generated 3243 bytes in 319 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 13 04:13:25.231764 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64f60825-d678-49b9-8d05-13d255d92db7 None None] REQUEST_METHOD: `GET` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:13:25.232422 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64f60825-d678-49b9-8d05-13d255d92db7 None None] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:13:25.232422 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64f60825-d678-49b9-8d05-13d255d92db7 None None] PATH_INFO: `/` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:13:25.233650 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 493/1007] 10.222.0.27 () {54 vars in 927 bytes} [Thu Oct 13 04:13:25 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 13 04:13:25.239625 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-166fbf8d-833f-4e78-b779-370975a9c1d1 None None] REQUEST_METHOD: `POST` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:13:25.240028 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-166fbf8d-833f-4e78-b779-370975a9c1d1 None None] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:13:25.240159 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-166fbf8d-833f-4e78-b779-370975a9c1d1 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:13:25.305724 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.auth.core [None req-166fbf8d-833f-4e78-b779-370975a9c1d1 None None] MFA Rules not processed for user `0e2d979563d6421995451a0e86bf3022`. Rule list: `[]` (Enabled: `True`). {{(pid=81409) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 13 04:13:25.306938 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.fernet_utils [None req-166fbf8d-833f-4e78-b779-370975a9c1d1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81409) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:13:25.400663 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 515/1008] 10.222.0.27 () {58 vars in 1024 bytes} [Thu Oct 13 04:13:25 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 162 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 13 04:13:25.695511 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95201ae3-e959-4380-a9f6-48ae43f6ae12 None None] REQUEST_METHOD: `GET` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:13:25.695792 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95201ae3-e959-4380-a9f6-48ae43f6ae12 None None] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:13:25.696065 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95201ae3-e959-4380-a9f6-48ae43f6ae12 None None] PATH_INFO: `/` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:13:25.697425 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 494/1009] 10.222.0.27 () {54 vars in 964 bytes} [Thu Oct 13 04:13:25 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 13 04:13:25.703207 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dac2f9eb-eed9-4830-ae9a-5b9ea867ff5c None None] REQUEST_METHOD: `POST` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:13:25.703487 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dac2f9eb-eed9-4830-ae9a-5b9ea867ff5c None None] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:13:25.703783 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dac2f9eb-eed9-4830-ae9a-5b9ea867ff5c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:13:25.748726 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG dbcounter [-] [81409] Writing DB stats keystone:SELECT=100 {{(pid=81409) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:13:25.779664 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.auth.core [None req-dac2f9eb-eed9-4830-ae9a-5b9ea867ff5c None None] MFA Rules not processed for user `d8812c3974f84871bb2d761cf0016257`. Rule list: `[]` (Enabled: `True`). {{(pid=81409) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 13 04:13:25.780861 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.fernet_utils [None req-dac2f9eb-eed9-4830-ae9a-5b9ea867ff5c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81409) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:13:25.855799 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 516/1010] 10.222.0.27 () {58 vars in 1061 bytes} [Thu Oct 13 04:13:25 2022] POST /identity/v3/auth/tokens => generated 3121 bytes in 154 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 13 04:13:25.865317 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-95201ae3-e959-4380-a9f6-48ae43f6ae12 None None] Authenticating user token {{(pid=81408) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:13:25.866900 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.fernet_utils [None req-95201ae3-e959-4380-a9f6-48ae43f6ae12 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81408) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:13:25.968144 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-95201ae3-e959-4380-a9f6-48ae43f6ae12 None None] Validating token access rules against request {{(pid=81408) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:13:26.041545 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7c8dcf76-0fd5-40de-aaed-9eb0eaa0dbf1 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd8812c3974f84871bb2d761cf0016257', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dcec0431fb6849499d6aec35a234daa6', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81408) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:13:26.042145 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c8dcf76-0fd5-40de-aaed-9eb0eaa0dbf1 service neutron] REQUEST_METHOD: `GET` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:13:26.042444 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c8dcf76-0fd5-40de-aaed-9eb0eaa0dbf1 service neutron] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:13:26.042713 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c8dcf76-0fd5-40de-aaed-9eb0eaa0dbf1 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:13:26.044297 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.fernet_utils [None req-7c8dcf76-0fd5-40de-aaed-9eb0eaa0dbf1 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81408) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:13:26.081043 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG dbcounter [-] [81408] Writing DB stats keystone:SELECT=100 {{(pid=81408) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:13:26.164007 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7c8dcf76-0fd5-40de-aaed-9eb0eaa0dbf1 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:13:26.165444 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7c8dcf76-0fd5-40de-aaed-9eb0eaa0dbf1 service neutron] RBAC: Authorization granted {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:13:26.183529 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 495/1011] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Oct 13 04:13:25 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 319 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 13 04:13:28.596153 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90a775be-d092-40fa-afae-b73cf14f620d None None] REQUEST_METHOD: `GET` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:13:28.596722 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90a775be-d092-40fa-afae-b73cf14f620d None None] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:13:28.596722 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90a775be-d092-40fa-afae-b73cf14f620d None None] PATH_INFO: `/` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:13:28.598070 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 517/1012] 10.222.0.27 () {54 vars in 927 bytes} [Thu Oct 13 04:13:28 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 13 04:13:28.603486 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c36b30f6-f432-4e7a-b384-57c5bcb4c818 None None] REQUEST_METHOD: `POST` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:13:28.603856 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c36b30f6-f432-4e7a-b384-57c5bcb4c818 None None] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:13:28.604031 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c36b30f6-f432-4e7a-b384-57c5bcb4c818 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:13:28.668776 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.auth.core [None req-c36b30f6-f432-4e7a-b384-57c5bcb4c818 None None] MFA Rules not processed for user `0e2d979563d6421995451a0e86bf3022`. Rule list: `[]` (Enabled: `True`). {{(pid=81408) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 13 04:13:28.669894 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.fernet_utils [None req-c36b30f6-f432-4e7a-b384-57c5bcb4c818 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81408) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:13:28.741929 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG dbcounter [-] [81408] Writing DB stats keystone:SELECT=100 {{(pid=81408) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:13:28.764573 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 496/1013] 10.222.0.27 () {58 vars in 1024 bytes} [Thu Oct 13 04:13:28 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 162 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 13 04:13:29.050159 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-90a775be-d092-40fa-afae-b73cf14f620d None None] Authenticating user token {{(pid=81409) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:13:29.051745 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.fernet_utils [None req-90a775be-d092-40fa-afae-b73cf14f620d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81409) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:13:29.154631 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-90a775be-d092-40fa-afae-b73cf14f620d None None] Validating token access rules against request {{(pid=81409) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:13:29.182411 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG dbcounter [-] [81409] Writing DB stats keystone:SELECT=100 {{(pid=81409) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:13:29.228372 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-51367eb2-c17f-44c1-95ca-767f83508e5a service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd8812c3974f84871bb2d761cf0016257', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dcec0431fb6849499d6aec35a234daa6', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81409) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:13:29.228972 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51367eb2-c17f-44c1-95ca-767f83508e5a service neutron] REQUEST_METHOD: `GET` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:13:29.229247 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51367eb2-c17f-44c1-95ca-767f83508e5a service neutron] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:13:29.229515 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51367eb2-c17f-44c1-95ca-767f83508e5a service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:13:29.231148 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.fernet_utils [None req-51367eb2-c17f-44c1-95ca-767f83508e5a service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81409) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:13:29.339092 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-51367eb2-c17f-44c1-95ca-767f83508e5a service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:13:29.340429 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-51367eb2-c17f-44c1-95ca-767f83508e5a service neutron] RBAC: Authorization granted {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:13:29.358558 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 518/1014] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Oct 13 04:13:29 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 309 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 13 04:13:32.126469 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c318541d-32af-41ed-b706-43989c07dc5d None None] REQUEST_METHOD: `GET` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:13:32.127024 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c318541d-32af-41ed-b706-43989c07dc5d None None] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:13:32.127024 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c318541d-32af-41ed-b706-43989c07dc5d None None] PATH_INFO: `/` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:13:32.128358 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 497/1015] 10.222.0.27 () {54 vars in 927 bytes} [Thu Oct 13 04:13:32 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 13 04:13:32.133497 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea02952f-ad07-4c14-b016-65cdf4a9e3aa None None] REQUEST_METHOD: `POST` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:13:32.133884 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea02952f-ad07-4c14-b016-65cdf4a9e3aa None None] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:13:32.134037 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea02952f-ad07-4c14-b016-65cdf4a9e3aa None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:13:32.199538 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.auth.core [None req-ea02952f-ad07-4c14-b016-65cdf4a9e3aa None None] MFA Rules not processed for user `0e2d979563d6421995451a0e86bf3022`. Rule list: `[]` (Enabled: `True`). {{(pid=81409) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 13 04:13:32.200707 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.fernet_utils [None req-ea02952f-ad07-4c14-b016-65cdf4a9e3aa None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81409) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:13:32.209165 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG dbcounter [-] [81409] Writing DB stats keystone:SELECT=100 {{(pid=81409) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:13:32.297141 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 519/1016] 10.222.0.27 () {58 vars in 1024 bytes} [Thu Oct 13 04:13:32 2022] POST /identity/v3/auth/tokens => generated 3238 bytes in 165 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 13 04:13:32.582333 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c318541d-32af-41ed-b706-43989c07dc5d None None] Authenticating user token {{(pid=81408) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:13:32.583857 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.fernet_utils [None req-c318541d-32af-41ed-b706-43989c07dc5d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81408) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:13:32.686313 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c318541d-32af-41ed-b706-43989c07dc5d None None] Validating token access rules against request {{(pid=81408) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:13:32.759044 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-23996573-1c01-46e8-b14a-0c875973b63c service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd8812c3974f84871bb2d761cf0016257', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dcec0431fb6849499d6aec35a234daa6', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81408) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:13:32.759639 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23996573-1c01-46e8-b14a-0c875973b63c service neutron] REQUEST_METHOD: `GET` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:13:32.759910 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23996573-1c01-46e8-b14a-0c875973b63c service neutron] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:13:32.760179 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23996573-1c01-46e8-b14a-0c875973b63c service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:13:32.761776 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.fernet_utils [None req-23996573-1c01-46e8-b14a-0c875973b63c service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81408) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:13:32.796217 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG dbcounter [-] [81408] Writing DB stats keystone:SELECT=100 {{(pid=81408) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:13:32.870541 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-23996573-1c01-46e8-b14a-0c875973b63c service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:13:32.871915 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-23996573-1c01-46e8-b14a-0c875973b63c service neutron] RBAC: Authorization granted {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:13:32.893803 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 498/1017] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Oct 13 04:13:32 2022] GET /identity/v3/auth/tokens => generated 3238 bytes in 309 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 13 04:13:37.092815 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d138931c-adf5-484a-b914-22e69731a43f None None] REQUEST_METHOD: `GET` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:13:37.093324 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d138931c-adf5-484a-b914-22e69731a43f None None] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:13:37.093324 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d138931c-adf5-484a-b914-22e69731a43f None None] PATH_INFO: `/` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:13:37.095232 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 520/1018] 10.222.0.27 () {54 vars in 927 bytes} [Thu Oct 13 04:13:37 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 13 04:13:37.100437 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96675259-f7cf-462b-b228-40f51e538add None None] REQUEST_METHOD: `POST` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:13:37.100826 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96675259-f7cf-462b-b228-40f51e538add None None] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:13:37.101153 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96675259-f7cf-462b-b228-40f51e538add None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:13:37.167654 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.auth.core [None req-96675259-f7cf-462b-b228-40f51e538add None None] MFA Rules not processed for user `0e2d979563d6421995451a0e86bf3022`. Rule list: `[]` (Enabled: `True`). {{(pid=81408) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 13 04:13:37.168784 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.fernet_utils [None req-96675259-f7cf-462b-b228-40f51e538add None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81408) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:13:37.254500 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG dbcounter [-] [81408] Writing DB stats keystone:SELECT=100 {{(pid=81408) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:13:37.265730 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 499/1019] 10.222.0.27 () {58 vars in 1024 bytes} [Thu Oct 13 04:13:37 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 166 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 13 04:13:37.547016 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d138931c-adf5-484a-b914-22e69731a43f None None] Authenticating user token {{(pid=81409) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:13:37.548670 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.fernet_utils [None req-d138931c-adf5-484a-b914-22e69731a43f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81409) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:13:37.653180 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d138931c-adf5-484a-b914-22e69731a43f None None] Validating token access rules against request {{(pid=81409) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:13:37.691687 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG dbcounter [-] [81409] Writing DB stats keystone:SELECT=100 {{(pid=81409) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:13:37.728342 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-88f7c7de-feca-442e-8ae0-5bd50cb21514 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd8812c3974f84871bb2d761cf0016257', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dcec0431fb6849499d6aec35a234daa6', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81409) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:13:37.728940 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88f7c7de-feca-442e-8ae0-5bd50cb21514 service neutron] REQUEST_METHOD: `GET` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:13:37.729222 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88f7c7de-feca-442e-8ae0-5bd50cb21514 service neutron] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:13:37.729487 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88f7c7de-feca-442e-8ae0-5bd50cb21514 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:13:37.731102 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.fernet_utils [None req-88f7c7de-feca-442e-8ae0-5bd50cb21514 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81409) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:13:37.840756 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-88f7c7de-feca-442e-8ae0-5bd50cb21514 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:13:37.842220 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-88f7c7de-feca-442e-8ae0-5bd50cb21514 service neutron] RBAC: Authorization granted {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:13:37.861239 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 521/1020] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Oct 13 04:13:37 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 315 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 13 04:13:40.643064 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e952d604-b625-4646-a900-3085ad035aff None None] REQUEST_METHOD: `GET` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:13:40.643628 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e952d604-b625-4646-a900-3085ad035aff None None] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:13:40.643628 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e952d604-b625-4646-a900-3085ad035aff None None] PATH_INFO: `/` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:13:40.644934 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 500/1021] 10.222.0.27 () {54 vars in 927 bytes} [Thu Oct 13 04:13:40 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 13 04:13:40.650026 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d6fac9c-875b-4e4c-b176-0b6384351f34 None None] REQUEST_METHOD: `POST` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:13:40.650472 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d6fac9c-875b-4e4c-b176-0b6384351f34 None None] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:13:40.650605 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d6fac9c-875b-4e4c-b176-0b6384351f34 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:13:40.717142 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.auth.core [None req-0d6fac9c-875b-4e4c-b176-0b6384351f34 None None] MFA Rules not processed for user `0e2d979563d6421995451a0e86bf3022`. Rule list: `[]` (Enabled: `True`). {{(pid=81409) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 13 04:13:40.718396 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.fernet_utils [None req-0d6fac9c-875b-4e4c-b176-0b6384351f34 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81409) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:13:40.737797 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG dbcounter [-] [81409] Writing DB stats keystone:SELECT=100 {{(pid=81409) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:13:40.816828 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 522/1022] 10.222.0.27 () {58 vars in 1024 bytes} [Thu Oct 13 04:13:40 2022] POST /identity/v3/auth/tokens => generated 3238 bytes in 168 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 13 04:13:41.106267 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e952d604-b625-4646-a900-3085ad035aff None None] Authenticating user token {{(pid=81408) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:13:41.108000 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.fernet_utils [None req-e952d604-b625-4646-a900-3085ad035aff None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81408) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:13:41.214731 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e952d604-b625-4646-a900-3085ad035aff None None] Validating token access rules against request {{(pid=81408) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:13:41.294794 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3ae92948-4651-40eb-99a0-0edf529acfd5 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd8812c3974f84871bb2d761cf0016257', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dcec0431fb6849499d6aec35a234daa6', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81408) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:13:41.295394 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ae92948-4651-40eb-99a0-0edf529acfd5 service neutron] REQUEST_METHOD: `GET` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:13:41.295669 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ae92948-4651-40eb-99a0-0edf529acfd5 service neutron] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:13:41.295937 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ae92948-4651-40eb-99a0-0edf529acfd5 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:13:41.298093 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.fernet_utils [None req-3ae92948-4651-40eb-99a0-0edf529acfd5 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81408) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:13:41.342744 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG dbcounter [-] [81408] Writing DB stats keystone:SELECT=100 {{(pid=81408) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:13:41.417888 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3ae92948-4651-40eb-99a0-0edf529acfd5 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:13:41.419371 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3ae92948-4651-40eb-99a0-0edf529acfd5 service neutron] RBAC: Authorization granted {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:13:41.440057 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 501/1023] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Oct 13 04:13:41 2022] GET /identity/v3/auth/tokens => generated 3238 bytes in 334 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 13 04:13:45.792154 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b07c5a6-98ac-456d-8af7-83c64d65e9ad None None] REQUEST_METHOD: `GET` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:13:45.792712 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b07c5a6-98ac-456d-8af7-83c64d65e9ad None None] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:13:45.792712 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b07c5a6-98ac-456d-8af7-83c64d65e9ad None None] PATH_INFO: `/` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:13:45.794130 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 523/1024] 10.222.0.27 () {54 vars in 927 bytes} [Thu Oct 13 04:13:45 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 13 04:13:45.799474 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bfba7c9d-a118-4c60-a99d-e67d3f9070f1 None None] REQUEST_METHOD: `POST` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:13:45.799900 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bfba7c9d-a118-4c60-a99d-e67d3f9070f1 None None] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:13:45.800045 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bfba7c9d-a118-4c60-a99d-e67d3f9070f1 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:13:45.871738 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.auth.core [None req-bfba7c9d-a118-4c60-a99d-e67d3f9070f1 None None] MFA Rules not processed for user `0e2d979563d6421995451a0e86bf3022`. Rule list: `[]` (Enabled: `True`). {{(pid=81408) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 13 04:13:45.872957 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.fernet_utils [None req-bfba7c9d-a118-4c60-a99d-e67d3f9070f1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81408) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:13:45.973631 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG dbcounter [-] [81408] Writing DB stats keystone:SELECT=100 {{(pid=81408) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:13:45.976025 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 502/1025] 10.222.0.27 () {58 vars in 1024 bytes} [Thu Oct 13 04:13:45 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 178 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 13 04:13:46.265430 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5b07c5a6-98ac-456d-8af7-83c64d65e9ad None None] Authenticating user token {{(pid=81409) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:13:46.267617 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.fernet_utils [None req-5b07c5a6-98ac-456d-8af7-83c64d65e9ad None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81409) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:13:46.369518 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5b07c5a6-98ac-456d-8af7-83c64d65e9ad None None] Validating token access rules against request {{(pid=81409) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:13:46.418662 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG dbcounter [-] [81409] Writing DB stats keystone:SELECT=100 {{(pid=81409) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:13:46.445451 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eb54b652-3d7e-4bbd-9184-8456faf23be7 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd8812c3974f84871bb2d761cf0016257', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dcec0431fb6849499d6aec35a234daa6', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81409) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:13:46.446226 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb54b652-3d7e-4bbd-9184-8456faf23be7 service neutron] REQUEST_METHOD: `GET` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:13:46.446501 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb54b652-3d7e-4bbd-9184-8456faf23be7 service neutron] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:13:46.446764 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb54b652-3d7e-4bbd-9184-8456faf23be7 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:13:46.448335 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.fernet_utils [None req-eb54b652-3d7e-4bbd-9184-8456faf23be7 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81409) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:13:46.560449 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eb54b652-3d7e-4bbd-9184-8456faf23be7 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:13:46.561881 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eb54b652-3d7e-4bbd-9184-8456faf23be7 service neutron] RBAC: Authorization granted {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:13:46.580483 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 524/1026] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Oct 13 04:13:46 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 316 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 13 04:13:51.586631 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79089fa4-ea97-4784-9d47-5c49a5e840fd None None] REQUEST_METHOD: `GET` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:13:51.587518 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79089fa4-ea97-4784-9d47-5c49a5e840fd None None] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:13:51.587991 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79089fa4-ea97-4784-9d47-5c49a5e840fd None None] PATH_INFO: `/` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:13:51.589796 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 503/1027] 10.222.0.27 () {54 vars in 921 bytes} [Thu Oct 13 04:13:51 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 13 04:13:51.595905 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c665b30c-66da-4c18-8949-cf7a137affb4 None None] REQUEST_METHOD: `POST` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:13:51.596831 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c665b30c-66da-4c18-8949-cf7a137affb4 None None] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:13:51.597342 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c665b30c-66da-4c18-8949-cf7a137affb4 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:13:51.717514 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.auth.core [None req-c665b30c-66da-4c18-8949-cf7a137affb4 None None] MFA Rules not processed for user `ba1b6e5868a549a99760ebe6068fd6b9`. Rule list: `[]` (Enabled: `True`). {{(pid=81409) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 13 04:13:51.719600 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.fernet_utils [None req-c665b30c-66da-4c18-8949-cf7a137affb4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81409) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:13:51.741291 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG dbcounter [-] [81409] Writing DB stats keystone:SELECT=100 {{(pid=81409) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:13:51.848518 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 525/1028] 10.222.0.27 () {58 vars in 1018 bytes} [Thu Oct 13 04:13:51 2022] POST /identity/v3/auth/tokens => generated 3308 bytes in 254 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 13 04:13:55.992725 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dfa5b434-e227-4cce-aad0-38018a68844e None None] REQUEST_METHOD: `GET` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:13:55.993206 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dfa5b434-e227-4cce-aad0-38018a68844e None None] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:13:55.993295 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dfa5b434-e227-4cce-aad0-38018a68844e None None] PATH_INFO: `/` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:13:55.994633 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 504/1029] 10.222.0.27 () {54 vars in 921 bytes} [Thu Oct 13 04:13:55 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 13 04:13:55.999609 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e13448c8-cf60-4103-b7e7-737ca9978a64 None None] REQUEST_METHOD: `POST` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:13:56.000081 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e13448c8-cf60-4103-b7e7-737ca9978a64 None None] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:13:56.000314 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e13448c8-cf60-4103-b7e7-737ca9978a64 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:13:56.082802 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.auth.core [None req-e13448c8-cf60-4103-b7e7-737ca9978a64 None None] MFA Rules not processed for user `ba1b6e5868a549a99760ebe6068fd6b9`. Rule list: `[]` (Enabled: `True`). {{(pid=81409) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 13 04:13:56.084039 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.fernet_utils [None req-e13448c8-cf60-4103-b7e7-737ca9978a64 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81409) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:13:56.160052 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG dbcounter [-] [81409] Writing DB stats keystone:SELECT=100 {{(pid=81409) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:13:56.198458 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 526/1030] 10.222.0.27 () {58 vars in 1018 bytes} [Thu Oct 13 04:13:55 2022] POST /identity/v3/auth/tokens => generated 3308 bytes in 200 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 13 04:13:57.538731 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4bb99848-8f86-4999-976d-997aef0a43ae None None] REQUEST_METHOD: `GET` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:13:57.539283 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4bb99848-8f86-4999-976d-997aef0a43ae None None] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:13:57.539283 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4bb99848-8f86-4999-976d-997aef0a43ae None None] PATH_INFO: `/` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:13:57.540619 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 505/1031] 10.222.0.27 () {54 vars in 921 bytes} [Thu Oct 13 04:13:57 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 13 04:13:57.545880 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-709fc040-e3f3-415a-b02c-7c2c5ca1c9b8 None None] REQUEST_METHOD: `POST` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:13:57.546343 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-709fc040-e3f3-415a-b02c-7c2c5ca1c9b8 None None] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:13:57.546745 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-709fc040-e3f3-415a-b02c-7c2c5ca1c9b8 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:13:57.623099 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.auth.core [None req-709fc040-e3f3-415a-b02c-7c2c5ca1c9b8 None None] MFA Rules not processed for user `ba1b6e5868a549a99760ebe6068fd6b9`. Rule list: `[]` (Enabled: `True`). {{(pid=81409) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 13 04:13:57.624302 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.fernet_utils [None req-709fc040-e3f3-415a-b02c-7c2c5ca1c9b8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81409) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:13:57.731767 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 527/1032] 10.222.0.27 () {58 vars in 1018 bytes} [Thu Oct 13 04:13:57 2022] POST /identity/v3/auth/tokens => generated 3308 bytes in 187 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 13 04:14:01.308747 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39150bd1-9c6b-4e63-85da-2534abc915d9 None None] REQUEST_METHOD: `GET` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:14:01.309721 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39150bd1-9c6b-4e63-85da-2534abc915d9 None None] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:14:01.310273 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39150bd1-9c6b-4e63-85da-2534abc915d9 None None] PATH_INFO: `/` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:14:01.312108 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 506/1033] 10.222.0.27 () {54 vars in 927 bytes} [Thu Oct 13 04:14:01 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 13 04:14:01.318224 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70543631-48c8-4a46-a1e1-a59fad304b76 None None] REQUEST_METHOD: `POST` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:14:01.319144 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70543631-48c8-4a46-a1e1-a59fad304b76 None None] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:14:01.319656 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70543631-48c8-4a46-a1e1-a59fad304b76 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:14:01.360587 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG dbcounter [-] [81409] Writing DB stats keystone:SELECT=100 {{(pid=81409) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:14:01.395505 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.auth.core [None req-70543631-48c8-4a46-a1e1-a59fad304b76 None None] MFA Rules not processed for user `0e2d979563d6421995451a0e86bf3022`. Rule list: `[]` (Enabled: `True`). {{(pid=81409) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 13 04:14:01.397080 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.fernet_utils [None req-70543631-48c8-4a46-a1e1-a59fad304b76 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81409) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:14:01.512476 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 528/1034] 10.222.0.27 () {58 vars in 1024 bytes} [Thu Oct 13 04:14:01 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 195 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 13 04:14:01.819265 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a472d31-e2b0-444e-8151-81de17b00e43 None None] REQUEST_METHOD: `GET` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:14:01.820150 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a472d31-e2b0-444e-8151-81de17b00e43 None None] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:14:01.821068 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a472d31-e2b0-444e-8151-81de17b00e43 None None] PATH_INFO: `/` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:14:01.825564 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 507/1035] 10.222.0.27 () {54 vars in 961 bytes} [Thu Oct 13 04:14:01 2022] GET /identity => generated 270 bytes in 9 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 13 04:14:01.837324 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1656e729-8a69-4895-81a8-12e119b68e3f None None] REQUEST_METHOD: `POST` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:14:01.838300 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1656e729-8a69-4895-81a8-12e119b68e3f None None] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:14:01.839183 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1656e729-8a69-4895-81a8-12e119b68e3f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:14:01.928879 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.auth.core [None req-1656e729-8a69-4895-81a8-12e119b68e3f None None] MFA Rules not processed for user `cb740ed7f7684b079e45f887be83f795`. Rule list: `[]` (Enabled: `True`). {{(pid=81409) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 13 04:14:01.930070 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.fernet_utils [None req-1656e729-8a69-4895-81a8-12e119b68e3f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81409) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:14:01.982901 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG dbcounter [-] [81409] Writing DB stats keystone:SELECT=100 {{(pid=81409) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:14:02.056605 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 529/1036] 10.222.0.27 () {58 vars in 1058 bytes} [Thu Oct 13 04:14:01 2022] POST /identity/v3/auth/tokens => generated 3372 bytes in 222 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 13 04:14:02.065176 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4a472d31-e2b0-444e-8151-81de17b00e43 None None] Authenticating user token {{(pid=81408) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:14:02.066711 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.fernet_utils [None req-4a472d31-e2b0-444e-8151-81de17b00e43 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81408) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:14:02.225672 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4a472d31-e2b0-444e-8151-81de17b00e43 None None] Validating token access rules against request {{(pid=81408) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:14:02.305604 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG dbcounter [-] [81408] Writing DB stats keystone:SELECT=100 {{(pid=81408) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:14:02.368469 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a9be4beb-a6cd-4ffd-b4f8-31e8317e29e7 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cb740ed7f7684b079e45f887be83f795', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dcec0431fb6849499d6aec35a234daa6', 'project_domain_id': 'default', 'roles': ['ResellerAdmin', 'admin', 'reader', 'service', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81408) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:14:02.369223 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9be4beb-a6cd-4ffd-b4f8-31e8317e29e7 service nova] REQUEST_METHOD: `GET` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:14:02.369702 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9be4beb-a6cd-4ffd-b4f8-31e8317e29e7 service nova] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:14:02.370049 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9be4beb-a6cd-4ffd-b4f8-31e8317e29e7 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:14:02.371841 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.fernet_utils [None req-a9be4beb-a6cd-4ffd-b4f8-31e8317e29e7 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81408) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:14:02.488812 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a9be4beb-a6cd-4ffd-b4f8-31e8317e29e7 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:14:02.490436 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a9be4beb-a6cd-4ffd-b4f8-31e8317e29e7 service nova] RBAC: Authorization granted {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:14:02.510676 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 508/1037] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Oct 13 04:14:02 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 446 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 13 04:14:04.916902 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-760ecb78-3aba-419a-be60-0ade14b745b2 None None] REQUEST_METHOD: `GET` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:14:04.917345 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-760ecb78-3aba-419a-be60-0ade14b745b2 None None] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:14:04.917604 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-760ecb78-3aba-419a-be60-0ade14b745b2 None None] PATH_INFO: `/` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:14:04.922541 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 530/1038] 10.222.0.27 () {54 vars in 927 bytes} [Thu Oct 13 04:14:04 2022] GET /identity => generated 270 bytes in 7 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 13 04:14:04.924401 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9af3dc60-9ecc-4c8d-a87a-be1bb4f31b7e None None] REQUEST_METHOD: `POST` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:14:04.924835 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9af3dc60-9ecc-4c8d-a87a-be1bb4f31b7e None None] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:14:04.925133 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9af3dc60-9ecc-4c8d-a87a-be1bb4f31b7e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:14:04.998997 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.auth.core [None req-9af3dc60-9ecc-4c8d-a87a-be1bb4f31b7e None None] MFA Rules not processed for user `0e2d979563d6421995451a0e86bf3022`. Rule list: `[]` (Enabled: `True`). {{(pid=81408) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 13 04:14:05.000851 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.fernet_utils [None req-9af3dc60-9ecc-4c8d-a87a-be1bb4f31b7e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81408) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:14:05.044731 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG dbcounter [-] [81408] Writing DB stats keystone:SELECT=100 {{(pid=81408) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:14:05.119286 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 509/1039] 10.222.0.27 () {58 vars in 1024 bytes} [Thu Oct 13 04:14:04 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 189 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 13 04:14:05.406287 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-760ecb78-3aba-419a-be60-0ade14b745b2 None None] Authenticating user token {{(pid=81409) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:14:05.410554 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.fernet_utils [None req-760ecb78-3aba-419a-be60-0ade14b745b2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81409) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:14:05.589186 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-760ecb78-3aba-419a-be60-0ade14b745b2 None None] Validating token access rules against request {{(pid=81409) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:14:05.599921 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG dbcounter [-] [81409] Writing DB stats keystone:SELECT=100 {{(pid=81409) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:14:05.721043 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d160a4ad-0465-42e0-bc5e-317dd8b80619 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cb740ed7f7684b079e45f887be83f795', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dcec0431fb6849499d6aec35a234daa6', 'project_domain_id': 'default', 'roles': ['ResellerAdmin', 'admin', 'reader', 'service', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81409) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:14:05.721659 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d160a4ad-0465-42e0-bc5e-317dd8b80619 service nova] REQUEST_METHOD: `GET` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:14:05.721933 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d160a4ad-0465-42e0-bc5e-317dd8b80619 service nova] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:14:05.722199 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d160a4ad-0465-42e0-bc5e-317dd8b80619 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:14:05.723804 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.fernet_utils [None req-d160a4ad-0465-42e0-bc5e-317dd8b80619 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81409) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:14:05.816955 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d160a4ad-0465-42e0-bc5e-317dd8b80619 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:14:05.818580 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d160a4ad-0465-42e0-bc5e-317dd8b80619 service nova] RBAC: Authorization granted {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:14:05.837414 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 531/1040] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Oct 13 04:14:05 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 432 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 13 04:14:08.367070 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1489244f-ef6f-4ef8-923e-dc2f5953cf62 None None] REQUEST_METHOD: `GET` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:14:08.367606 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1489244f-ef6f-4ef8-923e-dc2f5953cf62 None None] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:14:08.367606 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1489244f-ef6f-4ef8-923e-dc2f5953cf62 None None] PATH_INFO: `/` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:14:08.369218 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 510/1041] 10.222.0.27 () {54 vars in 927 bytes} [Thu Oct 13 04:14:08 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 13 04:14:08.375009 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35b1dea4-b1c8-44d9-8cf7-ed7826bff3a8 None None] REQUEST_METHOD: `POST` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:14:08.375653 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35b1dea4-b1c8-44d9-8cf7-ed7826bff3a8 None None] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:14:08.375653 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35b1dea4-b1c8-44d9-8cf7-ed7826bff3a8 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:14:08.391676 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG dbcounter [-] [81409] Writing DB stats keystone:SELECT=100 {{(pid=81409) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:14:08.451838 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.auth.core [None req-35b1dea4-b1c8-44d9-8cf7-ed7826bff3a8 None None] MFA Rules not processed for user `0e2d979563d6421995451a0e86bf3022`. Rule list: `[]` (Enabled: `True`). {{(pid=81409) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 13 04:14:08.453251 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.fernet_utils [None req-35b1dea4-b1c8-44d9-8cf7-ed7826bff3a8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81409) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:14:08.561258 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 532/1042] 10.222.0.27 () {58 vars in 1024 bytes} [Thu Oct 13 04:14:08 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 188 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 13 04:14:08.850599 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22694a9d-c88d-429e-86cf-8af5020a3b01 None None] REQUEST_METHOD: `GET` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:14:08.850996 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22694a9d-c88d-429e-86cf-8af5020a3b01 None None] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:14:08.851367 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22694a9d-c88d-429e-86cf-8af5020a3b01 None None] PATH_INFO: `/` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:14:08.853295 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 511/1043] 10.222.0.27 () {54 vars in 961 bytes} [Thu Oct 13 04:14:08 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 13 04:14:08.859253 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e558346-d160-46b9-9314-99f5142b6410 None None] REQUEST_METHOD: `POST` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:14:08.859689 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e558346-d160-46b9-9314-99f5142b6410 None None] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:14:08.860005 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e558346-d160-46b9-9314-99f5142b6410 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:14:08.938952 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.auth.core [None req-9e558346-d160-46b9-9314-99f5142b6410 None None] MFA Rules not processed for user `cb740ed7f7684b079e45f887be83f795`. Rule list: `[]` (Enabled: `True`). {{(pid=81409) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 13 04:14:08.940137 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.fernet_utils [None req-9e558346-d160-46b9-9314-99f5142b6410 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81409) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:14:08.976299 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG dbcounter [-] [81409] Writing DB stats keystone:SELECT=100 {{(pid=81409) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:14:09.067110 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 533/1044] 10.222.0.27 () {58 vars in 1058 bytes} [Thu Oct 13 04:14:08 2022] POST /identity/v3/auth/tokens => generated 3372 bytes in 209 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 13 04:14:09.077038 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-22694a9d-c88d-429e-86cf-8af5020a3b01 None None] Authenticating user token {{(pid=81408) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:14:09.078997 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.fernet_utils [None req-22694a9d-c88d-429e-86cf-8af5020a3b01 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81408) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:14:09.255401 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-22694a9d-c88d-429e-86cf-8af5020a3b01 None None] Validating token access rules against request {{(pid=81408) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:14:09.277608 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG dbcounter [-] [81408] Writing DB stats keystone:SELECT=100 {{(pid=81408) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:14:09.387674 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e4ad447c-f8f4-4515-8f70-81aff0988e1f service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cb740ed7f7684b079e45f887be83f795', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dcec0431fb6849499d6aec35a234daa6', 'project_domain_id': 'default', 'roles': ['ResellerAdmin', 'admin', 'reader', 'service', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81408) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:14:09.389345 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4ad447c-f8f4-4515-8f70-81aff0988e1f service nova] REQUEST_METHOD: `GET` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:14:09.390670 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4ad447c-f8f4-4515-8f70-81aff0988e1f service nova] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:14:09.391358 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4ad447c-f8f4-4515-8f70-81aff0988e1f service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:14:09.393350 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.fernet_utils [None req-e4ad447c-f8f4-4515-8f70-81aff0988e1f service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81408) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:14:09.514164 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13157db6-93d6-4bc3-9579-76f08af2605a None None] REQUEST_METHOD: `GET` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:14:09.514767 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13157db6-93d6-4bc3-9579-76f08af2605a None None] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:14:09.514767 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13157db6-93d6-4bc3-9579-76f08af2605a None None] PATH_INFO: `/` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:14:09.516259 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 534/1045] 10.222.0.27 () {54 vars in 927 bytes} [Thu Oct 13 04:14:09 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Oct 13 04:14:09.521200 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce7ba770-46cd-4d4d-8b71-f3a6bb48e681 None None] REQUEST_METHOD: `POST` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:14:09.521488 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce7ba770-46cd-4d4d-8b71-f3a6bb48e681 None None] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:14:09.521762 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce7ba770-46cd-4d4d-8b71-f3a6bb48e681 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:14:09.523693 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e4ad447c-f8f4-4515-8f70-81aff0988e1f service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:14:09.525584 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e4ad447c-f8f4-4515-8f70-81aff0988e1f service nova] RBAC: Authorization granted {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:14:09.554689 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 512/1046] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Oct 13 04:14:09 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 478 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 13 04:14:09.602691 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.auth.core [None req-ce7ba770-46cd-4d4d-8b71-f3a6bb48e681 None None] MFA Rules not processed for user `0e2d979563d6421995451a0e86bf3022`. Rule list: `[]` (Enabled: `True`). {{(pid=81409) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 13 04:14:09.602691 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.fernet_utils [None req-ce7ba770-46cd-4d4d-8b71-f3a6bb48e681 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81409) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:14:09.708376 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG dbcounter [-] [81409] Writing DB stats keystone:SELECT=100 {{(pid=81409) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:14:09.719099 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 535/1047] 10.222.0.27 () {58 vars in 1024 bytes} [Thu Oct 13 04:14:09 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 198 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 13 04:14:10.055896 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8bbffd63-6401-40cb-a13c-7683f879d389 None None] REQUEST_METHOD: `GET` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:14:10.056228 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8bbffd63-6401-40cb-a13c-7683f879d389 None None] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:14:10.056564 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8bbffd63-6401-40cb-a13c-7683f879d389 None None] PATH_INFO: `/` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:14:10.058306 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 513/1048] 10.222.0.27 () {54 vars in 963 bytes} [Thu Oct 13 04:14:10 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 13 04:14:10.063635 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cfa9aba9-84cd-4074-bf4d-926f6d6464b1 None None] REQUEST_METHOD: `POST` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:14:10.064008 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cfa9aba9-84cd-4074-bf4d-926f6d6464b1 None None] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:14:10.064346 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cfa9aba9-84cd-4074-bf4d-926f6d6464b1 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:14:10.146986 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.auth.core [None req-cfa9aba9-84cd-4074-bf4d-926f6d6464b1 None None] MFA Rules not processed for user `6141b8a6c3d54e179d6879bb019f4d12`. Rule list: `[]` (Enabled: `True`). {{(pid=81409) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 13 04:14:10.148233 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.fernet_utils [None req-cfa9aba9-84cd-4074-bf4d-926f6d6464b1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81409) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:14:10.232894 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 536/1049] 10.222.0.27 () {58 vars in 1060 bytes} [Thu Oct 13 04:14:10 2022] POST /identity/v3/auth/tokens => generated 3120 bytes in 171 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 13 04:14:10.242451 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8bbffd63-6401-40cb-a13c-7683f879d389 None None] Authenticating user token {{(pid=81408) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:14:10.244198 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.fernet_utils [None req-8bbffd63-6401-40cb-a13c-7683f879d389 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81408) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:14:10.275449 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG dbcounter [-] [81408] Writing DB stats keystone:SELECT=100 {{(pid=81408) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:14:10.365200 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8bbffd63-6401-40cb-a13c-7683f879d389 None None] Validating token access rules against request {{(pid=81408) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:14:10.457134 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-43990939-5aae-4fb8-a8b3-2fcd02ae72a2 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6141b8a6c3d54e179d6879bb019f4d12', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dcec0431fb6849499d6aec35a234daa6', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81408) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:14:10.457745 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43990939-5aae-4fb8-a8b3-2fcd02ae72a2 service cinder] REQUEST_METHOD: `GET` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:14:10.457998 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43990939-5aae-4fb8-a8b3-2fcd02ae72a2 service cinder] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:14:10.458403 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43990939-5aae-4fb8-a8b3-2fcd02ae72a2 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:14:10.459876 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.fernet_utils [None req-43990939-5aae-4fb8-a8b3-2fcd02ae72a2 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81408) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:14:10.554284 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG dbcounter [-] [81408] Writing DB stats keystone:SELECT=100 {{(pid=81408) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:14:10.592508 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-43990939-5aae-4fb8-a8b3-2fcd02ae72a2 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:14:10.595064 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-43990939-5aae-4fb8-a8b3-2fcd02ae72a2 service cinder] RBAC: Authorization granted {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:14:10.615106 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 514/1050] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Oct 13 04:14:10 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 373 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 13 04:14:11.823918 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e733780-dda0-4dae-a7d1-278a4d887b04 None None] REQUEST_METHOD: `GET` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:14:11.824456 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e733780-dda0-4dae-a7d1-278a4d887b04 None None] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:14:11.824456 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e733780-dda0-4dae-a7d1-278a4d887b04 None None] PATH_INFO: `/` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:14:11.825796 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 537/1051] 10.222.0.27 () {54 vars in 927 bytes} [Thu Oct 13 04:14:11 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 13 04:14:11.831379 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72f2e5f3-9b6d-419c-8144-d3deb280faef None None] REQUEST_METHOD: `POST` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:14:11.831825 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72f2e5f3-9b6d-419c-8144-d3deb280faef None None] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:14:11.831992 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72f2e5f3-9b6d-419c-8144-d3deb280faef None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:14:11.899264 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.auth.core [None req-72f2e5f3-9b6d-419c-8144-d3deb280faef None None] MFA Rules not processed for user `0e2d979563d6421995451a0e86bf3022`. Rule list: `[]` (Enabled: `True`). {{(pid=81408) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 13 04:14:11.900492 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.fernet_utils [None req-72f2e5f3-9b6d-419c-8144-d3deb280faef None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81408) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:14:12.003116 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 515/1052] 10.222.0.27 () {58 vars in 1024 bytes} [Thu Oct 13 04:14:11 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 173 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 13 04:14:12.298092 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4e733780-dda0-4dae-a7d1-278a4d887b04 None None] Authenticating user token {{(pid=81409) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:14:12.299663 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.fernet_utils [None req-4e733780-dda0-4dae-a7d1-278a4d887b04 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81409) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:14:12.389461 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG dbcounter [-] [81409] Writing DB stats keystone:SELECT=100 {{(pid=81409) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:14:12.462803 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4e733780-dda0-4dae-a7d1-278a4d887b04 None None] Validating token access rules against request {{(pid=81409) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:14:12.582701 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1f74622a-dd1b-48cc-b35b-6787c0326389 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cb740ed7f7684b079e45f887be83f795', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dcec0431fb6849499d6aec35a234daa6', 'project_domain_id': 'default', 'roles': ['ResellerAdmin', 'admin', 'reader', 'service', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81409) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:14:12.583316 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f74622a-dd1b-48cc-b35b-6787c0326389 service nova] REQUEST_METHOD: `GET` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:14:12.583593 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f74622a-dd1b-48cc-b35b-6787c0326389 service nova] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:14:12.583863 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f74622a-dd1b-48cc-b35b-6787c0326389 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:14:12.585470 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.fernet_utils [None req-1f74622a-dd1b-48cc-b35b-6787c0326389 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81409) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:14:12.636946 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG dbcounter [-] [81409] Writing DB stats keystone:SELECT=100 {{(pid=81409) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:14:12.683637 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1f74622a-dd1b-48cc-b35b-6787c0326389 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:14:12.685277 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1f74622a-dd1b-48cc-b35b-6787c0326389 service nova] RBAC: Authorization granted {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:14:12.704565 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 538/1053] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Oct 13 04:14:12 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 407 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 13 04:14:14.955459 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22da561b-92d2-46c4-aa72-6feecad09927 None None] REQUEST_METHOD: `GET` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:14:14.956031 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22da561b-92d2-46c4-aa72-6feecad09927 None None] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:14:14.956031 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22da561b-92d2-46c4-aa72-6feecad09927 None None] PATH_INFO: `/` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:14:14.957551 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 516/1054] 10.222.0.27 () {54 vars in 927 bytes} [Thu Oct 13 04:14:14 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 13 04:14:14.963000 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9650f200-b59f-4f58-8bcc-c5dfa6cc4629 None None] REQUEST_METHOD: `POST` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:14:14.963425 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9650f200-b59f-4f58-8bcc-c5dfa6cc4629 None None] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:14:14.963621 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9650f200-b59f-4f58-8bcc-c5dfa6cc4629 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:14:15.033557 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.auth.core [None req-9650f200-b59f-4f58-8bcc-c5dfa6cc4629 None None] MFA Rules not processed for user `0e2d979563d6421995451a0e86bf3022`. Rule list: `[]` (Enabled: `True`). {{(pid=81409) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 13 04:14:15.034996 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.fernet_utils [None req-9650f200-b59f-4f58-8bcc-c5dfa6cc4629 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81409) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:14:15.142360 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 539/1055] 10.222.0.27 () {58 vars in 1024 bytes} [Thu Oct 13 04:14:14 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 180 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 13 04:14:15.429790 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-22da561b-92d2-46c4-aa72-6feecad09927 None None] Authenticating user token {{(pid=81408) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:14:15.431505 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.fernet_utils [None req-22da561b-92d2-46c4-aa72-6feecad09927 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81408) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:14:15.472061 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG dbcounter [-] [81408] Writing DB stats keystone:SELECT=100 {{(pid=81408) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:14:15.589005 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-22da561b-92d2-46c4-aa72-6feecad09927 None None] Validating token access rules against request {{(pid=81408) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:14:15.708013 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG dbcounter [-] [81408] Writing DB stats keystone:SELECT=100 {{(pid=81408) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:14:15.714494 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2cdab4e3-c384-4c1e-89d5-2a48347bd055 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cb740ed7f7684b079e45f887be83f795', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dcec0431fb6849499d6aec35a234daa6', 'project_domain_id': 'default', 'roles': ['ResellerAdmin', 'admin', 'reader', 'service', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81408) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:14:15.715219 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2cdab4e3-c384-4c1e-89d5-2a48347bd055 service nova] REQUEST_METHOD: `GET` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:14:15.715593 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2cdab4e3-c384-4c1e-89d5-2a48347bd055 service nova] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:14:15.716117 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2cdab4e3-c384-4c1e-89d5-2a48347bd055 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:14:15.718575 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.fernet_utils [None req-2cdab4e3-c384-4c1e-89d5-2a48347bd055 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81408) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:14:15.821985 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2cdab4e3-c384-4c1e-89d5-2a48347bd055 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:14:15.823959 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2cdab4e3-c384-4c1e-89d5-2a48347bd055 service nova] RBAC: Authorization granted {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:14:15.846689 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 517/1056] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Oct 13 04:14:15 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 417 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 13 04:14:18.096811 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a80049d1-cca1-40c9-a258-ee93d8bc00fd None None] REQUEST_METHOD: `GET` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:14:18.097337 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a80049d1-cca1-40c9-a258-ee93d8bc00fd None None] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:14:18.098118 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a80049d1-cca1-40c9-a258-ee93d8bc00fd None None] PATH_INFO: `/` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:14:18.105704 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 540/1057] 10.222.0.27 () {54 vars in 927 bytes} [Thu Oct 13 04:14:18 2022] GET /identity => generated 270 bytes in 11 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 13 04:14:18.106787 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f95b82c4-22db-4583-a0da-3f76b455dd85 None None] REQUEST_METHOD: `POST` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:14:18.107239 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f95b82c4-22db-4583-a0da-3f76b455dd85 None None] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:14:18.107239 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f95b82c4-22db-4583-a0da-3f76b455dd85 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:14:18.193199 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.auth.core [None req-f95b82c4-22db-4583-a0da-3f76b455dd85 None None] MFA Rules not processed for user `0e2d979563d6421995451a0e86bf3022`. Rule list: `[]` (Enabled: `True`). {{(pid=81408) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 13 04:14:18.194533 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.fernet_utils [None req-f95b82c4-22db-4583-a0da-3f76b455dd85 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81408) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:14:18.263251 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG dbcounter [-] [81408] Writing DB stats keystone:SELECT=100 {{(pid=81408) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:14:18.299720 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 518/1058] 10.222.0.27 () {58 vars in 1024 bytes} [Thu Oct 13 04:14:18 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 194 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 13 04:14:18.653952 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a80049d1-cca1-40c9-a258-ee93d8bc00fd None None] Authenticating user token {{(pid=81409) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:14:18.655497 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.fernet_utils [None req-a80049d1-cca1-40c9-a258-ee93d8bc00fd None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81409) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:14:18.685114 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG dbcounter [-] [81409] Writing DB stats keystone:SELECT=100 {{(pid=81409) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:14:18.823045 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a80049d1-cca1-40c9-a258-ee93d8bc00fd None None] Validating token access rules against request {{(pid=81409) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:14:18.943091 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG dbcounter [-] [81409] Writing DB stats keystone:SELECT=100 {{(pid=81409) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:14:18.963626 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-852910bd-f6ba-4c4f-a340-a83afb65a003 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cb740ed7f7684b079e45f887be83f795', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dcec0431fb6849499d6aec35a234daa6', 'project_domain_id': 'default', 'roles': ['ResellerAdmin', 'admin', 'reader', 'service', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81409) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:14:18.964403 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-852910bd-f6ba-4c4f-a340-a83afb65a003 service nova] REQUEST_METHOD: `GET` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:14:18.964680 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-852910bd-f6ba-4c4f-a340-a83afb65a003 service nova] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:14:18.965015 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-852910bd-f6ba-4c4f-a340-a83afb65a003 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:14:18.967054 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.fernet_utils [None req-852910bd-f6ba-4c4f-a340-a83afb65a003 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81409) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:14:19.071683 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-852910bd-f6ba-4c4f-a340-a83afb65a003 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:14:19.073296 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-852910bd-f6ba-4c4f-a340-a83afb65a003 service nova] RBAC: Authorization granted {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:14:19.092187 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 541/1059] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Oct 13 04:14:18 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 439 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 13 04:14:21.502738 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f4ce5a6-5954-47ea-a0ad-284729e1b728 None None] REQUEST_METHOD: `GET` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:14:21.503290 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f4ce5a6-5954-47ea-a0ad-284729e1b728 None None] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:14:21.503386 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f4ce5a6-5954-47ea-a0ad-284729e1b728 None None] PATH_INFO: `/` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:14:21.504880 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 519/1060] 10.222.0.27 () {54 vars in 927 bytes} [Thu Oct 13 04:14:21 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 13 04:14:21.510858 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f0bcfc6-58b4-4215-8608-54db836295a1 None None] REQUEST_METHOD: `POST` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:14:21.511262 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f0bcfc6-58b4-4215-8608-54db836295a1 None None] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:14:21.511537 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f0bcfc6-58b4-4215-8608-54db836295a1 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:14:21.585168 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.auth.core [None req-2f0bcfc6-58b4-4215-8608-54db836295a1 None None] MFA Rules not processed for user `0e2d979563d6421995451a0e86bf3022`. Rule list: `[]` (Enabled: `True`). {{(pid=81409) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 13 04:14:21.586480 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.fernet_utils [None req-2f0bcfc6-58b4-4215-8608-54db836295a1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81409) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:14:21.651760 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG dbcounter [-] [81409] Writing DB stats keystone:SELECT=100 {{(pid=81409) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:14:21.700052 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 542/1061] 10.222.0.27 () {58 vars in 1024 bytes} [Thu Oct 13 04:14:21 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 191 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 13 04:14:22.040224 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2f4ce5a6-5954-47ea-a0ad-284729e1b728 None None] Authenticating user token {{(pid=81408) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:14:22.041971 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.fernet_utils [None req-2f4ce5a6-5954-47ea-a0ad-284729e1b728 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81408) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:14:22.215599 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2f4ce5a6-5954-47ea-a0ad-284729e1b728 None None] Validating token access rules against request {{(pid=81408) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:14:22.274219 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG dbcounter [-] [81408] Writing DB stats keystone:SELECT=100 {{(pid=81408) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:14:22.341664 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-20cc44e6-92dc-4d1f-be33-83bdf1f99eea service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cb740ed7f7684b079e45f887be83f795', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dcec0431fb6849499d6aec35a234daa6', 'project_domain_id': 'default', 'roles': ['ResellerAdmin', 'admin', 'reader', 'service', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81408) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:14:22.342292 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20cc44e6-92dc-4d1f-be33-83bdf1f99eea service nova] REQUEST_METHOD: `GET` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:14:22.343219 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20cc44e6-92dc-4d1f-be33-83bdf1f99eea service nova] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:14:22.343669 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20cc44e6-92dc-4d1f-be33-83bdf1f99eea service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:14:22.345711 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.fernet_utils [None req-20cc44e6-92dc-4d1f-be33-83bdf1f99eea service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81408) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:14:22.472378 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-20cc44e6-92dc-4d1f-be33-83bdf1f99eea service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:14:22.474867 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-20cc44e6-92dc-4d1f-be33-83bdf1f99eea service nova] RBAC: Authorization granted {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:14:22.501134 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 520/1062] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Oct 13 04:14:22 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 462 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 13 04:14:24.750499 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a1e3567-422b-4e13-8564-32106a91c8c3 None None] REQUEST_METHOD: `GET` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:14:24.751042 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a1e3567-422b-4e13-8564-32106a91c8c3 None None] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:14:24.751042 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a1e3567-422b-4e13-8564-32106a91c8c3 None None] PATH_INFO: `/` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:14:24.752372 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 543/1063] 10.222.0.27 () {54 vars in 927 bytes} [Thu Oct 13 04:14:24 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 13 04:14:24.757804 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a368c80f-8e2a-46b6-ba04-a6c48ca4d0d5 None None] REQUEST_METHOD: `POST` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:14:24.758266 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a368c80f-8e2a-46b6-ba04-a6c48ca4d0d5 None None] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:14:24.758383 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a368c80f-8e2a-46b6-ba04-a6c48ca4d0d5 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:14:24.831389 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.auth.core [None req-a368c80f-8e2a-46b6-ba04-a6c48ca4d0d5 None None] MFA Rules not processed for user `0e2d979563d6421995451a0e86bf3022`. Rule list: `[]` (Enabled: `True`). {{(pid=81408) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 13 04:14:24.832616 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.fernet_utils [None req-a368c80f-8e2a-46b6-ba04-a6c48ca4d0d5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81408) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:14:24.840703 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG dbcounter [-] [81408] Writing DB stats keystone:SELECT=100 {{(pid=81408) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:14:24.934420 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 521/1064] 10.222.0.27 () {58 vars in 1024 bytes} [Thu Oct 13 04:14:24 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 178 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 13 04:14:25.225875 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5a1e3567-422b-4e13-8564-32106a91c8c3 None None] Authenticating user token {{(pid=81409) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:14:25.227649 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.fernet_utils [None req-5a1e3567-422b-4e13-8564-32106a91c8c3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81409) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:14:25.376030 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5a1e3567-422b-4e13-8564-32106a91c8c3 None None] Validating token access rules against request {{(pid=81409) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:14:25.425476 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG dbcounter [-] [81409] Writing DB stats keystone:SELECT=100 {{(pid=81409) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:14:25.494680 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-81efa058-12ed-486c-8a56-72bd8cf30ed5 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cb740ed7f7684b079e45f887be83f795', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dcec0431fb6849499d6aec35a234daa6', 'project_domain_id': 'default', 'roles': ['ResellerAdmin', 'admin', 'reader', 'service', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81409) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:14:25.495197 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81efa058-12ed-486c-8a56-72bd8cf30ed5 service nova] REQUEST_METHOD: `GET` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:14:25.495520 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81efa058-12ed-486c-8a56-72bd8cf30ed5 service nova] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:14:25.495783 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81efa058-12ed-486c-8a56-72bd8cf30ed5 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:14:25.497341 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.fernet_utils [None req-81efa058-12ed-486c-8a56-72bd8cf30ed5 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81409) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:14:25.591519 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-81efa058-12ed-486c-8a56-72bd8cf30ed5 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:14:25.593448 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-81efa058-12ed-486c-8a56-72bd8cf30ed5 service nova] RBAC: Authorization granted {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:14:25.612475 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 544/1065] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Oct 13 04:14:25 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 387 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 13 04:14:25.898934 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0770da32-2795-49cc-9050-9bcea11fd8c4 None None] REQUEST_METHOD: `GET` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:14:25.899472 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0770da32-2795-49cc-9050-9bcea11fd8c4 None None] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:14:25.899472 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0770da32-2795-49cc-9050-9bcea11fd8c4 None None] PATH_INFO: `/` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:14:25.900832 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 522/1066] 10.222.0.27 () {54 vars in 927 bytes} [Thu Oct 13 04:14:25 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 13 04:14:25.906382 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1feae5dd-4024-45b0-86d9-0e944ca73c24 None None] REQUEST_METHOD: `POST` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:14:25.906839 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1feae5dd-4024-45b0-86d9-0e944ca73c24 None None] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:14:25.906934 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1feae5dd-4024-45b0-86d9-0e944ca73c24 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:14:25.973700 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG dbcounter [-] [81409] Writing DB stats keystone:SELECT=100 {{(pid=81409) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:14:25.976611 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.auth.core [None req-1feae5dd-4024-45b0-86d9-0e944ca73c24 None None] MFA Rules not processed for user `0e2d979563d6421995451a0e86bf3022`. Rule list: `[]` (Enabled: `True`). {{(pid=81409) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 13 04:14:25.978703 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.fernet_utils [None req-1feae5dd-4024-45b0-86d9-0e944ca73c24 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81409) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:14:26.079137 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 545/1067] 10.222.0.27 () {58 vars in 1024 bytes} [Thu Oct 13 04:14:25 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 174 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 13 04:14:26.387576 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc6b94d1-ba7e-47ec-bb10-bb57643f0ae7 None None] REQUEST_METHOD: `GET` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:14:26.387865 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc6b94d1-ba7e-47ec-bb10-bb57643f0ae7 None None] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:14:26.388140 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc6b94d1-ba7e-47ec-bb10-bb57643f0ae7 None None] PATH_INFO: `/` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:14:26.390360 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 523/1068] 10.222.0.27 () {54 vars in 962 bytes} [Thu Oct 13 04:14:26 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 13 04:14:26.395709 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ff317f5-e938-4d0b-b39a-d5268a9804ab None None] REQUEST_METHOD: `POST` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:14:26.396207 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ff317f5-e938-4d0b-b39a-d5268a9804ab None None] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:14:26.396486 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ff317f5-e938-4d0b-b39a-d5268a9804ab None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:14:26.478390 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.auth.core [None req-2ff317f5-e938-4d0b-b39a-d5268a9804ab None None] MFA Rules not processed for user `a6700f49b86543f6a5b92cad8faaba42`. Rule list: `[]` (Enabled: `True`). {{(pid=81409) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 13 04:14:26.479957 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.fernet_utils [None req-2ff317f5-e938-4d0b-b39a-d5268a9804ab None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81409) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:14:26.565374 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG dbcounter [-] [81409] Writing DB stats keystone:SELECT=100 {{(pid=81409) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:14:26.572133 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 546/1069] 10.222.0.27 () {58 vars in 1059 bytes} [Thu Oct 13 04:14:26 2022] POST /identity/v3/auth/tokens => generated 3120 bytes in 177 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 13 04:14:26.579828 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dc6b94d1-ba7e-47ec-bb10-bb57643f0ae7 None None] Authenticating user token {{(pid=81408) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:14:26.581344 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.fernet_utils [None req-dc6b94d1-ba7e-47ec-bb10-bb57643f0ae7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81408) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:14:26.685116 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dc6b94d1-ba7e-47ec-bb10-bb57643f0ae7 None None] Validating token access rules against request {{(pid=81408) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:14:26.723801 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG dbcounter [-] [81408] Writing DB stats keystone:SELECT=100 {{(pid=81408) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:14:26.761455 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c4ffdc5f-1765-4c12-8010-818407633dad service glance] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a6700f49b86543f6a5b92cad8faaba42', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dcec0431fb6849499d6aec35a234daa6', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81408) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:14:26.762050 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4ffdc5f-1765-4c12-8010-818407633dad service glance] REQUEST_METHOD: `GET` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:14:26.762361 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4ffdc5f-1765-4c12-8010-818407633dad service glance] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:14:26.762633 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4ffdc5f-1765-4c12-8010-818407633dad service glance] PATH_INFO: `/v3/auth/tokens` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:14:26.764909 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.fernet_utils [None req-c4ffdc5f-1765-4c12-8010-818407633dad service glance] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81408) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:14:26.873374 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c4ffdc5f-1765-4c12-8010-818407633dad service glance] RBAC: Authorizing `identity:validate_token()` {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:14:26.875454 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c4ffdc5f-1765-4c12-8010-818407633dad service glance] RBAC: Authorization granted {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:14:26.893892 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 524/1070] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Oct 13 04:14:26 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 314 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 13 04:14:27.659451 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a6e0941-c18a-45b2-9cce-f93ef9214ced None None] REQUEST_METHOD: `POST` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:14:27.659937 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a6e0941-c18a-45b2-9cce-f93ef9214ced None None] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:14:27.660048 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a6e0941-c18a-45b2-9cce-f93ef9214ced None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:14:27.731906 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.auth.core [None req-8a6e0941-c18a-45b2-9cce-f93ef9214ced None None] MFA Rules not processed for user `487b1605fd4143cfaa422a3a7a0344c7`. Rule list: `[]` (Enabled: `True`). {{(pid=81409) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 13 04:14:27.733489 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.fernet_utils [None req-8a6e0941-c18a-45b2-9cce-f93ef9214ced None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81409) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:14:27.823732 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 547/1071] 10.222.0.27 () {58 vars in 1009 bytes} [Thu Oct 13 04:14:27 2022] POST /identity/v3/auth/tokens => generated 3195 bytes in 165 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 13 04:14:27.826883 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae576f52-e90c-4aeb-9508-0f08bbbbd39b None None] REQUEST_METHOD: `GET` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:14:27.827385 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae576f52-e90c-4aeb-9508-0f08bbbbd39b None None] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:14:27.827517 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae576f52-e90c-4aeb-9508-0f08bbbbd39b None None] PATH_INFO: `/` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:14:27.828801 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 525/1072] 10.222.0.27 () {54 vars in 927 bytes} [Thu Oct 13 04:14:27 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 13 04:14:27.834307 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce994662-3b44-44fa-ae6d-7847f7660fc2 None None] REQUEST_METHOD: `POST` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:14:27.834592 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce994662-3b44-44fa-ae6d-7847f7660fc2 None None] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:14:27.834866 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce994662-3b44-44fa-ae6d-7847f7660fc2 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:14:27.871036 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e040c505-33eb-47e9-8d4d-dc5cf274fff9 None None] REQUEST_METHOD: `GET` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:14:27.871394 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e040c505-33eb-47e9-8d4d-dc5cf274fff9 None None] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:14:27.871737 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e040c505-33eb-47e9-8d4d-dc5cf274fff9 None None] PATH_INFO: `/` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:14:27.873445 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 526/1073] 10.222.0.27 () {54 vars in 943 bytes} [Thu Oct 13 04:14:27 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Oct 13 04:14:27.879640 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-104f92bf-eab8-4444-a792-3398cb86bc13 None None] REQUEST_METHOD: `POST` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:14:27.880000 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-104f92bf-eab8-4444-a792-3398cb86bc13 None None] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:14:27.880283 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-104f92bf-eab8-4444-a792-3398cb86bc13 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:14:27.904326 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.auth.core [None req-ce994662-3b44-44fa-ae6d-7847f7660fc2 None None] MFA Rules not processed for user `0e2d979563d6421995451a0e86bf3022`. Rule list: `[]` (Enabled: `True`). {{(pid=81409) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 13 04:14:27.906400 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.fernet_utils [None req-ce994662-3b44-44fa-ae6d-7847f7660fc2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81409) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:14:27.947716 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG dbcounter [-] [81409] Writing DB stats keystone:SELECT=100 {{(pid=81409) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:14:27.959874 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.auth.core [None req-104f92bf-eab8-4444-a792-3398cb86bc13 None None] MFA Rules not processed for user `90a0f466b664458898cf9dc4112ae651`. Rule list: `[]` (Enabled: `True`). {{(pid=81408) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 13 04:14:27.961092 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.fernet_utils [None req-104f92bf-eab8-4444-a792-3398cb86bc13 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81408) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:14:27.970150 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG dbcounter [-] [81408] Writing DB stats keystone:SELECT=100 {{(pid=81408) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:14:28.010702 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 548/1074] 10.222.0.27 () {58 vars in 1024 bytes} [Thu Oct 13 04:14:27 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 177 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 13 04:14:28.073287 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 527/1075] 10.222.0.27 () {58 vars in 1040 bytes} [Thu Oct 13 04:14:27 2022] POST /identity/v3/auth/tokens => generated 3304 bytes in 194 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 13 04:14:28.082974 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ce994662-3b44-44fa-ae6d-7847f7660fc2 None None] Authenticating user token {{(pid=81409) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:14:28.084175 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.fernet_utils [None req-ce994662-3b44-44fa-ae6d-7847f7660fc2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81409) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:14:28.188279 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ce994662-3b44-44fa-ae6d-7847f7660fc2 None None] Validating token access rules against request {{(pid=81409) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:14:28.248882 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG dbcounter [-] [81409] Writing DB stats keystone:SELECT=100 {{(pid=81409) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:14:28.293014 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e8509314-571d-490a-b7d9-4c38860072f5 service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90a0f466b664458898cf9dc4112ae651', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dcec0431fb6849499d6aec35a234daa6', 'project_domain_id': 'default', 'roles': ['service', 'member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81409) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:14:28.293875 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8509314-571d-490a-b7d9-4c38860072f5 service swift] REQUEST_METHOD: `GET` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:14:28.294145 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8509314-571d-490a-b7d9-4c38860072f5 service swift] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:14:28.294145 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8509314-571d-490a-b7d9-4c38860072f5 service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:14:28.295840 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.fernet_utils [None req-e8509314-571d-490a-b7d9-4c38860072f5 service swift] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81409) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:14:28.306255 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-104f92bf-eab8-4444-a792-3398cb86bc13 None None] Authenticating user token {{(pid=81408) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:14:28.307667 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.fernet_utils [None req-104f92bf-eab8-4444-a792-3398cb86bc13 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81408) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:14:28.386721 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e8509314-571d-490a-b7d9-4c38860072f5 service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:14:28.388305 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e8509314-571d-490a-b7d9-4c38860072f5 service swift] RBAC: Authorization granted {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:14:28.396746 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 549/1076] 10.222.0.27 () {60 vars in 1384 bytes} [Thu Oct 13 04:14:28 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 314 msecs (HTTP/1.1 200) 6 headers in 379 bytes (1 switches on core 0) Oct 13 04:14:28.411011 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-104f92bf-eab8-4444-a792-3398cb86bc13 None None] Validating token access rules against request {{(pid=81408) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:14:28.450909 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG dbcounter [-] [81408] Writing DB stats keystone:SELECT=100 {{(pid=81408) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:14:28.548398 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8f501a4a-f1e5-481b-bdb4-ce6fca515bee service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cb740ed7f7684b079e45f887be83f795', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dcec0431fb6849499d6aec35a234daa6', 'project_domain_id': 'default', 'roles': ['ResellerAdmin', 'admin', 'reader', 'service', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81408) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:14:28.549106 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f501a4a-f1e5-481b-bdb4-ce6fca515bee service nova] REQUEST_METHOD: `GET` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:14:28.549384 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f501a4a-f1e5-481b-bdb4-ce6fca515bee service nova] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:14:28.549655 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f501a4a-f1e5-481b-bdb4-ce6fca515bee service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:14:28.551449 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.fernet_utils [None req-8f501a4a-f1e5-481b-bdb4-ce6fca515bee service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81408) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:14:28.649645 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8f501a4a-f1e5-481b-bdb4-ce6fca515bee service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:14:28.651279 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8f501a4a-f1e5-481b-bdb4-ce6fca515bee service nova] RBAC: Authorization granted {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:14:28.674984 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 528/1077] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Oct 13 04:14:28 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 370 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Oct 13 04:14:30.892708 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da1dc3e1-e705-41aa-845a-de7b0d9d9570 None None] REQUEST_METHOD: `GET` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:14:30.893634 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da1dc3e1-e705-41aa-845a-de7b0d9d9570 None None] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:14:30.893634 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da1dc3e1-e705-41aa-845a-de7b0d9d9570 None None] PATH_INFO: `/` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:14:30.894618 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 550/1078] 10.222.0.27 () {54 vars in 927 bytes} [Thu Oct 13 04:14:30 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 13 04:14:30.900777 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5dbe71c7-d451-4e74-a000-793485f2152c None None] REQUEST_METHOD: `POST` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:14:30.901460 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5dbe71c7-d451-4e74-a000-793485f2152c None None] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:14:30.901743 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5dbe71c7-d451-4e74-a000-793485f2152c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:14:30.965561 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG dbcounter [-] [81408] Writing DB stats keystone:SELECT=100 {{(pid=81408) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:14:30.989634 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.auth.core [None req-5dbe71c7-d451-4e74-a000-793485f2152c None None] MFA Rules not processed for user `0e2d979563d6421995451a0e86bf3022`. Rule list: `[]` (Enabled: `True`). {{(pid=81408) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 13 04:14:30.990886 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.fernet_utils [None req-5dbe71c7-d451-4e74-a000-793485f2152c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81408) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:14:31.112798 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 529/1079] 10.222.0.27 () {58 vars in 1024 bytes} [Thu Oct 13 04:14:30 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 214 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 13 04:14:31.404989 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-da1dc3e1-e705-41aa-845a-de7b0d9d9570 None None] Authenticating user token {{(pid=81409) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:14:31.404989 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.fernet_utils [None req-da1dc3e1-e705-41aa-845a-de7b0d9d9570 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81409) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:14:31.502716 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG dbcounter [-] [81409] Writing DB stats keystone:SELECT=100 {{(pid=81409) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:14:31.562922 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-da1dc3e1-e705-41aa-845a-de7b0d9d9570 None None] Validating token access rules against request {{(pid=81409) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:14:31.702062 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f6f0f6c1-7028-4ffc-8525-5f8ba220a1a5 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cb740ed7f7684b079e45f887be83f795', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dcec0431fb6849499d6aec35a234daa6', 'project_domain_id': 'default', 'roles': ['ResellerAdmin', 'admin', 'reader', 'service', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81409) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:14:31.703022 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6f0f6c1-7028-4ffc-8525-5f8ba220a1a5 service nova] REQUEST_METHOD: `GET` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:14:31.703297 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6f0f6c1-7028-4ffc-8525-5f8ba220a1a5 service nova] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:14:31.703560 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6f0f6c1-7028-4ffc-8525-5f8ba220a1a5 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:14:31.705169 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.fernet_utils [None req-f6f0f6c1-7028-4ffc-8525-5f8ba220a1a5 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81409) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:14:31.779864 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG dbcounter [-] [81409] Writing DB stats keystone:SELECT=100 {{(pid=81409) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:14:31.817215 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f6f0f6c1-7028-4ffc-8525-5f8ba220a1a5 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:14:31.818932 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f6f0f6c1-7028-4ffc-8525-5f8ba220a1a5 service nova] RBAC: Authorization granted {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:14:31.845228 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 551/1080] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Oct 13 04:14:31 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 443 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 13 04:14:34.164841 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35c05882-d5fb-466f-b00c-af0d14661e6e None None] REQUEST_METHOD: `GET` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:14:34.165388 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35c05882-d5fb-466f-b00c-af0d14661e6e None None] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:14:34.165388 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35c05882-d5fb-466f-b00c-af0d14661e6e None None] PATH_INFO: `/` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:14:34.167105 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 530/1081] 10.222.0.27 () {54 vars in 927 bytes} [Thu Oct 13 04:14:34 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 13 04:14:34.172970 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13a7400a-c080-42a0-9940-101515b96cec None None] REQUEST_METHOD: `POST` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:14:34.173465 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13a7400a-c080-42a0-9940-101515b96cec None None] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:14:34.173562 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13a7400a-c080-42a0-9940-101515b96cec None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:14:34.242253 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.auth.core [None req-13a7400a-c080-42a0-9940-101515b96cec None None] MFA Rules not processed for user `0e2d979563d6421995451a0e86bf3022`. Rule list: `[]` (Enabled: `True`). {{(pid=81409) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 13 04:14:34.243405 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.fernet_utils [None req-13a7400a-c080-42a0-9940-101515b96cec None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81409) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:14:34.357652 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 552/1082] 10.222.0.27 () {58 vars in 1024 bytes} [Thu Oct 13 04:14:34 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 186 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 13 04:14:34.651541 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-35c05882-d5fb-466f-b00c-af0d14661e6e None None] Authenticating user token {{(pid=81408) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:14:34.653077 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.fernet_utils [None req-35c05882-d5fb-466f-b00c-af0d14661e6e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81408) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:14:34.777849 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG dbcounter [-] [81408] Writing DB stats keystone:SELECT=100 {{(pid=81408) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:14:34.819481 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-35c05882-d5fb-466f-b00c-af0d14661e6e None None] Validating token access rules against request {{(pid=81408) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:14:34.964445 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f193c69d-b019-4840-9d32-966308450fae service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cb740ed7f7684b079e45f887be83f795', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dcec0431fb6849499d6aec35a234daa6', 'project_domain_id': 'default', 'roles': ['ResellerAdmin', 'admin', 'reader', 'service', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81408) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:14:34.965448 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f193c69d-b019-4840-9d32-966308450fae service nova] REQUEST_METHOD: `GET` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:14:34.965821 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f193c69d-b019-4840-9d32-966308450fae service nova] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:14:34.966192 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f193c69d-b019-4840-9d32-966308450fae service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:14:34.968915 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.fernet_utils [None req-f193c69d-b019-4840-9d32-966308450fae service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81408) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:14:35.063335 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG dbcounter [-] [81408] Writing DB stats keystone:SELECT=100 {{(pid=81408) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:14:35.078300 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f193c69d-b019-4840-9d32-966308450fae service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:14:35.079903 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f193c69d-b019-4840-9d32-966308450fae service nova] RBAC: Authorization granted {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:14:35.100137 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 531/1083] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Oct 13 04:14:34 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 449 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 13 04:14:37.364227 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e3d8884-6d53-4ede-95c6-9583032bfbd5 None None] REQUEST_METHOD: `GET` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:14:37.364776 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e3d8884-6d53-4ede-95c6-9583032bfbd5 None None] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:14:37.364776 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e3d8884-6d53-4ede-95c6-9583032bfbd5 None None] PATH_INFO: `/` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:14:37.366108 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 553/1084] 10.222.0.27 () {54 vars in 927 bytes} [Thu Oct 13 04:14:37 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 13 04:14:37.371788 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e44f0f5c-3244-4276-b993-a06c1a7aed37 None None] REQUEST_METHOD: `POST` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:14:37.372313 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e44f0f5c-3244-4276-b993-a06c1a7aed37 None None] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:14:37.372465 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e44f0f5c-3244-4276-b993-a06c1a7aed37 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:14:37.456513 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.auth.core [None req-e44f0f5c-3244-4276-b993-a06c1a7aed37 None None] MFA Rules not processed for user `0e2d979563d6421995451a0e86bf3022`. Rule list: `[]` (Enabled: `True`). {{(pid=81408) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 13 04:14:37.457736 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.fernet_utils [None req-e44f0f5c-3244-4276-b993-a06c1a7aed37 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81408) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:14:37.568767 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 532/1085] 10.222.0.27 () {58 vars in 1024 bytes} [Thu Oct 13 04:14:37 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 198 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 13 04:14:37.870296 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9e3d8884-6d53-4ede-95c6-9583032bfbd5 None None] Authenticating user token {{(pid=81409) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:14:37.872167 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.fernet_utils [None req-9e3d8884-6d53-4ede-95c6-9583032bfbd5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81409) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:14:37.916851 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG dbcounter [-] [81409] Writing DB stats keystone:SELECT=100 {{(pid=81409) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:14:38.035130 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9e3d8884-6d53-4ede-95c6-9583032bfbd5 None None] Validating token access rules against request {{(pid=81409) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:14:38.162158 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG dbcounter [-] [81409] Writing DB stats keystone:SELECT=100 {{(pid=81409) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:14:38.164942 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d29a288b-3ad5-47df-a102-63db17db3737 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cb740ed7f7684b079e45f887be83f795', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dcec0431fb6849499d6aec35a234daa6', 'project_domain_id': 'default', 'roles': ['ResellerAdmin', 'admin', 'reader', 'service', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81409) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:14:38.165725 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d29a288b-3ad5-47df-a102-63db17db3737 service nova] REQUEST_METHOD: `GET` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:14:38.166240 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d29a288b-3ad5-47df-a102-63db17db3737 service nova] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:14:38.166676 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d29a288b-3ad5-47df-a102-63db17db3737 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:14:38.168476 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.fernet_utils [None req-d29a288b-3ad5-47df-a102-63db17db3737 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81409) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:14:38.264488 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d29a288b-3ad5-47df-a102-63db17db3737 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:14:38.266379 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d29a288b-3ad5-47df-a102-63db17db3737 service nova] RBAC: Authorization granted {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:14:38.287905 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 554/1086] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Oct 13 04:14:37 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 418 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 13 04:14:47.566963 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG dbcounter [-] [81408] Writing DB stats keystone:SELECT=76 {{(pid=81408) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:14:48.285648 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG dbcounter [-] [81409] Writing DB stats keystone:SELECT=46 {{(pid=81409) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:14:51.300818 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01183470-9acb-44a6-8dcf-e21540674298 None None] REQUEST_METHOD: `GET` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:14:51.301384 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01183470-9acb-44a6-8dcf-e21540674298 None None] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:14:51.301384 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01183470-9acb-44a6-8dcf-e21540674298 None None] PATH_INFO: `/` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:14:51.302739 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 533/1087] 10.222.0.27 () {54 vars in 918 bytes} [Thu Oct 13 04:14:51 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 13 04:14:51.307749 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab682d18-8d3f-484e-9875-fecdc2ac7344 None None] REQUEST_METHOD: `POST` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:14:51.308154 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab682d18-8d3f-484e-9875-fecdc2ac7344 None None] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:14:51.308288 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab682d18-8d3f-484e-9875-fecdc2ac7344 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:14:51.383163 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.auth.core [None req-ab682d18-8d3f-484e-9875-fecdc2ac7344 None None] MFA Rules not processed for user `ba1b6e5868a549a99760ebe6068fd6b9`. Rule list: `[]` (Enabled: `True`). {{(pid=81409) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 13 04:14:51.384381 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.fernet_utils [None req-ab682d18-8d3f-484e-9875-fecdc2ac7344 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81409) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:14:51.493709 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 555/1088] 10.222.0.27 () {58 vars in 1015 bytes} [Thu Oct 13 04:14:51 2022] POST /identity/v3/auth/tokens => generated 3308 bytes in 187 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 13 04:14:51.508751 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5da51752-d9de-4b14-9298-70011dd0e3e8 None None] REQUEST_METHOD: `GET` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:14:51.509036 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5da51752-d9de-4b14-9298-70011dd0e3e8 None None] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:14:51.509308 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5da51752-d9de-4b14-9298-70011dd0e3e8 None None] PATH_INFO: `/` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:14:51.510714 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 534/1089] 10.222.0.27 () {54 vars in 918 bytes} [Thu Oct 13 04:14:51 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 13 04:14:51.516377 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d9723fb-ea67-4e02-b011-9f87d7f68401 None None] REQUEST_METHOD: `POST` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:14:51.516679 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d9723fb-ea67-4e02-b011-9f87d7f68401 None None] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:14:51.516949 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d9723fb-ea67-4e02-b011-9f87d7f68401 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:14:51.587697 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG dbcounter [-] [81409] Writing DB stats keystone:SELECT=100 {{(pid=81409) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:14:51.590862 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.auth.core [None req-3d9723fb-ea67-4e02-b011-9f87d7f68401 None None] MFA Rules not processed for user `cb740ed7f7684b079e45f887be83f795`. Rule list: `[]` (Enabled: `True`). {{(pid=81409) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 13 04:14:51.592926 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.fernet_utils [None req-3d9723fb-ea67-4e02-b011-9f87d7f68401 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81409) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:14:51.705686 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 556/1090] 10.222.0.27 () {58 vars in 1015 bytes} [Thu Oct 13 04:14:51 2022] POST /identity/v3/auth/tokens => generated 3372 bytes in 190 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 13 04:14:54.284834 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bbbdd788-8c60-46b6-a3fb-28249d3fd113 None None] REQUEST_METHOD: `GET` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:14:54.285360 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bbbdd788-8c60-46b6-a3fb-28249d3fd113 None None] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:14:54.285360 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bbbdd788-8c60-46b6-a3fb-28249d3fd113 None None] PATH_INFO: `/` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:14:54.286770 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 535/1091] 127.0.0.1 () {54 vars in 917 bytes} [Thu Oct 13 04:14:54 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 13 04:14:54.291931 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1241ca83-8275-49c1-91e0-ec454649dd91 None None] REQUEST_METHOD: `POST` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:14:54.292374 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1241ca83-8275-49c1-91e0-ec454649dd91 None None] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:14:54.292486 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1241ca83-8275-49c1-91e0-ec454649dd91 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:14:54.360469 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.auth.core [None req-1241ca83-8275-49c1-91e0-ec454649dd91 None None] MFA Rules not processed for user `0e2d979563d6421995451a0e86bf3022`. Rule list: `[]` (Enabled: `True`). {{(pid=81409) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 13 04:14:54.361687 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.fernet_utils [None req-1241ca83-8275-49c1-91e0-ec454649dd91 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81409) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:14:54.417461 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG dbcounter [-] [81409] Writing DB stats keystone:SELECT=100 {{(pid=81409) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:14:54.460730 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 557/1092] 127.0.0.1 () {58 vars in 1014 bytes} [Thu Oct 13 04:14:54 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 170 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 13 04:14:54.755447 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bbbdd788-8c60-46b6-a3fb-28249d3fd113 None None] Authenticating user token {{(pid=81408) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:14:54.756959 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.fernet_utils [None req-bbbdd788-8c60-46b6-a3fb-28249d3fd113 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81408) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:14:54.902892 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bbbdd788-8c60-46b6-a3fb-28249d3fd113 None None] Validating token access rules against request {{(pid=81408) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:14:54.975263 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG dbcounter [-] [81408] Writing DB stats keystone:SELECT=100 {{(pid=81408) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:14:55.015445 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6b886575-292a-4a60-8855-8b59896032fe service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cb740ed7f7684b079e45f887be83f795', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dcec0431fb6849499d6aec35a234daa6', 'project_domain_id': 'default', 'roles': ['ResellerAdmin', 'admin', 'reader', 'service', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81408) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:14:55.016046 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b886575-292a-4a60-8855-8b59896032fe service nova] REQUEST_METHOD: `GET` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:14:55.016321 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b886575-292a-4a60-8855-8b59896032fe service nova] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:14:55.016590 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b886575-292a-4a60-8855-8b59896032fe service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:14:55.018173 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.fernet_utils [None req-6b886575-292a-4a60-8855-8b59896032fe service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81408) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:14:55.106407 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6b886575-292a-4a60-8855-8b59896032fe service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:14:55.108123 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6b886575-292a-4a60-8855-8b59896032fe service nova] RBAC: Authorization granted {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:14:55.125665 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 536/1093] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Oct 13 04:14:54 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 371 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 13 04:14:56.268560 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d3b3bd9-096e-48d1-a0c1-7ffe2fa825cf None None] REQUEST_METHOD: `GET` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:14:56.269088 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d3b3bd9-096e-48d1-a0c1-7ffe2fa825cf None None] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:14:56.269209 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d3b3bd9-096e-48d1-a0c1-7ffe2fa825cf None None] PATH_INFO: `/` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:14:56.270584 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 558/1094] 127.0.0.1 () {54 vars in 901 bytes} [Thu Oct 13 04:14:56 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 13 04:14:56.276916 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fba413eb-76ac-4320-8bea-352b858f6da1 None None] REQUEST_METHOD: `POST` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:14:56.277365 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fba413eb-76ac-4320-8bea-352b858f6da1 None None] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:14:56.277463 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fba413eb-76ac-4320-8bea-352b858f6da1 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:14:56.352526 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.auth.core [None req-fba413eb-76ac-4320-8bea-352b858f6da1 None None] MFA Rules not processed for user `0e2d979563d6421995451a0e86bf3022`. Rule list: `[]` (Enabled: `True`). {{(pid=81408) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 13 04:14:56.353728 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.fernet_utils [None req-fba413eb-76ac-4320-8bea-352b858f6da1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81408) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:14:56.382459 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG dbcounter [-] [81408] Writing DB stats keystone:SELECT=100 {{(pid=81408) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:14:56.452803 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 537/1095] 127.0.0.1 () {58 vars in 998 bytes} [Thu Oct 13 04:14:56 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 177 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 13 04:14:56.603916 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0d3b3bd9-096e-48d1-a0c1-7ffe2fa825cf None None] Authenticating user token {{(pid=81409) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:14:56.605335 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.fernet_utils [None req-0d3b3bd9-096e-48d1-a0c1-7ffe2fa825cf None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81409) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:14:56.752334 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0d3b3bd9-096e-48d1-a0c1-7ffe2fa825cf None None] Validating token access rules against request {{(pid=81409) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:14:56.791802 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG dbcounter [-] [81409] Writing DB stats keystone:SELECT=100 {{(pid=81409) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:14:56.865471 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-59de94f0-51e0-4f69-9ee1-1d4a8259094a service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cb740ed7f7684b079e45f887be83f795', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dcec0431fb6849499d6aec35a234daa6', 'project_domain_id': 'default', 'roles': ['ResellerAdmin', 'admin', 'reader', 'service', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81409) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:14:56.866010 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59de94f0-51e0-4f69-9ee1-1d4a8259094a service nova] REQUEST_METHOD: `GET` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:14:56.866318 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59de94f0-51e0-4f69-9ee1-1d4a8259094a service nova] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:14:56.866584 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59de94f0-51e0-4f69-9ee1-1d4a8259094a service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:14:56.868156 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.fernet_utils [None req-59de94f0-51e0-4f69-9ee1-1d4a8259094a service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81409) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:14:56.956547 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-59de94f0-51e0-4f69-9ee1-1d4a8259094a service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:14:56.958126 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-59de94f0-51e0-4f69-9ee1-1d4a8259094a service nova] RBAC: Authorization granted {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:14:56.976264 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 559/1096] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Oct 13 04:14:56 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 373 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 13 04:14:58.110040 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b8ed076-d981-45f6-921b-ab4389877f86 None None] REQUEST_METHOD: `GET` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:14:58.110637 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b8ed076-d981-45f6-921b-ab4389877f86 None None] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:14:58.110637 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b8ed076-d981-45f6-921b-ab4389877f86 None None] PATH_INFO: `/` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:14:58.112014 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 538/1097] 127.0.0.1 () {54 vars in 901 bytes} [Thu Oct 13 04:14:58 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 13 04:14:58.118346 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-524f9cef-5b73-4696-81c7-84aac763381b None None] REQUEST_METHOD: `POST` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:14:58.118804 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-524f9cef-5b73-4696-81c7-84aac763381b None None] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:14:58.118913 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-524f9cef-5b73-4696-81c7-84aac763381b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:14:58.172692 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG dbcounter [-] [81409] Writing DB stats keystone:SELECT=100 {{(pid=81409) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:14:58.194848 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.auth.core [None req-524f9cef-5b73-4696-81c7-84aac763381b None None] MFA Rules not processed for user `0e2d979563d6421995451a0e86bf3022`. Rule list: `[]` (Enabled: `True`). {{(pid=81409) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 13 04:14:58.196048 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.fernet_utils [None req-524f9cef-5b73-4696-81c7-84aac763381b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81409) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:14:58.291383 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 560/1098] 127.0.0.1 () {58 vars in 998 bytes} [Thu Oct 13 04:14:58 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 174 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 13 04:14:58.441844 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6b8ed076-d981-45f6-921b-ab4389877f86 None None] Authenticating user token {{(pid=81408) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:14:58.443402 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.fernet_utils [None req-6b8ed076-d981-45f6-921b-ab4389877f86 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81408) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:14:58.589618 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6b8ed076-d981-45f6-921b-ab4389877f86 None None] Validating token access rules against request {{(pid=81408) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:14:58.602484 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG dbcounter [-] [81408] Writing DB stats keystone:SELECT=100 {{(pid=81408) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:14:58.703447 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cfb0c92e-a799-4a24-b6e4-423b0993c702 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cb740ed7f7684b079e45f887be83f795', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dcec0431fb6849499d6aec35a234daa6', 'project_domain_id': 'default', 'roles': ['ResellerAdmin', 'admin', 'reader', 'service', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81408) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:14:58.704050 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cfb0c92e-a799-4a24-b6e4-423b0993c702 service nova] REQUEST_METHOD: `GET` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:14:58.704327 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cfb0c92e-a799-4a24-b6e4-423b0993c702 service nova] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:14:58.704594 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cfb0c92e-a799-4a24-b6e4-423b0993c702 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:14:58.706220 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.fernet_utils [None req-cfb0c92e-a799-4a24-b6e4-423b0993c702 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81408) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:14:58.798023 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cfb0c92e-a799-4a24-b6e4-423b0993c702 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:14:58.799666 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cfb0c92e-a799-4a24-b6e4-423b0993c702 service nova] RBAC: Authorization granted {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:14:58.817583 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 539/1099] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Oct 13 04:14:58 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 376 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 13 04:14:59.917847 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b076388f-d183-42af-a50c-b004c7e70fb0 None None] REQUEST_METHOD: `GET` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:14:59.918507 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b076388f-d183-42af-a50c-b004c7e70fb0 None None] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:14:59.918507 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b076388f-d183-42af-a50c-b004c7e70fb0 None None] PATH_INFO: `/` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:14:59.920419 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 561/1100] 127.0.0.1 () {54 vars in 901 bytes} [Thu Oct 13 04:14:59 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 13 04:14:59.926740 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81ddca74-4b09-4ce6-884f-8989c5ef4d41 None None] REQUEST_METHOD: `POST` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:14:59.927137 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81ddca74-4b09-4ce6-884f-8989c5ef4d41 None None] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:14:59.927334 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81ddca74-4b09-4ce6-884f-8989c5ef4d41 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:14:59.946891 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG dbcounter [-] [81408] Writing DB stats keystone:SELECT=100 {{(pid=81408) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:15:00.003666 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.auth.core [None req-81ddca74-4b09-4ce6-884f-8989c5ef4d41 None None] MFA Rules not processed for user `0e2d979563d6421995451a0e86bf3022`. Rule list: `[]` (Enabled: `True`). {{(pid=81408) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 13 04:15:00.004851 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.fernet_utils [None req-81ddca74-4b09-4ce6-884f-8989c5ef4d41 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81408) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:15:00.100837 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 540/1101] 127.0.0.1 () {58 vars in 998 bytes} [Thu Oct 13 04:14:59 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 175 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 13 04:15:00.250468 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b076388f-d183-42af-a50c-b004c7e70fb0 None None] Authenticating user token {{(pid=81409) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:15:00.252009 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.fernet_utils [None req-b076388f-d183-42af-a50c-b004c7e70fb0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81409) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:15:00.362866 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG dbcounter [-] [81409] Writing DB stats keystone:SELECT=100 {{(pid=81409) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:15:00.399257 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b076388f-d183-42af-a50c-b004c7e70fb0 None None] Validating token access rules against request {{(pid=81409) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:15:00.512496 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b4e055c9-f4e3-40c0-9621-ba0ec72e0123 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cb740ed7f7684b079e45f887be83f795', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dcec0431fb6849499d6aec35a234daa6', 'project_domain_id': 'default', 'roles': ['ResellerAdmin', 'admin', 'reader', 'service', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81409) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:15:00.513727 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4e055c9-f4e3-40c0-9621-ba0ec72e0123 service nova] REQUEST_METHOD: `GET` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:15:00.514079 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4e055c9-f4e3-40c0-9621-ba0ec72e0123 service nova] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:15:00.514378 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4e055c9-f4e3-40c0-9621-ba0ec72e0123 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:15:00.515998 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.fernet_utils [None req-b4e055c9-f4e3-40c0-9621-ba0ec72e0123 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81409) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:15:00.591944 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG dbcounter [-] [81409] Writing DB stats keystone:SELECT=100 {{(pid=81409) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:15:00.605647 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b4e055c9-f4e3-40c0-9621-ba0ec72e0123 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:15:00.607307 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b4e055c9-f4e3-40c0-9621-ba0ec72e0123 service nova] RBAC: Authorization granted {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:15:00.625389 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 562/1102] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Oct 13 04:15:00 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 375 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 13 04:15:01.736691 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3456a089-f436-4077-8982-175d2fd3472b None None] REQUEST_METHOD: `GET` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:15:01.737228 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3456a089-f436-4077-8982-175d2fd3472b None None] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:15:01.737228 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3456a089-f436-4077-8982-175d2fd3472b None None] PATH_INFO: `/` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:15:01.738528 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 541/1103] 127.0.0.1 () {54 vars in 901 bytes} [Thu Oct 13 04:15:01 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 13 04:15:01.744890 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ca603cf-dffb-4a98-8666-32cf5c95bc77 None None] REQUEST_METHOD: `POST` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:15:01.745369 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ca603cf-dffb-4a98-8666-32cf5c95bc77 None None] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:15:01.745565 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ca603cf-dffb-4a98-8666-32cf5c95bc77 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:15:01.820748 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.auth.core [None req-3ca603cf-dffb-4a98-8666-32cf5c95bc77 None None] MFA Rules not processed for user `0e2d979563d6421995451a0e86bf3022`. Rule list: `[]` (Enabled: `True`). {{(pid=81409) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 13 04:15:01.821962 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.fernet_utils [None req-3ca603cf-dffb-4a98-8666-32cf5c95bc77 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81409) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:15:01.918918 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 563/1104] 127.0.0.1 () {58 vars in 998 bytes} [Thu Oct 13 04:15:01 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 175 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 13 04:15:02.070179 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3456a089-f436-4077-8982-175d2fd3472b None None] Authenticating user token {{(pid=81408) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:15:02.071749 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.fernet_utils [None req-3456a089-f436-4077-8982-175d2fd3472b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81408) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:15:02.155140 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG dbcounter [-] [81408] Writing DB stats keystone:SELECT=100 {{(pid=81408) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:15:02.216266 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3456a089-f436-4077-8982-175d2fd3472b None None] Validating token access rules against request {{(pid=81408) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:15:02.328007 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-113acdc8-ae89-493c-ba1f-7fb1452f6abb service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cb740ed7f7684b079e45f887be83f795', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dcec0431fb6849499d6aec35a234daa6', 'project_domain_id': 'default', 'roles': ['ResellerAdmin', 'admin', 'reader', 'service', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81408) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:15:02.328528 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-113acdc8-ae89-493c-ba1f-7fb1452f6abb service nova] REQUEST_METHOD: `GET` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:15:02.328807 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-113acdc8-ae89-493c-ba1f-7fb1452f6abb service nova] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:15:02.329076 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-113acdc8-ae89-493c-ba1f-7fb1452f6abb service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:15:02.330712 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.fernet_utils [None req-113acdc8-ae89-493c-ba1f-7fb1452f6abb service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81408) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:15:02.381928 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG dbcounter [-] [81408] Writing DB stats keystone:SELECT=100 {{(pid=81408) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:15:02.423648 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-113acdc8-ae89-493c-ba1f-7fb1452f6abb service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:15:02.425243 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-113acdc8-ae89-493c-ba1f-7fb1452f6abb service nova] RBAC: Authorization granted {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:15:02.444218 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 542/1105] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Oct 13 04:15:02 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 375 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 13 04:15:03.507754 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a1ea5a5-d3d5-4d0e-beae-90a1e4e448b0 None None] REQUEST_METHOD: `GET` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:15:03.508357 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a1ea5a5-d3d5-4d0e-beae-90a1e4e448b0 None None] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:15:03.508357 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a1ea5a5-d3d5-4d0e-beae-90a1e4e448b0 None None] PATH_INFO: `/` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:15:03.509627 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 564/1106] 127.0.0.1 () {54 vars in 901 bytes} [Thu Oct 13 04:15:03 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 13 04:15:03.516585 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45b45114-dee5-417e-a57c-cd210329e193 None None] REQUEST_METHOD: `POST` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:15:03.517080 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45b45114-dee5-417e-a57c-cd210329e193 None None] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:15:03.517214 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45b45114-dee5-417e-a57c-cd210329e193 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:15:03.591016 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.auth.core [None req-45b45114-dee5-417e-a57c-cd210329e193 None None] MFA Rules not processed for user `0e2d979563d6421995451a0e86bf3022`. Rule list: `[]` (Enabled: `True`). {{(pid=81408) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 13 04:15:03.592216 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.fernet_utils [None req-45b45114-dee5-417e-a57c-cd210329e193 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81408) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:15:03.687943 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 543/1107] 127.0.0.1 () {58 vars in 998 bytes} [Thu Oct 13 04:15:03 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 172 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 13 04:15:03.836691 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1a1ea5a5-d3d5-4d0e-beae-90a1e4e448b0 None None] Authenticating user token {{(pid=81409) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:15:03.838195 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.fernet_utils [None req-1a1ea5a5-d3d5-4d0e-beae-90a1e4e448b0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81409) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:15:03.889282 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG dbcounter [-] [81409] Writing DB stats keystone:SELECT=100 {{(pid=81409) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:15:03.986070 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1a1ea5a5-d3d5-4d0e-beae-90a1e4e448b0 None None] Validating token access rules against request {{(pid=81409) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:15:04.100660 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-029ad505-cce2-4a97-a6ab-a6ed1a04a2c8 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cb740ed7f7684b079e45f887be83f795', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dcec0431fb6849499d6aec35a234daa6', 'project_domain_id': 'default', 'roles': ['ResellerAdmin', 'admin', 'reader', 'service', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81409) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:15:04.101298 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-029ad505-cce2-4a97-a6ab-a6ed1a04a2c8 service nova] REQUEST_METHOD: `GET` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:15:04.101572 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-029ad505-cce2-4a97-a6ab-a6ed1a04a2c8 service nova] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:15:04.101840 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-029ad505-cce2-4a97-a6ab-a6ed1a04a2c8 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:15:04.103750 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.fernet_utils [None req-029ad505-cce2-4a97-a6ab-a6ed1a04a2c8 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81409) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:15:04.110784 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG dbcounter [-] [81409] Writing DB stats keystone:SELECT=100 {{(pid=81409) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:15:04.193539 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-029ad505-cce2-4a97-a6ab-a6ed1a04a2c8 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:15:04.195152 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-029ad505-cce2-4a97-a6ab-a6ed1a04a2c8 service nova] RBAC: Authorization granted {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:15:04.213012 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 565/1108] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Oct 13 04:15:03 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 377 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 13 04:15:05.271570 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5cdd16a-2cd9-4e27-85d1-2817b155ef94 None None] REQUEST_METHOD: `GET` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:15:05.272183 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5cdd16a-2cd9-4e27-85d1-2817b155ef94 None None] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:15:05.272183 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5cdd16a-2cd9-4e27-85d1-2817b155ef94 None None] PATH_INFO: `/` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:15:05.273456 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 544/1109] 127.0.0.1 () {54 vars in 901 bytes} [Thu Oct 13 04:15:05 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 13 04:15:05.280294 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fafcc582-3d2a-46f9-8c44-b5022d7573f1 None None] REQUEST_METHOD: `POST` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:15:05.280717 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fafcc582-3d2a-46f9-8c44-b5022d7573f1 None None] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:15:05.280819 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fafcc582-3d2a-46f9-8c44-b5022d7573f1 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:15:05.356348 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.auth.core [None req-fafcc582-3d2a-46f9-8c44-b5022d7573f1 None None] MFA Rules not processed for user `0e2d979563d6421995451a0e86bf3022`. Rule list: `[]` (Enabled: `True`). {{(pid=81409) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 13 04:15:05.358184 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.fernet_utils [None req-fafcc582-3d2a-46f9-8c44-b5022d7573f1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81409) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:15:05.426591 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG dbcounter [-] [81409] Writing DB stats keystone:SELECT=100 {{(pid=81409) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:15:05.457089 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 566/1110] 127.0.0.1 () {58 vars in 998 bytes} [Thu Oct 13 04:15:05 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 179 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 13 04:15:05.608279 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f5cdd16a-2cd9-4e27-85d1-2817b155ef94 None None] Authenticating user token {{(pid=81408) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:15:05.609731 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.fernet_utils [None req-f5cdd16a-2cd9-4e27-85d1-2817b155ef94 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81408) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:15:05.627082 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG dbcounter [-] [81408] Writing DB stats keystone:SELECT=100 {{(pid=81408) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:15:05.754165 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f5cdd16a-2cd9-4e27-85d1-2817b155ef94 None None] Validating token access rules against request {{(pid=81408) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:15:05.839686 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG dbcounter [-] [81408] Writing DB stats keystone:SELECT=100 {{(pid=81408) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:15:05.864946 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cf60f7b3-7991-4066-af47-fcbdccc8ad89 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cb740ed7f7684b079e45f887be83f795', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dcec0431fb6849499d6aec35a234daa6', 'project_domain_id': 'default', 'roles': ['ResellerAdmin', 'admin', 'reader', 'service', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81408) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:15:05.865477 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf60f7b3-7991-4066-af47-fcbdccc8ad89 service nova] REQUEST_METHOD: `GET` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:15:05.865753 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf60f7b3-7991-4066-af47-fcbdccc8ad89 service nova] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:15:05.866029 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf60f7b3-7991-4066-af47-fcbdccc8ad89 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:15:05.867650 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.fernet_utils [None req-cf60f7b3-7991-4066-af47-fcbdccc8ad89 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81408) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:15:05.956977 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cf60f7b3-7991-4066-af47-fcbdccc8ad89 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:15:05.958600 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cf60f7b3-7991-4066-af47-fcbdccc8ad89 service nova] RBAC: Authorization granted {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:15:05.976436 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 545/1111] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Oct 13 04:15:05 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 369 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 13 04:15:07.035142 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-933392e1-3d8b-4e39-853b-382a9d88cf30 None None] REQUEST_METHOD: `GET` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:15:07.035721 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-933392e1-3d8b-4e39-853b-382a9d88cf30 None None] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:15:07.035721 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-933392e1-3d8b-4e39-853b-382a9d88cf30 None None] PATH_INFO: `/` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:15:07.037018 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 567/1112] 127.0.0.1 () {54 vars in 901 bytes} [Thu Oct 13 04:15:07 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 13 04:15:07.043607 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d36730b-8385-4810-b11e-7f76060b6288 None None] REQUEST_METHOD: `POST` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:15:07.044075 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d36730b-8385-4810-b11e-7f76060b6288 None None] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:15:07.044463 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d36730b-8385-4810-b11e-7f76060b6288 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:15:07.120726 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.auth.core [None req-3d36730b-8385-4810-b11e-7f76060b6288 None None] MFA Rules not processed for user `0e2d979563d6421995451a0e86bf3022`. Rule list: `[]` (Enabled: `True`). {{(pid=81408) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 13 04:15:07.122040 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.fernet_utils [None req-3d36730b-8385-4810-b11e-7f76060b6288 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81408) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:15:07.163767 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG dbcounter [-] [81408] Writing DB stats keystone:SELECT=100 {{(pid=81408) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:15:07.218512 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 546/1113] 127.0.0.1 () {58 vars in 998 bytes} [Thu Oct 13 04:15:07 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 176 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 13 04:15:07.369750 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-933392e1-3d8b-4e39-853b-382a9d88cf30 None None] Authenticating user token {{(pid=81409) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:15:07.371215 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.fernet_utils [None req-933392e1-3d8b-4e39-853b-382a9d88cf30 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81409) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:15:07.521048 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-933392e1-3d8b-4e39-853b-382a9d88cf30 None None] Validating token access rules against request {{(pid=81409) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:15:07.573374 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG dbcounter [-] [81409] Writing DB stats keystone:SELECT=100 {{(pid=81409) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:15:07.632135 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1c9f7b8d-f750-4f5c-85ab-669642523fd9 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cb740ed7f7684b079e45f887be83f795', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dcec0431fb6849499d6aec35a234daa6', 'project_domain_id': 'default', 'roles': ['ResellerAdmin', 'admin', 'reader', 'service', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81409) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:15:07.632670 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c9f7b8d-f750-4f5c-85ab-669642523fd9 service nova] REQUEST_METHOD: `GET` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:15:07.632946 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c9f7b8d-f750-4f5c-85ab-669642523fd9 service nova] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:15:07.633216 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c9f7b8d-f750-4f5c-85ab-669642523fd9 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:15:07.634839 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.fernet_utils [None req-1c9f7b8d-f750-4f5c-85ab-669642523fd9 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81409) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:15:07.722471 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1c9f7b8d-f750-4f5c-85ab-669642523fd9 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:15:07.724055 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1c9f7b8d-f750-4f5c-85ab-669642523fd9 service nova] RBAC: Authorization granted {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:15:07.742393 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 568/1114] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Oct 13 04:15:07 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 373 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 13 04:15:08.790594 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e84cd574-9807-454b-ad25-55d366a43f63 None None] REQUEST_METHOD: `GET` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:15:08.791234 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e84cd574-9807-454b-ad25-55d366a43f63 None None] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:15:08.791234 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e84cd574-9807-454b-ad25-55d366a43f63 None None] PATH_INFO: `/` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:15:08.792496 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 547/1115] 127.0.0.1 () {54 vars in 901 bytes} [Thu Oct 13 04:15:08 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 13 04:15:08.799466 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22522e30-ea9b-42dc-81df-f4691a359bda None None] REQUEST_METHOD: `POST` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:15:08.799904 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22522e30-ea9b-42dc-81df-f4691a359bda None None] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:15:08.800010 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22522e30-ea9b-42dc-81df-f4691a359bda None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:15:08.874620 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.auth.core [None req-22522e30-ea9b-42dc-81df-f4691a359bda None None] MFA Rules not processed for user `0e2d979563d6421995451a0e86bf3022`. Rule list: `[]` (Enabled: `True`). {{(pid=81409) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 13 04:15:08.875810 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.fernet_utils [None req-22522e30-ea9b-42dc-81df-f4691a359bda None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81409) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:15:08.877811 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG dbcounter [-] [81409] Writing DB stats keystone:SELECT=100 {{(pid=81409) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:15:08.972058 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 569/1116] 127.0.0.1 () {58 vars in 998 bytes} [Thu Oct 13 04:15:08 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 174 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 13 04:15:09.120968 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e84cd574-9807-454b-ad25-55d366a43f63 None None] Authenticating user token {{(pid=81408) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:15:09.122484 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.fernet_utils [None req-e84cd574-9807-454b-ad25-55d366a43f63 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81408) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:15:09.266842 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e84cd574-9807-454b-ad25-55d366a43f63 None None] Validating token access rules against request {{(pid=81408) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:15:09.293086 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG dbcounter [-] [81408] Writing DB stats keystone:SELECT=100 {{(pid=81408) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:15:09.379549 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-61faf942-3837-4915-bd35-98b09f94d016 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cb740ed7f7684b079e45f887be83f795', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dcec0431fb6849499d6aec35a234daa6', 'project_domain_id': 'default', 'roles': ['ResellerAdmin', 'admin', 'reader', 'service', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81408) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:15:09.380067 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61faf942-3837-4915-bd35-98b09f94d016 service nova] REQUEST_METHOD: `GET` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:15:09.380340 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61faf942-3837-4915-bd35-98b09f94d016 service nova] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:15:09.380603 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61faf942-3837-4915-bd35-98b09f94d016 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:15:09.382198 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.fernet_utils [None req-61faf942-3837-4915-bd35-98b09f94d016 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81408) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:15:09.471253 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-61faf942-3837-4915-bd35-98b09f94d016 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:15:09.472848 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-61faf942-3837-4915-bd35-98b09f94d016 service nova] RBAC: Authorization granted {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:15:09.491312 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 548/1117] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Oct 13 04:15:09 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 371 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 13 04:15:11.564345 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f4f6f51-03f8-482b-b24d-6ef3792aff1a None None] REQUEST_METHOD: `GET` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:15:11.564900 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f4f6f51-03f8-482b-b24d-6ef3792aff1a None None] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:15:11.564900 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f4f6f51-03f8-482b-b24d-6ef3792aff1a None None] PATH_INFO: `/` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:15:11.566224 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 570/1118] 127.0.0.1 () {54 vars in 917 bytes} [Thu Oct 13 04:15:11 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 13 04:15:11.571483 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69418381-a618-4841-af96-ad0e6c79ab96 None None] REQUEST_METHOD: `POST` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:15:11.571897 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69418381-a618-4841-af96-ad0e6c79ab96 None None] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:15:11.572026 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69418381-a618-4841-af96-ad0e6c79ab96 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:15:11.613068 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG dbcounter [-] [81408] Writing DB stats keystone:SELECT=100 {{(pid=81408) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:15:11.639221 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.auth.core [None req-69418381-a618-4841-af96-ad0e6c79ab96 None None] MFA Rules not processed for user `0e2d979563d6421995451a0e86bf3022`. Rule list: `[]` (Enabled: `True`). {{(pid=81408) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 13 04:15:11.640418 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.fernet_utils [None req-69418381-a618-4841-af96-ad0e6c79ab96 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81408) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:15:11.734556 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 549/1119] 127.0.0.1 () {58 vars in 1014 bytes} [Thu Oct 13 04:15:11 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 164 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 13 04:15:12.020643 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4f4f6f51-03f8-482b-b24d-6ef3792aff1a None None] Authenticating user token {{(pid=81409) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:15:12.022226 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.fernet_utils [None req-4f4f6f51-03f8-482b-b24d-6ef3792aff1a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81409) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:15:12.128071 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4f4f6f51-03f8-482b-b24d-6ef3792aff1a None None] Validating token access rules against request {{(pid=81409) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:15:12.159783 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG dbcounter [-] [81409] Writing DB stats keystone:SELECT=100 {{(pid=81409) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:15:12.201425 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c78b836c-8a69-4bd7-a441-ecf867ca2da4 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd8812c3974f84871bb2d761cf0016257', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dcec0431fb6849499d6aec35a234daa6', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81409) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:15:12.202022 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c78b836c-8a69-4bd7-a441-ecf867ca2da4 service neutron] REQUEST_METHOD: `GET` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:15:12.202335 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c78b836c-8a69-4bd7-a441-ecf867ca2da4 service neutron] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:15:12.202609 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c78b836c-8a69-4bd7-a441-ecf867ca2da4 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:15:12.204183 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.fernet_utils [None req-c78b836c-8a69-4bd7-a441-ecf867ca2da4 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81409) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:15:12.311582 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c78b836c-8a69-4bd7-a441-ecf867ca2da4 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:15:12.312923 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c78b836c-8a69-4bd7-a441-ecf867ca2da4 service neutron] RBAC: Authorization granted {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:15:12.330835 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 571/1120] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Oct 13 04:15:12 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 311 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 13 04:15:14.642424 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2fdf43a1-a314-417c-bd6e-b12edeeb2210 None None] REQUEST_METHOD: `GET` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:15:14.642938 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2fdf43a1-a314-417c-bd6e-b12edeeb2210 None None] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:15:14.642938 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2fdf43a1-a314-417c-bd6e-b12edeeb2210 None None] PATH_INFO: `/` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:15:14.644234 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 550/1121] 127.0.0.1 () {54 vars in 917 bytes} [Thu Oct 13 04:15:14 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 13 04:15:14.649438 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bebce497-8fab-498c-9cab-1ba1bec60a13 None None] REQUEST_METHOD: `POST` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:15:14.649833 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bebce497-8fab-498c-9cab-1ba1bec60a13 None None] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:15:14.649973 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bebce497-8fab-498c-9cab-1ba1bec60a13 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:15:14.716346 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.auth.core [None req-bebce497-8fab-498c-9cab-1ba1bec60a13 None None] MFA Rules not processed for user `0e2d979563d6421995451a0e86bf3022`. Rule list: `[]` (Enabled: `True`). {{(pid=81409) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 13 04:15:14.717579 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.fernet_utils [None req-bebce497-8fab-498c-9cab-1ba1bec60a13 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81409) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:15:14.730627 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG dbcounter [-] [81409] Writing DB stats keystone:SELECT=100 {{(pid=81409) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:15:14.818134 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 572/1122] 127.0.0.1 () {58 vars in 1014 bytes} [Thu Oct 13 04:15:14 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 170 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 13 04:15:15.104742 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2fdf43a1-a314-417c-bd6e-b12edeeb2210 None None] Authenticating user token {{(pid=81408) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:15:15.107046 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.fernet_utils [None req-2fdf43a1-a314-417c-bd6e-b12edeeb2210 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81408) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:15:15.208319 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2fdf43a1-a314-417c-bd6e-b12edeeb2210 None None] Validating token access rules against request {{(pid=81408) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:15:15.221293 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG dbcounter [-] [81408] Writing DB stats keystone:SELECT=100 {{(pid=81408) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:15:15.282724 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ef0e4525-9a66-4b08-9c56-c8e97fceb8af service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd8812c3974f84871bb2d761cf0016257', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dcec0431fb6849499d6aec35a234daa6', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81408) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:15:15.283326 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef0e4525-9a66-4b08-9c56-c8e97fceb8af service neutron] REQUEST_METHOD: `GET` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:15:15.283604 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef0e4525-9a66-4b08-9c56-c8e97fceb8af service neutron] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:15:15.283874 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef0e4525-9a66-4b08-9c56-c8e97fceb8af service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:15:15.285479 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.fernet_utils [None req-ef0e4525-9a66-4b08-9c56-c8e97fceb8af service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81408) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:15:15.393444 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ef0e4525-9a66-4b08-9c56-c8e97fceb8af service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:15:15.394869 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ef0e4525-9a66-4b08-9c56-c8e97fceb8af service neutron] RBAC: Authorization granted {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:15:15.413586 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 551/1123] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Oct 13 04:15:15 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 309 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 13 04:15:17.915347 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1735a1e9-2b5a-4a7a-ba4c-af2ee97193e4 None None] REQUEST_METHOD: `GET` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:15:17.915887 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1735a1e9-2b5a-4a7a-ba4c-af2ee97193e4 None None] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:15:17.915887 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1735a1e9-2b5a-4a7a-ba4c-af2ee97193e4 None None] PATH_INFO: `/` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:15:17.917239 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 573/1124] 127.0.0.1 () {54 vars in 917 bytes} [Thu Oct 13 04:15:17 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 13 04:15:17.922818 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e860fb1-83bd-4c72-ac97-a13f68d89ec1 None None] REQUEST_METHOD: `POST` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:15:17.923228 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e860fb1-83bd-4c72-ac97-a13f68d89ec1 None None] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:15:17.923355 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e860fb1-83bd-4c72-ac97-a13f68d89ec1 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:15:17.971766 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG dbcounter [-] [81408] Writing DB stats keystone:SELECT=100 {{(pid=81408) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:15:17.989559 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.auth.core [None req-4e860fb1-83bd-4c72-ac97-a13f68d89ec1 None None] MFA Rules not processed for user `0e2d979563d6421995451a0e86bf3022`. Rule list: `[]` (Enabled: `True`). {{(pid=81408) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 13 04:15:17.990786 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.fernet_utils [None req-4e860fb1-83bd-4c72-ac97-a13f68d89ec1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81408) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:15:18.086890 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 552/1125] 127.0.0.1 () {58 vars in 1014 bytes} [Thu Oct 13 04:15:17 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 165 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 13 04:15:18.371525 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1735a1e9-2b5a-4a7a-ba4c-af2ee97193e4 None None] Authenticating user token {{(pid=81409) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:15:18.374012 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.fernet_utils [None req-1735a1e9-2b5a-4a7a-ba4c-af2ee97193e4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81409) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:15:18.476733 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1735a1e9-2b5a-4a7a-ba4c-af2ee97193e4 None None] Validating token access rules against request {{(pid=81409) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:15:18.516939 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG dbcounter [-] [81409] Writing DB stats keystone:SELECT=100 {{(pid=81409) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:15:18.551045 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d2b5a98f-03af-476a-8835-24929702539e service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd8812c3974f84871bb2d761cf0016257', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dcec0431fb6849499d6aec35a234daa6', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81409) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:15:18.551640 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2b5a98f-03af-476a-8835-24929702539e service neutron] REQUEST_METHOD: `GET` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:15:18.551912 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2b5a98f-03af-476a-8835-24929702539e service neutron] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:15:18.552177 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2b5a98f-03af-476a-8835-24929702539e service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:15:18.553754 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.fernet_utils [None req-d2b5a98f-03af-476a-8835-24929702539e service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81409) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:15:18.660843 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d2b5a98f-03af-476a-8835-24929702539e service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:15:18.662194 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d2b5a98f-03af-476a-8835-24929702539e service neutron] RBAC: Authorization granted {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:15:18.680389 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 574/1126] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Oct 13 04:15:18 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 309 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 13 04:15:20.959865 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d88e1176-3a0c-4eaf-a4cc-6a4655261930 None None] REQUEST_METHOD: `GET` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:15:20.960424 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d88e1176-3a0c-4eaf-a4cc-6a4655261930 None None] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:15:20.960424 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d88e1176-3a0c-4eaf-a4cc-6a4655261930 None None] PATH_INFO: `/` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:15:20.961761 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 553/1127] 127.0.0.1 () {54 vars in 917 bytes} [Thu Oct 13 04:15:20 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 13 04:15:20.968077 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-572db375-1a92-4537-88e7-b3b2f0c7c803 None None] REQUEST_METHOD: `POST` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:15:20.968574 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-572db375-1a92-4537-88e7-b3b2f0c7c803 None None] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:15:20.968658 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-572db375-1a92-4537-88e7-b3b2f0c7c803 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:15:21.034160 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.auth.core [None req-572db375-1a92-4537-88e7-b3b2f0c7c803 None None] MFA Rules not processed for user `0e2d979563d6421995451a0e86bf3022`. Rule list: `[]` (Enabled: `True`). {{(pid=81409) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 13 04:15:21.035369 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.fernet_utils [None req-572db375-1a92-4537-88e7-b3b2f0c7c803 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81409) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:15:21.056292 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG dbcounter [-] [81409] Writing DB stats keystone:SELECT=100 {{(pid=81409) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:15:21.133731 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 575/1128] 127.0.0.1 () {58 vars in 1014 bytes} [Thu Oct 13 04:15:20 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 167 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 13 04:15:21.425304 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d88e1176-3a0c-4eaf-a4cc-6a4655261930 None None] Authenticating user token {{(pid=81408) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:15:21.426869 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.fernet_utils [None req-d88e1176-3a0c-4eaf-a4cc-6a4655261930 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81408) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:15:21.530877 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d88e1176-3a0c-4eaf-a4cc-6a4655261930 None None] Validating token access rules against request {{(pid=81408) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:15:21.551863 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG dbcounter [-] [81408] Writing DB stats keystone:SELECT=100 {{(pid=81408) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:15:21.605324 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4f6272fa-32fe-4204-ab2c-a7dd61881759 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd8812c3974f84871bb2d761cf0016257', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dcec0431fb6849499d6aec35a234daa6', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81408) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:15:21.606495 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f6272fa-32fe-4204-ab2c-a7dd61881759 service neutron] REQUEST_METHOD: `GET` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:15:21.606746 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f6272fa-32fe-4204-ab2c-a7dd61881759 service neutron] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:15:21.607013 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f6272fa-32fe-4204-ab2c-a7dd61881759 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:15:21.608607 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.fernet_utils [None req-4f6272fa-32fe-4204-ab2c-a7dd61881759 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81408) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:15:21.715976 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4f6272fa-32fe-4204-ab2c-a7dd61881759 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:15:21.717333 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4f6272fa-32fe-4204-ab2c-a7dd61881759 service neutron] RBAC: Authorization granted {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:15:21.737192 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 554/1129] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Oct 13 04:15:21 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 312 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 13 04:15:23.956510 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61d92ed5-774a-4c10-9b14-930a6e5409a6 None None] REQUEST_METHOD: `GET` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:15:23.957071 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61d92ed5-774a-4c10-9b14-930a6e5409a6 None None] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:15:23.957071 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61d92ed5-774a-4c10-9b14-930a6e5409a6 None None] PATH_INFO: `/` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:15:23.958460 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 576/1130] 127.0.0.1 () {54 vars in 917 bytes} [Thu Oct 13 04:15:23 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 13 04:15:23.963929 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f9f5cac-925a-42ef-a57a-e7c9ed0b11eb None None] REQUEST_METHOD: `POST` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:15:23.963929 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f9f5cac-925a-42ef-a57a-e7c9ed0b11eb None None] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:15:23.964356 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f9f5cac-925a-42ef-a57a-e7c9ed0b11eb None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:15:24.027049 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG dbcounter [-] [81408] Writing DB stats keystone:SELECT=100 {{(pid=81408) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:15:24.030624 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.auth.core [None req-9f9f5cac-925a-42ef-a57a-e7c9ed0b11eb None None] MFA Rules not processed for user `0e2d979563d6421995451a0e86bf3022`. Rule list: `[]` (Enabled: `True`). {{(pid=81408) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 13 04:15:24.032405 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.fernet_utils [None req-9f9f5cac-925a-42ef-a57a-e7c9ed0b11eb None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81408) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:15:24.129494 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 555/1131] 127.0.0.1 () {58 vars in 1014 bytes} [Thu Oct 13 04:15:23 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 167 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 13 04:15:24.414112 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-61d92ed5-774a-4c10-9b14-930a6e5409a6 None None] Authenticating user token {{(pid=81409) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:15:24.415670 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.fernet_utils [None req-61d92ed5-774a-4c10-9b14-930a6e5409a6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81409) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:15:24.520559 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-61d92ed5-774a-4c10-9b14-930a6e5409a6 None None] Validating token access rules against request {{(pid=81409) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:15:24.570225 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG dbcounter [-] [81409] Writing DB stats keystone:SELECT=100 {{(pid=81409) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:15:24.594741 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c7688c70-41c8-4b49-85c7-eb4721964397 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd8812c3974f84871bb2d761cf0016257', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dcec0431fb6849499d6aec35a234daa6', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81409) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:15:24.595337 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7688c70-41c8-4b49-85c7-eb4721964397 service neutron] REQUEST_METHOD: `GET` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:15:24.595603 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7688c70-41c8-4b49-85c7-eb4721964397 service neutron] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:15:24.595875 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7688c70-41c8-4b49-85c7-eb4721964397 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:15:24.597456 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.fernet_utils [None req-c7688c70-41c8-4b49-85c7-eb4721964397 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81409) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:15:24.703801 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c7688c70-41c8-4b49-85c7-eb4721964397 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:15:24.705112 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c7688c70-41c8-4b49-85c7-eb4721964397 service neutron] RBAC: Authorization granted {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:15:24.723168 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 577/1132] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Oct 13 04:15:24 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 310 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 13 04:15:26.922904 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45b031a6-ba5e-4987-93f7-6face8628044 None None] REQUEST_METHOD: `GET` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:15:26.923477 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45b031a6-ba5e-4987-93f7-6face8628044 None None] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:15:26.923477 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45b031a6-ba5e-4987-93f7-6face8628044 None None] PATH_INFO: `/` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:15:26.924823 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 556/1133] 127.0.0.1 () {54 vars in 917 bytes} [Thu Oct 13 04:15:26 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 13 04:15:26.930010 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-375d4803-badd-4a27-9250-15b0ae15ac1f None None] REQUEST_METHOD: `POST` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:15:26.930418 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-375d4803-badd-4a27-9250-15b0ae15ac1f None None] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:15:26.930589 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-375d4803-badd-4a27-9250-15b0ae15ac1f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:15:26.995700 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.auth.core [None req-375d4803-badd-4a27-9250-15b0ae15ac1f None None] MFA Rules not processed for user `0e2d979563d6421995451a0e86bf3022`. Rule list: `[]` (Enabled: `True`). {{(pid=81409) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 13 04:15:26.997448 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.fernet_utils [None req-375d4803-badd-4a27-9250-15b0ae15ac1f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81409) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:15:27.026561 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG dbcounter [-] [81409] Writing DB stats keystone:SELECT=100 {{(pid=81409) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:15:27.094843 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 578/1134] 127.0.0.1 () {58 vars in 1014 bytes} [Thu Oct 13 04:15:26 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 166 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 13 04:15:27.382661 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-45b031a6-ba5e-4987-93f7-6face8628044 None None] Authenticating user token {{(pid=81408) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:15:27.384194 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.fernet_utils [None req-45b031a6-ba5e-4987-93f7-6face8628044 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81408) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:15:27.490106 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-45b031a6-ba5e-4987-93f7-6face8628044 None None] Validating token access rules against request {{(pid=81408) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:15:27.518075 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG dbcounter [-] [81408] Writing DB stats keystone:SELECT=100 {{(pid=81408) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:15:27.562756 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aa508b80-ce6a-4473-bbf9-02a9d59609e9 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd8812c3974f84871bb2d761cf0016257', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dcec0431fb6849499d6aec35a234daa6', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81408) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:15:27.563353 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa508b80-ce6a-4473-bbf9-02a9d59609e9 service neutron] REQUEST_METHOD: `GET` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:15:27.563628 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa508b80-ce6a-4473-bbf9-02a9d59609e9 service neutron] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:15:27.563907 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa508b80-ce6a-4473-bbf9-02a9d59609e9 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:15:27.565510 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.fernet_utils [None req-aa508b80-ce6a-4473-bbf9-02a9d59609e9 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81408) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:15:27.674807 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aa508b80-ce6a-4473-bbf9-02a9d59609e9 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:15:27.676169 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aa508b80-ce6a-4473-bbf9-02a9d59609e9 service neutron] RBAC: Authorization granted {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:15:27.693719 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 557/1135] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Oct 13 04:15:27 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 312 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 13 04:15:29.943987 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-545b1f2f-3dec-415b-bcfc-73f391ec4346 None None] REQUEST_METHOD: `GET` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:15:29.944524 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-545b1f2f-3dec-415b-bcfc-73f391ec4346 None None] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:15:29.944524 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-545b1f2f-3dec-415b-bcfc-73f391ec4346 None None] PATH_INFO: `/` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:15:29.945872 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 579/1136] 127.0.0.1 () {54 vars in 917 bytes} [Thu Oct 13 04:15:29 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 13 04:15:29.951142 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cdacc1ba-faec-40c0-ba00-f992c012ce0e None None] REQUEST_METHOD: `POST` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:15:29.951574 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cdacc1ba-faec-40c0-ba00-f992c012ce0e None None] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:15:29.951683 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cdacc1ba-faec-40c0-ba00-f992c012ce0e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:15:30.017441 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.auth.core [None req-cdacc1ba-faec-40c0-ba00-f992c012ce0e None None] MFA Rules not processed for user `0e2d979563d6421995451a0e86bf3022`. Rule list: `[]` (Enabled: `True`). {{(pid=81408) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 13 04:15:30.018691 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.fernet_utils [None req-cdacc1ba-faec-40c0-ba00-f992c012ce0e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81408) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:15:30.026753 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG dbcounter [-] [81408] Writing DB stats keystone:SELECT=100 {{(pid=81408) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:15:30.116700 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 558/1137] 127.0.0.1 () {58 vars in 1014 bytes} [Thu Oct 13 04:15:29 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 167 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 13 04:15:30.402217 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-545b1f2f-3dec-415b-bcfc-73f391ec4346 None None] Authenticating user token {{(pid=81409) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:15:30.403928 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.fernet_utils [None req-545b1f2f-3dec-415b-bcfc-73f391ec4346 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81409) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:15:30.507598 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-545b1f2f-3dec-415b-bcfc-73f391ec4346 None None] Validating token access rules against request {{(pid=81409) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:15:30.564566 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG dbcounter [-] [81409] Writing DB stats keystone:SELECT=100 {{(pid=81409) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:15:30.582276 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4a1cfaa9-abff-497b-97b5-ad9c3598ef7c service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd8812c3974f84871bb2d761cf0016257', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dcec0431fb6849499d6aec35a234daa6', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81409) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:15:30.582900 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a1cfaa9-abff-497b-97b5-ad9c3598ef7c service neutron] REQUEST_METHOD: `GET` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:15:30.583168 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a1cfaa9-abff-497b-97b5-ad9c3598ef7c service neutron] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:15:30.583458 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a1cfaa9-abff-497b-97b5-ad9c3598ef7c service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:15:30.585021 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.fernet_utils [None req-4a1cfaa9-abff-497b-97b5-ad9c3598ef7c service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81409) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:15:30.693715 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4a1cfaa9-abff-497b-97b5-ad9c3598ef7c service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:15:30.695098 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4a1cfaa9-abff-497b-97b5-ad9c3598ef7c service neutron] RBAC: Authorization granted {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:15:30.712907 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 580/1138] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Oct 13 04:15:30 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 311 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 13 04:15:32.911849 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1225a18a-5a1a-4d4d-8934-6063fdd935ed None None] REQUEST_METHOD: `GET` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:15:32.912437 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1225a18a-5a1a-4d4d-8934-6063fdd935ed None None] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:15:32.912437 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1225a18a-5a1a-4d4d-8934-6063fdd935ed None None] PATH_INFO: `/` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:15:32.913720 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 559/1139] 127.0.0.1 () {54 vars in 917 bytes} [Thu Oct 13 04:15:32 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 13 04:15:32.919715 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-394935cd-f16e-451e-84da-29242b433292 None None] REQUEST_METHOD: `POST` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:15:32.920138 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-394935cd-f16e-451e-84da-29242b433292 None None] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:15:32.920261 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-394935cd-f16e-451e-84da-29242b433292 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:15:32.986039 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.auth.core [None req-394935cd-f16e-451e-84da-29242b433292 None None] MFA Rules not processed for user `0e2d979563d6421995451a0e86bf3022`. Rule list: `[]` (Enabled: `True`). {{(pid=81409) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 13 04:15:32.987270 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.fernet_utils [None req-394935cd-f16e-451e-84da-29242b433292 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81409) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:15:33.024969 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG dbcounter [-] [81409] Writing DB stats keystone:SELECT=100 {{(pid=81409) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:15:33.084646 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 581/1140] 127.0.0.1 () {58 vars in 1014 bytes} [Thu Oct 13 04:15:32 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 167 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 13 04:15:33.372636 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1225a18a-5a1a-4d4d-8934-6063fdd935ed None None] Authenticating user token {{(pid=81408) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:15:33.374230 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.fernet_utils [None req-1225a18a-5a1a-4d4d-8934-6063fdd935ed None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81408) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:15:33.478226 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1225a18a-5a1a-4d4d-8934-6063fdd935ed None None] Validating token access rules against request {{(pid=81408) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:15:33.515545 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG dbcounter [-] [81408] Writing DB stats keystone:SELECT=100 {{(pid=81408) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:15:33.555376 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3d798646-882a-45e0-8a19-bdbab972f317 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd8812c3974f84871bb2d761cf0016257', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dcec0431fb6849499d6aec35a234daa6', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81408) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:15:33.555985 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d798646-882a-45e0-8a19-bdbab972f317 service neutron] REQUEST_METHOD: `GET` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:15:33.556262 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d798646-882a-45e0-8a19-bdbab972f317 service neutron] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:15:33.556532 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d798646-882a-45e0-8a19-bdbab972f317 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:15:33.558138 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.fernet_utils [None req-3d798646-882a-45e0-8a19-bdbab972f317 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81408) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:15:33.666263 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3d798646-882a-45e0-8a19-bdbab972f317 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:15:33.667666 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3d798646-882a-45e0-8a19-bdbab972f317 service neutron] RBAC: Authorization granted {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:15:33.685764 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 560/1141] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Oct 13 04:15:33 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 314 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 13 04:15:35.846650 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5591df25-ebee-4050-b175-867d33505069 None None] REQUEST_METHOD: `GET` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:15:35.847234 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5591df25-ebee-4050-b175-867d33505069 None None] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:15:35.847234 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5591df25-ebee-4050-b175-867d33505069 None None] PATH_INFO: `/` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:15:35.848513 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 582/1142] 127.0.0.1 () {54 vars in 917 bytes} [Thu Oct 13 04:15:35 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 13 04:15:35.854156 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9295284d-818b-4086-b64d-0fd988a264c7 None None] REQUEST_METHOD: `POST` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:15:35.854582 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9295284d-818b-4086-b64d-0fd988a264c7 None None] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:15:35.854716 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9295284d-818b-4086-b64d-0fd988a264c7 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:15:35.921201 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.auth.core [None req-9295284d-818b-4086-b64d-0fd988a264c7 None None] MFA Rules not processed for user `0e2d979563d6421995451a0e86bf3022`. Rule list: `[]` (Enabled: `True`). {{(pid=81408) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 13 04:15:35.922424 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.fernet_utils [None req-9295284d-818b-4086-b64d-0fd988a264c7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81408) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:15:35.940143 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG dbcounter [-] [81408] Writing DB stats keystone:SELECT=100 {{(pid=81408) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:15:36.018786 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 561/1143] 127.0.0.1 () {58 vars in 1014 bytes} [Thu Oct 13 04:15:35 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 166 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 13 04:15:36.309647 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5591df25-ebee-4050-b175-867d33505069 None None] Authenticating user token {{(pid=81409) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:15:36.311216 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.fernet_utils [None req-5591df25-ebee-4050-b175-867d33505069 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81409) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:15:36.415371 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5591df25-ebee-4050-b175-867d33505069 None None] Validating token access rules against request {{(pid=81409) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:15:36.485027 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG dbcounter [-] [81409] Writing DB stats keystone:SELECT=100 {{(pid=81409) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:15:36.491946 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b5aa054c-6152-4bb1-a37c-6f09809ad4dd service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd8812c3974f84871bb2d761cf0016257', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dcec0431fb6849499d6aec35a234daa6', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81409) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:15:36.492539 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5aa054c-6152-4bb1-a37c-6f09809ad4dd service neutron] REQUEST_METHOD: `GET` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:15:36.492808 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5aa054c-6152-4bb1-a37c-6f09809ad4dd service neutron] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:15:36.493336 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5aa054c-6152-4bb1-a37c-6f09809ad4dd service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:15:36.494944 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.fernet_utils [None req-b5aa054c-6152-4bb1-a37c-6f09809ad4dd service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81409) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:15:36.602502 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b5aa054c-6152-4bb1-a37c-6f09809ad4dd service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:15:36.604123 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b5aa054c-6152-4bb1-a37c-6f09809ad4dd service neutron] RBAC: Authorization granted {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:15:36.622577 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 583/1144] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Oct 13 04:15:36 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 314 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 13 04:15:38.774975 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89fe370f-5b5c-4d24-99d5-167bf1d87a12 None None] REQUEST_METHOD: `GET` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:15:38.775526 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89fe370f-5b5c-4d24-99d5-167bf1d87a12 None None] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:15:38.775526 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89fe370f-5b5c-4d24-99d5-167bf1d87a12 None None] PATH_INFO: `/` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:15:38.776866 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 562/1145] 127.0.0.1 () {54 vars in 917 bytes} [Thu Oct 13 04:15:38 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 13 04:15:38.782727 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-237ac5e9-7f3f-42be-b075-c34965251e14 None None] REQUEST_METHOD: `POST` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:15:38.783154 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-237ac5e9-7f3f-42be-b075-c34965251e14 None None] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:15:38.783267 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-237ac5e9-7f3f-42be-b075-c34965251e14 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:15:38.848155 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.auth.core [None req-237ac5e9-7f3f-42be-b075-c34965251e14 None None] MFA Rules not processed for user `0e2d979563d6421995451a0e86bf3022`. Rule list: `[]` (Enabled: `True`). {{(pid=81409) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 13 04:15:38.849361 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.fernet_utils [None req-237ac5e9-7f3f-42be-b075-c34965251e14 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81409) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:15:38.898939 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG dbcounter [-] [81409] Writing DB stats keystone:SELECT=100 {{(pid=81409) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:15:38.947641 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 584/1146] 127.0.0.1 () {58 vars in 1014 bytes} [Thu Oct 13 04:15:38 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 166 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 13 04:15:39.231929 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-89fe370f-5b5c-4d24-99d5-167bf1d87a12 None None] Authenticating user token {{(pid=81408) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:15:39.233484 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.fernet_utils [None req-89fe370f-5b5c-4d24-99d5-167bf1d87a12 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81408) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:15:39.336506 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-89fe370f-5b5c-4d24-99d5-167bf1d87a12 None None] Validating token access rules against request {{(pid=81408) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:15:39.385667 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG dbcounter [-] [81408] Writing DB stats keystone:SELECT=100 {{(pid=81408) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:15:39.413437 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-428d8aee-ad6e-4b67-8a58-a4928cfdad7a service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd8812c3974f84871bb2d761cf0016257', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dcec0431fb6849499d6aec35a234daa6', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81408) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:15:39.414041 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-428d8aee-ad6e-4b67-8a58-a4928cfdad7a service neutron] REQUEST_METHOD: `GET` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:15:39.414346 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-428d8aee-ad6e-4b67-8a58-a4928cfdad7a service neutron] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:15:39.414614 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-428d8aee-ad6e-4b67-8a58-a4928cfdad7a service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:15:39.416215 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.fernet_utils [None req-428d8aee-ad6e-4b67-8a58-a4928cfdad7a service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81408) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:15:39.524330 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-428d8aee-ad6e-4b67-8a58-a4928cfdad7a service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:15:39.525775 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-428d8aee-ad6e-4b67-8a58-a4928cfdad7a service neutron] RBAC: Authorization granted {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:15:39.543993 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 563/1147] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Oct 13 04:15:39 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 313 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 13 04:15:48.946090 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG dbcounter [-] [81409] Writing DB stats keystone:SELECT=22 {{(pid=81409) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:15:49.541371 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG dbcounter [-] [81408] Writing DB stats keystone:SELECT=66 {{(pid=81408) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:15:59.816528 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d051f88-3213-4818-a794-e79e618d7fa6 None None] REQUEST_METHOD: `GET` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:15:59.817054 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d051f88-3213-4818-a794-e79e618d7fa6 None None] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:15:59.817054 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d051f88-3213-4818-a794-e79e618d7fa6 None None] PATH_INFO: `/` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:15:59.818461 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 585/1148] 10.222.0.27 () {54 vars in 927 bytes} [Thu Oct 13 04:15:59 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Oct 13 04:15:59.823885 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10395ad4-a85b-4fc1-820d-9baed03a1df9 None None] REQUEST_METHOD: `POST` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:15:59.824554 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10395ad4-a85b-4fc1-820d-9baed03a1df9 None None] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:15:59.824710 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10395ad4-a85b-4fc1-820d-9baed03a1df9 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:15:59.890791 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.auth.core [None req-10395ad4-a85b-4fc1-820d-9baed03a1df9 None None] MFA Rules not processed for user `0e2d979563d6421995451a0e86bf3022`. Rule list: `[]` (Enabled: `True`). {{(pid=81408) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 13 04:15:59.891988 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.fernet_utils [None req-10395ad4-a85b-4fc1-820d-9baed03a1df9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81408) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:15:59.987346 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 564/1149] 10.222.0.27 () {58 vars in 1024 bytes} [Thu Oct 13 04:15:59 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 164 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 13 04:16:00.291026 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49e49f9b-b684-4f48-96ff-5d7b53f86591 None None] REQUEST_METHOD: `GET` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:16:00.291325 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49e49f9b-b684-4f48-96ff-5d7b53f86591 None None] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:16:00.291575 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49e49f9b-b684-4f48-96ff-5d7b53f86591 None None] PATH_INFO: `/` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:16:00.292918 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 586/1150] 10.222.0.27 () {54 vars in 962 bytes} [Thu Oct 13 04:16:00 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 13 04:16:00.298220 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0cc21de1-0429-40b6-885e-c876e1876577 None None] REQUEST_METHOD: `POST` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:16:00.298514 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0cc21de1-0429-40b6-885e-c876e1876577 None None] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:16:00.298789 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0cc21de1-0429-40b6-885e-c876e1876577 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:16:00.374961 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.auth.core [None req-0cc21de1-0429-40b6-885e-c876e1876577 None None] MFA Rules not processed for user `a6700f49b86543f6a5b92cad8faaba42`. Rule list: `[]` (Enabled: `True`). {{(pid=81408) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 13 04:16:00.376181 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.fernet_utils [None req-0cc21de1-0429-40b6-885e-c876e1876577 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81408) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:16:00.396783 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG dbcounter [-] [81408] Writing DB stats keystone:SELECT=100 {{(pid=81408) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:16:00.454365 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 565/1151] 10.222.0.27 () {58 vars in 1059 bytes} [Thu Oct 13 04:16:00 2022] POST /identity/v3/auth/tokens => generated 3120 bytes in 157 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 13 04:16:00.461944 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-49e49f9b-b684-4f48-96ff-5d7b53f86591 None None] Authenticating user token {{(pid=81409) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:16:00.463467 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.fernet_utils [None req-49e49f9b-b684-4f48-96ff-5d7b53f86591 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81409) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:16:00.567621 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-49e49f9b-b684-4f48-96ff-5d7b53f86591 None None] Validating token access rules against request {{(pid=81409) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:16:00.639964 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a0492734-d415-4005-911d-3b3e782c2224 service glance] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a6700f49b86543f6a5b92cad8faaba42', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dcec0431fb6849499d6aec35a234daa6', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81409) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:16:00.640554 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0492734-d415-4005-911d-3b3e782c2224 service glance] REQUEST_METHOD: `GET` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:16:00.640822 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0492734-d415-4005-911d-3b3e782c2224 service glance] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:16:00.641086 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0492734-d415-4005-911d-3b3e782c2224 service glance] PATH_INFO: `/v3/auth/tokens` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:16:00.642669 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.fernet_utils [None req-a0492734-d415-4005-911d-3b3e782c2224 service glance] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81409) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:16:00.695857 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG dbcounter [-] [81409] Writing DB stats keystone:SELECT=100 {{(pid=81409) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:16:00.751467 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a0492734-d415-4005-911d-3b3e782c2224 service glance] RBAC: Authorizing `identity:validate_token()` {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:16:00.752888 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a0492734-d415-4005-911d-3b3e782c2224 service glance] RBAC: Authorization granted {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:16:00.770846 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 587/1152] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Oct 13 04:16:00 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 309 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 13 04:16:03.007116 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04ea44bd-4d77-4053-b171-2d57721f76c5 None None] REQUEST_METHOD: `GET` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:16:03.007657 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04ea44bd-4d77-4053-b171-2d57721f76c5 None None] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:16:03.007657 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04ea44bd-4d77-4053-b171-2d57721f76c5 None None] PATH_INFO: `/` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:16:03.008986 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 566/1153] 10.222.0.27 () {54 vars in 927 bytes} [Thu Oct 13 04:16:03 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 13 04:16:03.014999 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d380e425-57d2-4fe5-987f-cba7aab7fb8f None None] REQUEST_METHOD: `POST` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:16:03.015427 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d380e425-57d2-4fe5-987f-cba7aab7fb8f None None] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:16:03.015532 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d380e425-57d2-4fe5-987f-cba7aab7fb8f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:16:03.088036 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.auth.core [None req-d380e425-57d2-4fe5-987f-cba7aab7fb8f None None] MFA Rules not processed for user `0e2d979563d6421995451a0e86bf3022`. Rule list: `[]` (Enabled: `True`). {{(pid=81409) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 13 04:16:03.088895 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.fernet_utils [None req-d380e425-57d2-4fe5-987f-cba7aab7fb8f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81409) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:16:03.188547 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 588/1154] 10.222.0.27 () {58 vars in 1024 bytes} [Thu Oct 13 04:16:03 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 175 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 13 04:16:03.461776 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-04ea44bd-4d77-4053-b171-2d57721f76c5 None None] Authenticating user token {{(pid=81408) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:16:03.463331 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.fernet_utils [None req-04ea44bd-4d77-4053-b171-2d57721f76c5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81408) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:16:03.586347 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-04ea44bd-4d77-4053-b171-2d57721f76c5 None None] Validating token access rules against request {{(pid=81408) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:16:03.634482 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG dbcounter [-] [81408] Writing DB stats keystone:SELECT=100 {{(pid=81408) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:16:03.678497 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-714d9198-8aa6-463d-b82c-abb39b191889 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8ba823f313144da3b683bacef8fd62e0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81408) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:16:03.679022 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-714d9198-8aa6-463d-b82c-abb39b191889 admin admin] REQUEST_METHOD: `GET` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:16:03.679298 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-714d9198-8aa6-463d-b82c-abb39b191889 admin admin] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:16:03.679564 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-714d9198-8aa6-463d-b82c-abb39b191889 admin admin] PATH_INFO: `/v3/projects` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:16:03.680193 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-714d9198-8aa6-463d-b82c-abb39b191889 admin admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:16:03.681296 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: WARNING py.warnings [None req-714d9198-8aa6-463d-b82c-abb39b191889 admin admin] /usr/local/lib/python3.8/dist-packages/oslo_policy/policy.py:1129: UserWarning: Policy "identity:list_projects": "(role:reader and system_scope:all) or (role:reader and domain_id:%(target.domain_id)s)" failed scope check. The token used to make the request was project scoped but the policy requires ['system', 'domain'] scope. This behavior may change in the future where using the intended scope is required Oct 13 04:16:03.681296 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: warnings.warn(msg) Oct 13 04:16:03.681296 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: Oct 13 04:16:03.682455 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-714d9198-8aa6-463d-b82c-abb39b191889 admin admin] RBAC: Authorization granted {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:16:03.702445 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 567/1155] 10.222.0.27 () {56 vars in 1108 bytes} [Thu Oct 13 04:16:03 2022] GET /identity/v3/projects => generated 2553 bytes in 241 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 13 04:16:05.889893 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00e42805-82d8-4538-975b-b31ad1b7f386 None None] REQUEST_METHOD: `GET` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:16:05.890521 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00e42805-82d8-4538-975b-b31ad1b7f386 None None] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:16:05.890521 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00e42805-82d8-4538-975b-b31ad1b7f386 None None] PATH_INFO: `/` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:16:05.891815 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 589/1156] 10.222.0.27 () {54 vars in 927 bytes} [Thu Oct 13 04:16:05 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 13 04:16:05.896995 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d78c3cba-fe39-440c-8fd0-95f545df2fa4 None None] REQUEST_METHOD: `POST` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:16:05.897425 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d78c3cba-fe39-440c-8fd0-95f545df2fa4 None None] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:16:05.897544 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d78c3cba-fe39-440c-8fd0-95f545df2fa4 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:16:05.962553 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.auth.core [None req-d78c3cba-fe39-440c-8fd0-95f545df2fa4 None None] MFA Rules not processed for user `0e2d979563d6421995451a0e86bf3022`. Rule list: `[]` (Enabled: `True`). {{(pid=81408) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 13 04:16:05.963762 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.fernet_utils [None req-d78c3cba-fe39-440c-8fd0-95f545df2fa4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81408) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:16:06.059373 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 568/1157] 10.222.0.27 () {58 vars in 1024 bytes} [Thu Oct 13 04:16:05 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 163 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 13 04:16:06.352024 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-00e42805-82d8-4538-975b-b31ad1b7f386 None None] Authenticating user token {{(pid=81409) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:16:06.353825 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.fernet_utils [None req-00e42805-82d8-4538-975b-b31ad1b7f386 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81409) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:16:06.363832 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG dbcounter [-] [81409] Writing DB stats keystone:SELECT=100 {{(pid=81409) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:16:06.500456 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-00e42805-82d8-4538-975b-b31ad1b7f386 None None] Validating token access rules against request {{(pid=81409) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:16:06.579654 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG dbcounter [-] [81409] Writing DB stats keystone:SELECT=100 {{(pid=81409) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:16:06.614538 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-296e56cf-b21c-42eb-933b-0d398bed446b service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cb740ed7f7684b079e45f887be83f795', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dcec0431fb6849499d6aec35a234daa6', 'project_domain_id': 'default', 'roles': ['ResellerAdmin', 'admin', 'reader', 'service', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81409) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:16:06.615066 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-296e56cf-b21c-42eb-933b-0d398bed446b service nova] REQUEST_METHOD: `GET` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:16:06.615336 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-296e56cf-b21c-42eb-933b-0d398bed446b service nova] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:16:06.615599 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-296e56cf-b21c-42eb-933b-0d398bed446b service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:16:06.617194 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.fernet_utils [None req-296e56cf-b21c-42eb-933b-0d398bed446b service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81409) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:16:06.703911 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-296e56cf-b21c-42eb-933b-0d398bed446b service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:16:06.705490 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-296e56cf-b21c-42eb-933b-0d398bed446b service nova] RBAC: Authorization granted {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:16:06.723860 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 590/1158] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Oct 13 04:16:06 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 372 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 13 04:16:09.061381 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35daec72-f202-4aff-9bc4-2ac31db9bae0 None None] REQUEST_METHOD: `GET` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:16:09.061980 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35daec72-f202-4aff-9bc4-2ac31db9bae0 None None] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:16:09.062087 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35daec72-f202-4aff-9bc4-2ac31db9bae0 None None] PATH_INFO: `/` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:16:09.063415 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 569/1159] 10.222.0.27 () {54 vars in 927 bytes} [Thu Oct 13 04:16:09 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 13 04:16:09.068777 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cca4b290-7f2b-4518-8938-f8a9e8146920 None None] REQUEST_METHOD: `POST` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:16:09.069208 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cca4b290-7f2b-4518-8938-f8a9e8146920 None None] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:16:09.069314 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cca4b290-7f2b-4518-8938-f8a9e8146920 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:16:09.138087 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.auth.core [None req-cca4b290-7f2b-4518-8938-f8a9e8146920 None None] MFA Rules not processed for user `0e2d979563d6421995451a0e86bf3022`. Rule list: `[]` (Enabled: `True`). {{(pid=81409) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 13 04:16:09.139342 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.fernet_utils [None req-cca4b290-7f2b-4518-8938-f8a9e8146920 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81409) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:16:09.185336 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG dbcounter [-] [81409] Writing DB stats keystone:SELECT=100 {{(pid=81409) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:16:09.236164 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 591/1160] 10.222.0.27 () {58 vars in 1024 bytes} [Thu Oct 13 04:16:09 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 168 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 13 04:16:09.524471 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-35daec72-f202-4aff-9bc4-2ac31db9bae0 None None] Authenticating user token {{(pid=81408) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:16:09.526000 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.fernet_utils [None req-35daec72-f202-4aff-9bc4-2ac31db9bae0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81408) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:16:09.549822 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG dbcounter [-] [81408] Writing DB stats keystone:SELECT=100 {{(pid=81408) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:16:09.631310 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-35daec72-f202-4aff-9bc4-2ac31db9bae0 None None] Validating token access rules against request {{(pid=81408) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:16:09.703320 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6c23d8ee-8441-4b16-be05-2cb208bd9161 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd8812c3974f84871bb2d761cf0016257', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dcec0431fb6849499d6aec35a234daa6', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81408) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:16:09.703847 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c23d8ee-8441-4b16-be05-2cb208bd9161 service neutron] REQUEST_METHOD: `GET` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:16:09.704125 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c23d8ee-8441-4b16-be05-2cb208bd9161 service neutron] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:16:09.704397 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c23d8ee-8441-4b16-be05-2cb208bd9161 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:16:09.705989 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.fernet_utils [None req-6c23d8ee-8441-4b16-be05-2cb208bd9161 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81408) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:16:09.782829 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG dbcounter [-] [81408] Writing DB stats keystone:SELECT=100 {{(pid=81408) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:16:09.819665 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6c23d8ee-8441-4b16-be05-2cb208bd9161 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:16:09.821156 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6c23d8ee-8441-4b16-be05-2cb208bd9161 service neutron] RBAC: Authorization granted {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:16:09.847834 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 570/1161] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Oct 13 04:16:09 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 324 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 13 04:16:12.062681 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8193374a-9b33-4710-aa79-ca2e14132c02 None None] REQUEST_METHOD: `GET` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:16:12.063218 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8193374a-9b33-4710-aa79-ca2e14132c02 None None] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:16:12.063218 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8193374a-9b33-4710-aa79-ca2e14132c02 None None] PATH_INFO: `/` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:16:12.064570 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 592/1162] 10.222.0.27 () {54 vars in 927 bytes} [Thu Oct 13 04:16:12 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 13 04:16:12.069790 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39ee52b0-f331-4cf9-851b-c4c4bd82ec2c None None] REQUEST_METHOD: `POST` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:16:12.070183 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39ee52b0-f331-4cf9-851b-c4c4bd82ec2c None None] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:16:12.070383 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39ee52b0-f331-4cf9-851b-c4c4bd82ec2c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:16:12.137448 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.auth.core [None req-39ee52b0-f331-4cf9-851b-c4c4bd82ec2c None None] MFA Rules not processed for user `0e2d979563d6421995451a0e86bf3022`. Rule list: `[]` (Enabled: `True`). {{(pid=81408) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 13 04:16:12.138678 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.fernet_utils [None req-39ee52b0-f331-4cf9-851b-c4c4bd82ec2c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81408) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:16:12.233463 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 571/1163] 10.222.0.27 () {58 vars in 1024 bytes} [Thu Oct 13 04:16:12 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 165 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 13 04:16:12.523675 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8193374a-9b33-4710-aa79-ca2e14132c02 None None] Authenticating user token {{(pid=81409) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:16:12.525246 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.fernet_utils [None req-8193374a-9b33-4710-aa79-ca2e14132c02 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81409) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:16:12.627620 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8193374a-9b33-4710-aa79-ca2e14132c02 None None] Validating token access rules against request {{(pid=81409) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:16:12.699042 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG dbcounter [-] [81409] Writing DB stats keystone:SELECT=100 {{(pid=81409) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:16:12.700746 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0457f214-5924-46e0-a0c7-878a353b46f5 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd8812c3974f84871bb2d761cf0016257', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dcec0431fb6849499d6aec35a234daa6', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81409) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:16:12.701394 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0457f214-5924-46e0-a0c7-878a353b46f5 service neutron] REQUEST_METHOD: `GET` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:16:12.701853 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0457f214-5924-46e0-a0c7-878a353b46f5 service neutron] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:16:12.702400 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0457f214-5924-46e0-a0c7-878a353b46f5 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:16:12.704172 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.fernet_utils [None req-0457f214-5924-46e0-a0c7-878a353b46f5 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81409) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:16:12.813193 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0457f214-5924-46e0-a0c7-878a353b46f5 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:16:12.814665 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0457f214-5924-46e0-a0c7-878a353b46f5 service neutron] RBAC: Authorization granted {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:16:12.832428 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 593/1164] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Oct 13 04:16:12 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 309 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 13 04:16:15.055702 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4646fd2b-1c97-47c2-bf1e-26691c77707c None None] REQUEST_METHOD: `GET` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:16:15.056245 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4646fd2b-1c97-47c2-bf1e-26691c77707c None None] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:16:15.056245 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4646fd2b-1c97-47c2-bf1e-26691c77707c None None] PATH_INFO: `/` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:16:15.057626 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 572/1165] 10.222.0.27 () {54 vars in 927 bytes} [Thu Oct 13 04:16:15 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 13 04:16:15.063476 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e92f180-b5c8-469a-a32d-526351ec8508 None None] REQUEST_METHOD: `POST` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:16:15.063879 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e92f180-b5c8-469a-a32d-526351ec8508 None None] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:16:15.064004 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e92f180-b5c8-469a-a32d-526351ec8508 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:16:15.129409 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.auth.core [None req-5e92f180-b5c8-469a-a32d-526351ec8508 None None] MFA Rules not processed for user `0e2d979563d6421995451a0e86bf3022`. Rule list: `[]` (Enabled: `True`). {{(pid=81409) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 13 04:16:15.130631 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.fernet_utils [None req-5e92f180-b5c8-469a-a32d-526351ec8508 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81409) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:16:15.184823 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG dbcounter [-] [81409] Writing DB stats keystone:SELECT=100 {{(pid=81409) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:16:15.228079 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 594/1166] 10.222.0.27 () {58 vars in 1024 bytes} [Thu Oct 13 04:16:15 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 166 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 13 04:16:15.520714 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4646fd2b-1c97-47c2-bf1e-26691c77707c None None] Authenticating user token {{(pid=81408) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:16:15.522266 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.fernet_utils [None req-4646fd2b-1c97-47c2-bf1e-26691c77707c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81408) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:16:15.559035 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG dbcounter [-] [81408] Writing DB stats keystone:SELECT=100 {{(pid=81408) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:16:15.627694 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4646fd2b-1c97-47c2-bf1e-26691c77707c None None] Validating token access rules against request {{(pid=81408) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:16:15.701171 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eef749ff-462f-4402-b5cc-88c4903bfc3a service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd8812c3974f84871bb2d761cf0016257', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dcec0431fb6849499d6aec35a234daa6', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81408) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:16:15.701700 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eef749ff-462f-4402-b5cc-88c4903bfc3a service neutron] REQUEST_METHOD: `GET` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:16:15.701978 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eef749ff-462f-4402-b5cc-88c4903bfc3a service neutron] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:16:15.702275 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eef749ff-462f-4402-b5cc-88c4903bfc3a service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:16:15.703878 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.fernet_utils [None req-eef749ff-462f-4402-b5cc-88c4903bfc3a service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81408) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:16:15.782789 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG dbcounter [-] [81408] Writing DB stats keystone:SELECT=100 {{(pid=81408) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:16:15.812434 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eef749ff-462f-4402-b5cc-88c4903bfc3a service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:16:15.813868 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eef749ff-462f-4402-b5cc-88c4903bfc3a service neutron] RBAC: Authorization granted {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:16:15.831542 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 573/1167] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Oct 13 04:16:15 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 311 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 13 04:16:18.186604 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0af85090-ed8d-4587-af8d-c72f6f7da7f7 None None] REQUEST_METHOD: `GET` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:16:18.187172 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0af85090-ed8d-4587-af8d-c72f6f7da7f7 None None] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:16:18.187172 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0af85090-ed8d-4587-af8d-c72f6f7da7f7 None None] PATH_INFO: `/` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:16:18.188527 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 595/1168] 10.222.0.27 () {54 vars in 927 bytes} [Thu Oct 13 04:16:18 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Oct 13 04:16:18.193748 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c3fe43cd-0ab6-4fc8-8901-5378e25c93b1 None None] REQUEST_METHOD: `POST` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:16:18.194140 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c3fe43cd-0ab6-4fc8-8901-5378e25c93b1 None None] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:16:18.194330 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c3fe43cd-0ab6-4fc8-8901-5378e25c93b1 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:16:18.259891 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.auth.core [None req-c3fe43cd-0ab6-4fc8-8901-5378e25c93b1 None None] MFA Rules not processed for user `0e2d979563d6421995451a0e86bf3022`. Rule list: `[]` (Enabled: `True`). {{(pid=81408) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 13 04:16:18.261659 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.fernet_utils [None req-c3fe43cd-0ab6-4fc8-8901-5378e25c93b1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81408) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:16:18.356973 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 574/1169] 10.222.0.27 () {58 vars in 1024 bytes} [Thu Oct 13 04:16:18 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 164 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 13 04:16:18.649004 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0af85090-ed8d-4587-af8d-c72f6f7da7f7 None None] Authenticating user token {{(pid=81409) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:16:18.650595 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.fernet_utils [None req-0af85090-ed8d-4587-af8d-c72f6f7da7f7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81409) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:16:18.754486 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0af85090-ed8d-4587-af8d-c72f6f7da7f7 None None] Validating token access rules against request {{(pid=81409) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:16:18.826758 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1b27d03a-097c-4b53-aedc-d019d33b9d2d service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd8812c3974f84871bb2d761cf0016257', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dcec0431fb6849499d6aec35a234daa6', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81409) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:16:18.827358 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b27d03a-097c-4b53-aedc-d019d33b9d2d service neutron] REQUEST_METHOD: `GET` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:16:18.827646 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b27d03a-097c-4b53-aedc-d019d33b9d2d service neutron] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:16:18.827899 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b27d03a-097c-4b53-aedc-d019d33b9d2d service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:16:18.829464 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.fernet_utils [None req-1b27d03a-097c-4b53-aedc-d019d33b9d2d service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81409) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:16:18.837856 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG dbcounter [-] [81409] Writing DB stats keystone:SELECT=100 {{(pid=81409) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:16:18.939728 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1b27d03a-097c-4b53-aedc-d019d33b9d2d service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:16:18.941065 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1b27d03a-097c-4b53-aedc-d019d33b9d2d service neutron] RBAC: Authorization granted {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:16:18.959109 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 596/1170] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Oct 13 04:16:18 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 311 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 13 04:16:28.355411 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG dbcounter [-] [81408] Writing DB stats keystone:SELECT=82 {{(pid=81408) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:16:28.956882 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG dbcounter [-] [81409] Writing DB stats keystone:SELECT=50 {{(pid=81409) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:17:32.211835 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3beaed7-245f-41de-b072-4bc7e89bf2d7 None None] REQUEST_METHOD: `POST` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:17:32.212435 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3beaed7-245f-41de-b072-4bc7e89bf2d7 None None] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:17:32.212435 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3beaed7-245f-41de-b072-4bc7e89bf2d7 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:17:32.288183 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.auth.core [None req-a3beaed7-245f-41de-b072-4bc7e89bf2d7 None None] MFA Rules not processed for user `0e2d979563d6421995451a0e86bf3022`. Rule list: `[]` (Enabled: `True`). {{(pid=81408) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 13 04:17:32.289309 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.fernet_utils [None req-a3beaed7-245f-41de-b072-4bc7e89bf2d7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81408) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:17:32.388772 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 575/1171] 10.222.0.27 () {58 vars in 958 bytes} [Thu Oct 13 04:17:32 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 178 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 13 04:17:32.396667 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3aa9821c-96eb-439e-a305-1575c6f1e4a4 None None] REQUEST_METHOD: `POST` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:17:32.397137 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3aa9821c-96eb-439e-a305-1575c6f1e4a4 None None] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:17:32.397247 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3aa9821c-96eb-439e-a305-1575c6f1e4a4 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:17:32.472959 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.auth.core [None req-3aa9821c-96eb-439e-a305-1575c6f1e4a4 None None] MFA Rules not processed for user `0e2d979563d6421995451a0e86bf3022`. Rule list: `[]` (Enabled: `True`). {{(pid=81409) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 13 04:17:32.474151 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.fernet_utils [None req-3aa9821c-96eb-439e-a305-1575c6f1e4a4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81409) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:17:32.569649 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 597/1172] 10.222.0.27 () {58 vars in 958 bytes} [Thu Oct 13 04:17:32 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 174 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 13 04:17:32.574383 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a3beaed7-245f-41de-b072-4bc7e89bf2d7 None None] Authenticating user token {{(pid=81408) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:17:32.575580 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.fernet_utils [None req-a3beaed7-245f-41de-b072-4bc7e89bf2d7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81408) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:17:32.586950 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a3beaed7-245f-41de-b072-4bc7e89bf2d7 None None] Validating token access rules against request {{(pid=81408) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:17:32.647338 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG dbcounter [-] [81408] Writing DB stats keystone:SELECT=100 {{(pid=81408) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:17:32.681415 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c6286495-e238-4cae-84c6-1cffaf3c0f95 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8ba823f313144da3b683bacef8fd62e0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81408) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:17:32.682074 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6286495-e238-4cae-84c6-1cffaf3c0f95 admin admin] REQUEST_METHOD: `GET` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:17:32.682374 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6286495-e238-4cae-84c6-1cffaf3c0f95 admin admin] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:17:32.682644 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6286495-e238-4cae-84c6-1cffaf3c0f95 admin admin] PATH_INFO: `/v3/domains` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:17:32.683273 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c6286495-e238-4cae-84c6-1cffaf3c0f95 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81408) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 13 04:17:32.683566 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c6286495-e238-4cae-84c6-1cffaf3c0f95 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:17:32.684564 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: WARNING py.warnings [None req-c6286495-e238-4cae-84c6-1cffaf3c0f95 admin admin] /usr/local/lib/python3.8/dist-packages/oslo_policy/policy.py:1129: UserWarning: Policy "identity:list_domains": "role:reader and system_scope:all" failed scope check. The token used to make the request was project scoped but the policy requires ['system'] scope. This behavior may change in the future where using the intended scope is required Oct 13 04:17:32.684564 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: warnings.warn(msg) Oct 13 04:17:32.684564 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: Oct 13 04:17:32.685440 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c6286495-e238-4cae-84c6-1cffaf3c0f95 admin admin] RBAC: Authorization granted {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:17:32.696229 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 576/1173] 10.222.0.27 () {58 vars in 1153 bytes} [Thu Oct 13 04:17:32 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 122 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 13 04:17:32.706187 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3aa9821c-96eb-439e-a305-1575c6f1e4a4 None None] Authenticating user token {{(pid=81409) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:17:32.716009 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3aa9821c-96eb-439e-a305-1575c6f1e4a4 None None] Validating token access rules against request {{(pid=81409) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:17:32.777095 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG dbcounter [-] [81409] Writing DB stats keystone:SELECT=100 {{(pid=81409) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:17:32.810294 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4dac2d93-30a2-4416-b79a-2fd508c90dc7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8ba823f313144da3b683bacef8fd62e0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81409) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:17:32.810891 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4dac2d93-30a2-4416-b79a-2fd508c90dc7 admin admin] REQUEST_METHOD: `POST` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:17:32.811162 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4dac2d93-30a2-4416-b79a-2fd508c90dc7 admin admin] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:17:32.811428 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4dac2d93-30a2-4416-b79a-2fd508c90dc7 admin admin] PATH_INFO: `/v3/projects` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:17:32.812061 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4dac2d93-30a2-4416-b79a-2fd508c90dc7 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:17:32.813150 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: WARNING py.warnings [None req-4dac2d93-30a2-4416-b79a-2fd508c90dc7 admin admin] /usr/local/lib/python3.8/dist-packages/oslo_policy/policy.py:1129: UserWarning: Policy "identity:create_project": "(role:admin and system_scope:all) or (role:admin and domain_id:%(target.project.domain_id)s)" failed scope check. The token used to make the request was project scoped but the policy requires ['system', 'domain'] scope. This behavior may change in the future where using the intended scope is required Oct 13 04:17:32.813150 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: warnings.warn(msg) Oct 13 04:17:32.813150 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: Oct 13 04:17:32.814359 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4dac2d93-30a2-4416-b79a-2fd508c90dc7 admin admin] RBAC: Authorization granted {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:17:32.870630 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 598/1174] 10.222.0.27 () {60 vars in 1153 bytes} [Thu Oct 13 04:17:32 2022] POST /identity/v3/projects => generated 377 bytes in 165 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 13 04:17:32.874758 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c6286495-e238-4cae-84c6-1cffaf3c0f95 admin admin] Authenticating user token {{(pid=81408) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:17:32.876315 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.fernet_utils [None req-c6286495-e238-4cae-84c6-1cffaf3c0f95 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81408) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:17:32.909615 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c6286495-e238-4cae-84c6-1cffaf3c0f95 admin admin] Validating token access rules against request {{(pid=81408) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:17:33.003232 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e2f1440d-9f39-49c0-94c5-71e788c6e807 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8ba823f313144da3b683bacef8fd62e0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81408) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:17:33.003834 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2f1440d-9f39-49c0-94c5-71e788c6e807 admin admin] REQUEST_METHOD: `POST` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:17:33.004111 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2f1440d-9f39-49c0-94c5-71e788c6e807 admin admin] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:17:33.004381 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2f1440d-9f39-49c0-94c5-71e788c6e807 admin admin] PATH_INFO: `/v3/users` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:17:33.005027 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e2f1440d-9f39-49c0-94c5-71e788c6e807 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:17:33.006043 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: WARNING py.warnings [None req-e2f1440d-9f39-49c0-94c5-71e788c6e807 admin admin] /usr/local/lib/python3.8/dist-packages/oslo_policy/policy.py:1129: UserWarning: Policy "identity:create_user": "(role:admin and system_scope:all) or (role:admin and token.domain.id:%(target.user.domain_id)s)" failed scope check. The token used to make the request was project scoped but the policy requires ['system', 'domain'] scope. This behavior may change in the future where using the intended scope is required Oct 13 04:17:33.006043 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: warnings.warn(msg) Oct 13 04:17:33.006043 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: Oct 13 04:17:33.007285 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e2f1440d-9f39-49c0-94c5-71e788c6e807 admin admin] RBAC: Authorization granted {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:17:33.049142 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 577/1175] 10.222.0.27 () {60 vars in 1144 bytes} [Thu Oct 13 04:17:32 2022] POST /identity/v3/users => generated 289 bytes in 175 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 13 04:17:33.053362 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4dac2d93-30a2-4416-b79a-2fd508c90dc7 admin admin] Authenticating user token {{(pid=81409) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:17:33.054896 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.fernet_utils [None req-4dac2d93-30a2-4416-b79a-2fd508c90dc7 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81409) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:17:33.117276 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4dac2d93-30a2-4416-b79a-2fd508c90dc7 admin admin] Validating token access rules against request {{(pid=81409) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:17:33.204063 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG dbcounter [-] [81409] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81409) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:17:33.216358 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2e067134-f37b-48d8-b9f1-f59a1c8549ca admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8ba823f313144da3b683bacef8fd62e0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81409) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:17:33.216967 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e067134-f37b-48d8-b9f1-f59a1c8549ca admin admin] REQUEST_METHOD: `GET` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:17:33.217242 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e067134-f37b-48d8-b9f1-f59a1c8549ca admin admin] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:17:33.217511 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e067134-f37b-48d8-b9f1-f59a1c8549ca admin admin] PATH_INFO: `/v3/roles` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:17:33.218145 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2e067134-f37b-48d8-b9f1-f59a1c8549ca admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:17:33.219776 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2e067134-f37b-48d8-b9f1-f59a1c8549ca admin admin] RBAC: Authorization granted {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:17:33.232848 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 599/1176] 10.222.0.27 () {58 vars in 1122 bytes} [Thu Oct 13 04:17:33 2022] GET /identity/v3/roles => generated 1420 bytes in 180 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 13 04:17:33.236915 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e2f1440d-9f39-49c0-94c5-71e788c6e807 admin admin] Authenticating user token {{(pid=81408) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:17:33.238452 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.fernet_utils [None req-e2f1440d-9f39-49c0-94c5-71e788c6e807 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81408) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:17:33.249927 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e2f1440d-9f39-49c0-94c5-71e788c6e807 admin admin] Validating token access rules against request {{(pid=81408) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:17:33.274590 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG dbcounter [-] [81408] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81408) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:17:33.345805 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8e6873cb-45ee-474a-adaa-fbe3dc6bf965 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8ba823f313144da3b683bacef8fd62e0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81408) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:17:33.346448 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e6873cb-45ee-474a-adaa-fbe3dc6bf965 admin admin] REQUEST_METHOD: `GET` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:17:33.346720 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e6873cb-45ee-474a-adaa-fbe3dc6bf965 admin admin] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:17:33.346983 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e6873cb-45ee-474a-adaa-fbe3dc6bf965 admin admin] PATH_INFO: `/v3/roles` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:17:33.347611 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8e6873cb-45ee-474a-adaa-fbe3dc6bf965 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:17:33.349192 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8e6873cb-45ee-474a-adaa-fbe3dc6bf965 admin admin] RBAC: Authorization granted {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:17:33.362458 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 578/1177] 10.222.0.27 () {58 vars in 1122 bytes} [Thu Oct 13 04:17:33 2022] GET /identity/v3/roles => generated 1420 bytes in 126 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 13 04:17:33.366645 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2e067134-f37b-48d8-b9f1-f59a1c8549ca admin admin] Authenticating user token {{(pid=81409) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:17:33.368202 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.fernet_utils [None req-2e067134-f37b-48d8-b9f1-f59a1c8549ca admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81409) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:17:33.402015 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2e067134-f37b-48d8-b9f1-f59a1c8549ca admin admin] Validating token access rules against request {{(pid=81409) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:17:33.496394 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-73c7ba53-025e-4054-8c42-d13696be687e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8ba823f313144da3b683bacef8fd62e0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81409) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:17:33.496967 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-73c7ba53-025e-4054-8c42-d13696be687e admin admin] REQUEST_METHOD: `PUT` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:17:33.497236 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-73c7ba53-025e-4054-8c42-d13696be687e admin admin] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:17:33.497498 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-73c7ba53-025e-4054-8c42-d13696be687e admin admin] PATH_INFO: `/v3/projects/a7138f762b16473fb1b1de62028b8231/users/14b621bb47c74bf0a0670ad326fe76f5/roles/cc56c4ae13a7424ea771c15da4b153c1` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:17:33.519744 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-73c7ba53-025e-4054-8c42-d13696be687e admin admin] RBAC: Authorizing `identity:create_grant(project_id=a7138f762b16473fb1b1de62028b8231, user_id=14b621bb47c74bf0a0670ad326fe76f5, role_id=cc56c4ae13a7424ea771c15da4b153c1)` {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:17:33.522412 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-73c7ba53-025e-4054-8c42-d13696be687e admin admin] RBAC: Authorization granted {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:17:33.547592 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 600/1178] 10.222.0.27 () {60 vars in 1483 bytes} [Thu Oct 13 04:17:33 2022] PUT /identity/v3/projects/a7138f762b16473fb1b1de62028b8231/users/14b621bb47c74bf0a0670ad326fe76f5/roles/cc56c4ae13a7424ea771c15da4b153c1 => generated 0 bytes in 181 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 13 04:17:33.559600 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e0cfabe-6484-4a25-aedb-da7ed1cb27b6 None None] REQUEST_METHOD: `POST` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:17:33.559883 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e0cfabe-6484-4a25-aedb-da7ed1cb27b6 None None] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:17:33.560153 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e0cfabe-6484-4a25-aedb-da7ed1cb27b6 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:17:33.617969 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.auth.core [None req-6e0cfabe-6484-4a25-aedb-da7ed1cb27b6 None None] MFA Rules not processed for user `14b621bb47c74bf0a0670ad326fe76f5`. Rule list: `[]` (Enabled: `True`). {{(pid=81408) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 13 04:17:33.619939 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.fernet_utils [None req-6e0cfabe-6484-4a25-aedb-da7ed1cb27b6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81408) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:17:33.696063 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 579/1179] 10.222.0.27 () {58 vars in 958 bytes} [Thu Oct 13 04:17:33 2022] POST /identity/v3/auth/tokens => generated 3254 bytes in 137 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 13 04:17:33.709692 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70e518bf-ff91-4379-bbfd-e5ed19bc587a None None] REQUEST_METHOD: `GET` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:17:33.709974 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70e518bf-ff91-4379-bbfd-e5ed19bc587a None None] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:17:33.710284 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70e518bf-ff91-4379-bbfd-e5ed19bc587a None None] PATH_INFO: `/` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:17:33.711646 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 601/1180] 10.222.0.27 () {54 vars in 963 bytes} [Thu Oct 13 04:17:33 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 13 04:17:33.716931 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42085478-b3d8-4d03-830b-9d1d601f50c4 None None] REQUEST_METHOD: `POST` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:17:33.717208 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42085478-b3d8-4d03-830b-9d1d601f50c4 None None] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:17:33.717477 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42085478-b3d8-4d03-830b-9d1d601f50c4 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:17:33.747970 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG dbcounter [-] [81408] Writing DB stats keystone:SELECT=100 {{(pid=81408) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:17:33.792083 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.auth.core [None req-42085478-b3d8-4d03-830b-9d1d601f50c4 None None] MFA Rules not processed for user `6141b8a6c3d54e179d6879bb019f4d12`. Rule list: `[]` (Enabled: `True`). {{(pid=81408) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 13 04:17:33.793276 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.fernet_utils [None req-42085478-b3d8-4d03-830b-9d1d601f50c4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81408) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:17:33.867039 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 580/1181] 10.222.0.27 () {58 vars in 1060 bytes} [Thu Oct 13 04:17:33 2022] POST /identity/v3/auth/tokens => generated 3120 bytes in 151 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 13 04:17:33.875378 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-70e518bf-ff91-4379-bbfd-e5ed19bc587a None None] Authenticating user token {{(pid=81409) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:17:33.876861 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.fernet_utils [None req-70e518bf-ff91-4379-bbfd-e5ed19bc587a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81409) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:17:33.925945 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG dbcounter [-] [81409] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81409) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:17:33.980583 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-70e518bf-ff91-4379-bbfd-e5ed19bc587a None None] Validating token access rules against request {{(pid=81409) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:17:34.056922 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-73bc2266-7045-47a3-8517-44304e14a599 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6141b8a6c3d54e179d6879bb019f4d12', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dcec0431fb6849499d6aec35a234daa6', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81409) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:17:34.057508 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-73bc2266-7045-47a3-8517-44304e14a599 service cinder] REQUEST_METHOD: `GET` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:17:34.057782 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-73bc2266-7045-47a3-8517-44304e14a599 service cinder] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:17:34.058050 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-73bc2266-7045-47a3-8517-44304e14a599 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:17:34.059666 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.fernet_utils [None req-73bc2266-7045-47a3-8517-44304e14a599 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81409) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:17:34.158822 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-73bc2266-7045-47a3-8517-44304e14a599 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:17:34.160256 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-73bc2266-7045-47a3-8517-44304e14a599 service cinder] RBAC: Authorization granted {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:17:34.163709 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG dbcounter [-] [81409] Writing DB stats keystone:SELECT=100 {{(pid=81409) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:17:34.178743 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 602/1182] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Oct 13 04:17:33 2022] GET /identity/v3/auth/tokens => generated 3254 bytes in 304 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 13 04:17:34.223912 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e80f3512-7d83-4f99-97e9-af6445c27be6 None None] REQUEST_METHOD: `GET` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:17:34.224195 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e80f3512-7d83-4f99-97e9-af6445c27be6 None None] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:17:34.224466 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e80f3512-7d83-4f99-97e9-af6445c27be6 None None] PATH_INFO: `/` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:17:34.225722 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 581/1183] 10.222.0.27 () {52 vars in 830 bytes} [Thu Oct 13 04:17:34 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 13 04:17:34.230115 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-73bc2266-7045-47a3-8517-44304e14a599 service cinder] Authenticating user token {{(pid=81409) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:17:34.231354 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.fernet_utils [None req-73bc2266-7045-47a3-8517-44304e14a599 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81409) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:17:34.332117 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-73bc2266-7045-47a3-8517-44304e14a599 service cinder] Validating token access rules against request {{(pid=81409) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:17:34.425324 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-95ce0f14-1b1f-4506-92fc-9e33b869e8a9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8ba823f313144da3b683bacef8fd62e0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81409) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:17:34.425946 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95ce0f14-1b1f-4506-92fc-9e33b869e8a9 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:17:34.426244 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95ce0f14-1b1f-4506-92fc-9e33b869e8a9 admin admin] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:17:34.426503 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95ce0f14-1b1f-4506-92fc-9e33b869e8a9 admin admin] PATH_INFO: `/v3/users/14b621bb47c74bf0a0670ad326fe76f5` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:17:34.439758 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-95ce0f14-1b1f-4506-92fc-9e33b869e8a9 admin admin] RBAC: Authorizing `identity:delete_user(user_id=14b621bb47c74bf0a0670ad326fe76f5)` {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:17:34.440764 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: WARNING py.warnings [None req-95ce0f14-1b1f-4506-92fc-9e33b869e8a9 admin admin] /usr/local/lib/python3.8/dist-packages/oslo_policy/policy.py:1129: UserWarning: Policy "identity:delete_user": "(role:admin and system_scope:all) or (role:admin and token.domain.id:%(target.user.domain_id)s)" failed scope check. The token used to make the request was project scoped but the policy requires ['system', 'domain'] scope. This behavior may change in the future where using the intended scope is required Oct 13 04:17:34.440764 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: warnings.warn(msg) Oct 13 04:17:34.440764 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: Oct 13 04:17:34.441974 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-95ce0f14-1b1f-4506-92fc-9e33b869e8a9 admin admin] RBAC: Authorization granted {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:17:34.451802 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG dbcounter [-] [81409] Writing DB stats keystone:SELECT=100 {{(pid=81409) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:17:34.505698 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.notifications [None req-95ce0f14-1b1f-4506-92fc-9e33b869e8a9 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '14b621bb47c74bf0a0670ad326fe76f5'} {{(pid=81409) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 13 04:17:34.506135 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.notifications [None req-95ce0f14-1b1f-4506-92fc-9e33b869e8a9 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '14b621bb47c74bf0a0670ad326fe76f5'} {{(pid=81409) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 13 04:17:34.519566 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.notifications [None req-95ce0f14-1b1f-4506-92fc-9e33b869e8a9 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '14b621bb47c74bf0a0670ad326fe76f5'} {{(pid=81409) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 13 04:17:34.520108 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.notifications [None req-95ce0f14-1b1f-4506-92fc-9e33b869e8a9 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '14b621bb47c74bf0a0670ad326fe76f5'} {{(pid=81409) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 13 04:17:34.529172 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.notifications [None req-95ce0f14-1b1f-4506-92fc-9e33b869e8a9 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '14b621bb47c74bf0a0670ad326fe76f5'} {{(pid=81409) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 13 04:17:34.558601 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 603/1184] 10.222.0.27 () {58 vars in 1224 bytes} [Thu Oct 13 04:17:34 2022] DELETE /identity/v3/users/14b621bb47c74bf0a0670ad326fe76f5 => generated 0 bytes in 329 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 13 04:17:34.562864 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16082a0f-f5b5-4a6d-8679-93cc118102a9 None None] REQUEST_METHOD: `POST` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:17:34.563325 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16082a0f-f5b5-4a6d-8679-93cc118102a9 None None] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:17:34.563436 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16082a0f-f5b5-4a6d-8679-93cc118102a9 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:17:34.620117 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.auth.core [None req-16082a0f-f5b5-4a6d-8679-93cc118102a9 None None] MFA Rules not processed for user `0e2d979563d6421995451a0e86bf3022`. Rule list: `[]` (Enabled: `True`). {{(pid=81408) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 13 04:17:34.621304 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.fernet_utils [None req-16082a0f-f5b5-4a6d-8679-93cc118102a9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81408) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:17:34.695334 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG dbcounter [-] [81408] Writing DB stats keystone:SELECT=100 {{(pid=81408) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:17:34.708003 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 582/1185] 10.222.0.27 () {58 vars in 958 bytes} [Thu Oct 13 04:17:34 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 146 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 13 04:17:34.718329 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-95ce0f14-1b1f-4506-92fc-9e33b869e8a9 admin admin] Authenticating user token {{(pid=81409) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:17:34.719565 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.fernet_utils [None req-95ce0f14-1b1f-4506-92fc-9e33b869e8a9 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81409) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:17:34.821263 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-95ce0f14-1b1f-4506-92fc-9e33b869e8a9 admin admin] Validating token access rules against request {{(pid=81409) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:17:34.875561 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG dbcounter [-] [81409] Writing DB stats keystone:DELETE=10,keystone:SELECT=89,keystone:INSERT=1 {{(pid=81409) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:17:34.896266 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-caae6ee3-fd2a-49ef-9c33-7fe3c2495157 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd8812c3974f84871bb2d761cf0016257', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dcec0431fb6849499d6aec35a234daa6', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81409) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:17:34.896865 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-caae6ee3-fd2a-49ef-9c33-7fe3c2495157 service neutron] REQUEST_METHOD: `GET` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:17:34.897137 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-caae6ee3-fd2a-49ef-9c33-7fe3c2495157 service neutron] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:17:34.897401 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-caae6ee3-fd2a-49ef-9c33-7fe3c2495157 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:17:34.899037 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.fernet_utils [None req-caae6ee3-fd2a-49ef-9c33-7fe3c2495157 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81409) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:17:35.006057 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-caae6ee3-fd2a-49ef-9c33-7fe3c2495157 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:17:35.007490 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-caae6ee3-fd2a-49ef-9c33-7fe3c2495157 service neutron] RBAC: Authorization granted {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:17:35.026621 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 604/1186] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Oct 13 04:17:34 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 309 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 13 04:17:35.361195 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-16082a0f-f5b5-4a6d-8679-93cc118102a9 None None] Authenticating user token {{(pid=81408) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:17:35.362527 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.fernet_utils [None req-16082a0f-f5b5-4a6d-8679-93cc118102a9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81408) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:17:35.374672 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-16082a0f-f5b5-4a6d-8679-93cc118102a9 None None] Validating token access rules against request {{(pid=81408) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:17:35.477969 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-40cb5791-25af-4942-9f55-ae0001b80e2c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8ba823f313144da3b683bacef8fd62e0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81408) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:17:35.479375 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40cb5791-25af-4942-9f55-ae0001b80e2c admin admin] REQUEST_METHOD: `DELETE` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:17:35.479904 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40cb5791-25af-4942-9f55-ae0001b80e2c admin admin] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:17:35.480172 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40cb5791-25af-4942-9f55-ae0001b80e2c admin admin] PATH_INFO: `/v3/projects/a7138f762b16473fb1b1de62028b8231` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:17:35.489695 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-40cb5791-25af-4942-9f55-ae0001b80e2c admin admin] RBAC: Authorizing `identity:delete_project(project_id=a7138f762b16473fb1b1de62028b8231)` {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:17:35.490738 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: WARNING py.warnings [None req-40cb5791-25af-4942-9f55-ae0001b80e2c admin admin] /usr/local/lib/python3.8/dist-packages/oslo_policy/policy.py:1129: UserWarning: Policy "identity:delete_project": "(role:admin and system_scope:all) or (role:admin and domain_id:%(target.project.domain_id)s)" failed scope check. The token used to make the request was project scoped but the policy requires ['system', 'domain'] scope. This behavior may change in the future where using the intended scope is required Oct 13 04:17:35.490738 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: warnings.warn(msg) Oct 13 04:17:35.490738 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: Oct 13 04:17:35.491932 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-40cb5791-25af-4942-9f55-ae0001b80e2c admin admin] RBAC: Authorization granted {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:17:35.550519 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.notifications [None req-40cb5791-25af-4942-9f55-ae0001b80e2c admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'a7138f762b16473fb1b1de62028b8231', 'request_id': 'req-40cb5791-25af-4942-9f55-ae0001b80e2c'} {{(pid=81408) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 13 04:17:35.560390 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.notifications [None req-40cb5791-25af-4942-9f55-ae0001b80e2c admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'a7138f762b16473fb1b1de62028b8231', 'request_id': 'req-40cb5791-25af-4942-9f55-ae0001b80e2c'} {{(pid=81408) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 13 04:17:35.575791 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.notifications [None req-40cb5791-25af-4942-9f55-ae0001b80e2c admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'a7138f762b16473fb1b1de62028b8231', 'request_id': 'req-40cb5791-25af-4942-9f55-ae0001b80e2c'} {{(pid=81408) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 13 04:17:35.598555 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.notifications [None req-40cb5791-25af-4942-9f55-ae0001b80e2c admin admin] The token cache is being invalidate because project a7138f762b16473fb1b1de62028b8231 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81408) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 13 04:17:35.598835 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.notifications [None req-40cb5791-25af-4942-9f55-ae0001b80e2c admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81408) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 13 04:17:35.599314 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.notifications [None req-40cb5791-25af-4942-9f55-ae0001b80e2c admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81408) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 13 04:17:35.600960 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 583/1187] 10.222.0.27 () {58 vars in 1233 bytes} [Thu Oct 13 04:17:35 2022] DELETE /identity/v3/projects/a7138f762b16473fb1b1de62028b8231 => generated 0 bytes in 240 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 13 04:17:45.024101 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG dbcounter [-] [81409] Writing DB stats keystone:SELECT=62 {{(pid=81409) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:17:45.596285 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG dbcounter [-] [81408] Writing DB stats keystone:SELECT=84,keystone:DELETE=7,keystone:INSERT=1 {{(pid=81408) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:18:58.739709 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37b7ac24-9cf3-480f-9fbe-8ee661a52d0c None None] REQUEST_METHOD: `GET` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:18:58.740265 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37b7ac24-9cf3-480f-9fbe-8ee661a52d0c None None] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:18:58.740265 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37b7ac24-9cf3-480f-9fbe-8ee661a52d0c None None] PATH_INFO: `/` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:18:58.741526 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 605/1188] 127.0.0.1 () {54 vars in 901 bytes} [Thu Oct 13 04:18:58 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 13 04:18:58.746407 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8067d859-a4eb-4980-9d34-c9fe6d3102c6 None None] REQUEST_METHOD: `POST` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:18:58.746839 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8067d859-a4eb-4980-9d34-c9fe6d3102c6 None None] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:18:58.746951 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8067d859-a4eb-4980-9d34-c9fe6d3102c6 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:18:58.828716 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.auth.core [None req-8067d859-a4eb-4980-9d34-c9fe6d3102c6 None None] MFA Rules not processed for user `0e2d979563d6421995451a0e86bf3022`. Rule list: `[]` (Enabled: `True`). {{(pid=81408) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 13 04:18:58.829924 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.fernet_utils [None req-8067d859-a4eb-4980-9d34-c9fe6d3102c6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81408) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:18:58.930881 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 584/1189] 127.0.0.1 () {58 vars in 998 bytes} [Thu Oct 13 04:18:58 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 186 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 13 04:18:59.085430 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-37b7ac24-9cf3-480f-9fbe-8ee661a52d0c None None] Authenticating user token {{(pid=81409) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:18:59.087533 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.fernet_utils [None req-37b7ac24-9cf3-480f-9fbe-8ee661a52d0c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81409) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:18:59.208542 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-37b7ac24-9cf3-480f-9fbe-8ee661a52d0c None None] Validating token access rules against request {{(pid=81409) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:18:59.215207 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-58c05d20-0bc7-4aea-b518-a2f8a2cdaead service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cb740ed7f7684b079e45f887be83f795', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dcec0431fb6849499d6aec35a234daa6', 'project_domain_id': 'default', 'roles': ['ResellerAdmin', 'admin', 'reader', 'service', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81409) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:18:59.215991 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58c05d20-0bc7-4aea-b518-a2f8a2cdaead service nova] REQUEST_METHOD: `GET` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:18:59.216300 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58c05d20-0bc7-4aea-b518-a2f8a2cdaead service nova] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:18:59.216666 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58c05d20-0bc7-4aea-b518-a2f8a2cdaead service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:18:59.225441 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-58c05d20-0bc7-4aea-b518-a2f8a2cdaead service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:18:59.227053 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-58c05d20-0bc7-4aea-b518-a2f8a2cdaead service nova] RBAC: Authorization granted {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:18:59.237057 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 606/1190] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Oct 13 04:18:59 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 152 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 13 04:19:00.313045 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3eacc8a6-1f4a-4715-9850-293d38ba3a6d None None] REQUEST_METHOD: `GET` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:19:00.313446 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3eacc8a6-1f4a-4715-9850-293d38ba3a6d None None] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:19:00.313591 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3eacc8a6-1f4a-4715-9850-293d38ba3a6d None None] PATH_INFO: `/` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:19:00.314998 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 585/1191] 127.0.0.1 () {54 vars in 901 bytes} [Thu Oct 13 04:19:00 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 13 04:19:00.321078 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-016119c1-5697-42e7-9704-0a9d97038281 None None] REQUEST_METHOD: `POST` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:19:00.321481 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-016119c1-5697-42e7-9704-0a9d97038281 None None] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:19:00.321645 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-016119c1-5697-42e7-9704-0a9d97038281 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:19:00.342257 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.auth.core [None req-016119c1-5697-42e7-9704-0a9d97038281 None None] MFA Rules not processed for user `0e2d979563d6421995451a0e86bf3022`. Rule list: `[]` (Enabled: `True`). {{(pid=81409) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 13 04:19:00.343393 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.fernet_utils [None req-016119c1-5697-42e7-9704-0a9d97038281 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81409) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:19:00.351262 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 607/1192] 127.0.0.1 () {58 vars in 998 bytes} [Thu Oct 13 04:19:00 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 31 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 13 04:19:00.503139 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3eacc8a6-1f4a-4715-9850-293d38ba3a6d None None] Authenticating user token {{(pid=81408) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:19:00.518353 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3eacc8a6-1f4a-4715-9850-293d38ba3a6d None None] Validating token access rules against request {{(pid=81408) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:19:00.523851 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-68589828-91e1-44de-9d3b-0e0e7ed9c9cb service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cb740ed7f7684b079e45f887be83f795', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dcec0431fb6849499d6aec35a234daa6', 'project_domain_id': 'default', 'roles': ['ResellerAdmin', 'admin', 'reader', 'service', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81408) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:19:00.524448 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68589828-91e1-44de-9d3b-0e0e7ed9c9cb service nova] REQUEST_METHOD: `GET` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:19:00.524726 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68589828-91e1-44de-9d3b-0e0e7ed9c9cb service nova] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:19:00.525001 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68589828-91e1-44de-9d3b-0e0e7ed9c9cb service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:19:00.533133 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-68589828-91e1-44de-9d3b-0e0e7ed9c9cb service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:19:00.534759 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-68589828-91e1-44de-9d3b-0e0e7ed9c9cb service nova] RBAC: Authorization granted {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:19:00.544118 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 586/1193] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Oct 13 04:19:00 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 41 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 13 04:19:00.708505 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06110990-df4b-4c70-8db8-e7ba8e08535c None None] REQUEST_METHOD: `GET` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:19:00.708816 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06110990-df4b-4c70-8db8-e7ba8e08535c None None] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:19:00.709091 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06110990-df4b-4c70-8db8-e7ba8e08535c None None] PATH_INFO: `/v3` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:19:00.710411 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 608/1194] 10.222.0.36 () {54 vars in 925 bytes} [Thu Oct 13 04:19:00 2022] GET /identity/v3 => generated 255 bytes in 3 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 13 04:19:00.716630 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e53055a-febc-44bb-90ed-331d6b7a4d87 None None] REQUEST_METHOD: `POST` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:19:00.716914 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e53055a-febc-44bb-90ed-331d6b7a4d87 None None] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:19:00.717182 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e53055a-febc-44bb-90ed-331d6b7a4d87 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:19:00.777280 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.auth.core [None req-4e53055a-febc-44bb-90ed-331d6b7a4d87 None None] MFA Rules not processed for user `ba1b6e5868a549a99760ebe6068fd6b9`. Rule list: `[]` (Enabled: `True`). {{(pid=81408) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 13 04:19:00.778496 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.fernet_utils [None req-4e53055a-febc-44bb-90ed-331d6b7a4d87 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81408) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:19:00.795290 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG dbcounter [-] [81408] Writing DB stats keystone:SELECT=100 {{(pid=81408) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:19:00.850734 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 587/1195] 10.222.0.36 () {58 vars in 1015 bytes} [Thu Oct 13 04:19:00 2022] POST /identity/v3/auth/tokens => generated 3308 bytes in 135 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 13 04:19:09.653492 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9524bdad-326b-48e8-8f0b-a9e1a2521bd9 None None] REQUEST_METHOD: `GET` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:19:09.653950 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9524bdad-326b-48e8-8f0b-a9e1a2521bd9 None None] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:19:09.654075 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9524bdad-326b-48e8-8f0b-a9e1a2521bd9 None None] PATH_INFO: `/` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:19:09.655420 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 609/1196] 10.222.0.27 () {54 vars in 961 bytes} [Thu Oct 13 04:19:09 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 13 04:19:09.660321 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4443df5b-4bd7-4bb1-8f58-6256c717eb53 None None] REQUEST_METHOD: `POST` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:19:09.660764 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4443df5b-4bd7-4bb1-8f58-6256c717eb53 None None] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:19:09.660902 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4443df5b-4bd7-4bb1-8f58-6256c717eb53 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:19:09.681530 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.auth.core [None req-4443df5b-4bd7-4bb1-8f58-6256c717eb53 None None] MFA Rules not processed for user `ba1b6e5868a549a99760ebe6068fd6b9`. Rule list: `[]` (Enabled: `True`). {{(pid=81408) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 13 04:19:09.682697 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.fernet_utils [None req-4443df5b-4bd7-4bb1-8f58-6256c717eb53 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81408) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:19:09.692263 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 588/1197] 10.222.0.27 () {58 vars in 1058 bytes} [Thu Oct 13 04:19:09 2022] POST /identity/v3/auth/tokens => generated 3308 bytes in 33 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 13 04:19:09.699958 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9524bdad-326b-48e8-8f0b-a9e1a2521bd9 None None] Authenticating user token {{(pid=81409) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:19:09.713376 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9524bdad-326b-48e8-8f0b-a9e1a2521bd9 None None] Validating token access rules against request {{(pid=81409) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:19:09.718485 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-adad416f-1c65-4156-bdfd-fa40af3f7a1e service placement] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ba1b6e5868a549a99760ebe6068fd6b9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dcec0431fb6849499d6aec35a234daa6', 'project_domain_id': 'default', 'roles': ['service', 'member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81409) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:19:09.719076 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-adad416f-1c65-4156-bdfd-fa40af3f7a1e service placement] REQUEST_METHOD: `GET` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:19:09.719350 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-adad416f-1c65-4156-bdfd-fa40af3f7a1e service placement] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:19:09.719623 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-adad416f-1c65-4156-bdfd-fa40af3f7a1e service placement] PATH_INFO: `/v3/auth/tokens` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:19:09.727827 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-adad416f-1c65-4156-bdfd-fa40af3f7a1e service placement] RBAC: Authorizing `identity:validate_token()` {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:19:09.729388 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-adad416f-1c65-4156-bdfd-fa40af3f7a1e service placement] RBAC: Authorization granted {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:19:09.739666 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 610/1198] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Oct 13 04:19:09 2022] GET /identity/v3/auth/tokens => generated 3308 bytes in 40 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 13 04:19:12.013556 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21923639-101e-4fc0-8451-7d08672f887f None None] REQUEST_METHOD: `GET` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:19:12.014162 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21923639-101e-4fc0-8451-7d08672f887f None None] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:19:12.014162 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21923639-101e-4fc0-8451-7d08672f887f None None] PATH_INFO: `/` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:19:12.015430 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 589/1199] 127.0.0.1 () {54 vars in 901 bytes} [Thu Oct 13 04:19:12 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 13 04:19:12.021543 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b211ad5-a22c-496e-b82a-c48b8c6330a2 None None] REQUEST_METHOD: `POST` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:19:12.021970 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b211ad5-a22c-496e-b82a-c48b8c6330a2 None None] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:19:12.022107 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b211ad5-a22c-496e-b82a-c48b8c6330a2 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:19:12.041980 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.auth.core [None req-0b211ad5-a22c-496e-b82a-c48b8c6330a2 None None] MFA Rules not processed for user `0e2d979563d6421995451a0e86bf3022`. Rule list: `[]` (Enabled: `True`). {{(pid=81409) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 13 04:19:12.043202 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.fernet_utils [None req-0b211ad5-a22c-496e-b82a-c48b8c6330a2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81409) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:19:12.052684 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 611/1200] 127.0.0.1 () {58 vars in 998 bytes} [Thu Oct 13 04:19:12 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 32 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 13 04:19:12.206362 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-21923639-101e-4fc0-8451-7d08672f887f None None] Authenticating user token {{(pid=81408) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:19:12.221300 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-21923639-101e-4fc0-8451-7d08672f887f None None] Validating token access rules against request {{(pid=81408) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:19:12.226694 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-34dd4961-88e6-44a4-8d8f-c426de03d15c service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cb740ed7f7684b079e45f887be83f795', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dcec0431fb6849499d6aec35a234daa6', 'project_domain_id': 'default', 'roles': ['ResellerAdmin', 'admin', 'reader', 'service', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81408) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:19:12.227285 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34dd4961-88e6-44a4-8d8f-c426de03d15c service nova] REQUEST_METHOD: `GET` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:19:12.227559 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34dd4961-88e6-44a4-8d8f-c426de03d15c service nova] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:19:12.227828 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34dd4961-88e6-44a4-8d8f-c426de03d15c service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:19:12.236704 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-34dd4961-88e6-44a4-8d8f-c426de03d15c service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:19:12.238307 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-34dd4961-88e6-44a4-8d8f-c426de03d15c service nova] RBAC: Authorization granted {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:19:12.247805 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 590/1201] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Oct 13 04:19:12 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 42 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 13 04:19:13.308715 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f7269bc-8c88-44d9-b5a0-95aa47ac985d None None] REQUEST_METHOD: `GET` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:19:13.309282 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f7269bc-8c88-44d9-b5a0-95aa47ac985d None None] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:19:13.309282 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f7269bc-8c88-44d9-b5a0-95aa47ac985d None None] PATH_INFO: `/` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:19:13.310682 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 612/1202] 127.0.0.1 () {54 vars in 901 bytes} [Thu Oct 13 04:19:13 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 13 04:19:13.316805 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8e1523b-e16f-4cbd-afc1-3a55ce51fdb7 None None] REQUEST_METHOD: `POST` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:19:13.317262 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8e1523b-e16f-4cbd-afc1-3a55ce51fdb7 None None] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:19:13.317364 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8e1523b-e16f-4cbd-afc1-3a55ce51fdb7 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:19:13.336950 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.auth.core [None req-d8e1523b-e16f-4cbd-afc1-3a55ce51fdb7 None None] MFA Rules not processed for user `0e2d979563d6421995451a0e86bf3022`. Rule list: `[]` (Enabled: `True`). {{(pid=81408) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 13 04:19:13.338186 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.fernet_utils [None req-d8e1523b-e16f-4cbd-afc1-3a55ce51fdb7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81408) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:19:13.346796 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 591/1203] 127.0.0.1 () {58 vars in 998 bytes} [Thu Oct 13 04:19:13 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 31 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 13 04:19:13.499719 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6f7269bc-8c88-44d9-b5a0-95aa47ac985d None None] Authenticating user token {{(pid=81409) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:19:13.514675 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6f7269bc-8c88-44d9-b5a0-95aa47ac985d None None] Validating token access rules against request {{(pid=81409) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:19:13.520095 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a770371b-41a8-490b-a234-47eae90a98b0 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cb740ed7f7684b079e45f887be83f795', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dcec0431fb6849499d6aec35a234daa6', 'project_domain_id': 'default', 'roles': ['ResellerAdmin', 'admin', 'reader', 'service', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81409) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:19:13.520690 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a770371b-41a8-490b-a234-47eae90a98b0 service nova] REQUEST_METHOD: `GET` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:19:13.520969 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a770371b-41a8-490b-a234-47eae90a98b0 service nova] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:19:13.521242 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a770371b-41a8-490b-a234-47eae90a98b0 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:19:13.529204 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a770371b-41a8-490b-a234-47eae90a98b0 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:19:13.530803 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a770371b-41a8-490b-a234-47eae90a98b0 service nova] RBAC: Authorization granted {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:19:13.540011 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 613/1204] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Oct 13 04:19:13 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 41 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 13 04:19:17.418530 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7fbee561-a488-4bb2-b4c7-38cc7f44d9dd None None] REQUEST_METHOD: `GET` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:19:17.418934 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7fbee561-a488-4bb2-b4c7-38cc7f44d9dd None None] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:19:17.419123 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7fbee561-a488-4bb2-b4c7-38cc7f44d9dd None None] PATH_INFO: `/` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:19:17.421529 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 592/1205] 127.0.0.1 () {54 vars in 917 bytes} [Thu Oct 13 04:19:17 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 13 04:19:17.426821 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-73d95352-cc89-4985-b6a0-f5b9e3c0ccdc None None] REQUEST_METHOD: `POST` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:19:17.427338 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-73d95352-cc89-4985-b6a0-f5b9e3c0ccdc None None] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:19:17.427636 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-73d95352-cc89-4985-b6a0-f5b9e3c0ccdc None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:19:17.447018 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.auth.core [None req-73d95352-cc89-4985-b6a0-f5b9e3c0ccdc None None] MFA Rules not processed for user `0e2d979563d6421995451a0e86bf3022`. Rule list: `[]` (Enabled: `True`). {{(pid=81409) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 13 04:19:17.448221 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.fernet_utils [None req-73d95352-cc89-4985-b6a0-f5b9e3c0ccdc None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81409) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:19:17.456189 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 614/1206] 127.0.0.1 () {58 vars in 1014 bytes} [Thu Oct 13 04:19:17 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 30 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 13 04:19:17.745695 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7fbee561-a488-4bb2-b4c7-38cc7f44d9dd None None] Authenticating user token {{(pid=81408) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:19:17.747455 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.fernet_utils [None req-7fbee561-a488-4bb2-b4c7-38cc7f44d9dd None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81408) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:19:17.824326 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7fbee561-a488-4bb2-b4c7-38cc7f44d9dd None None] Validating token access rules against request {{(pid=81408) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:19:17.828013 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e8f06a1d-7e4e-45fa-bdae-d5819581a56d service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd8812c3974f84871bb2d761cf0016257', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dcec0431fb6849499d6aec35a234daa6', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81408) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:19:17.828611 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8f06a1d-7e4e-45fa-bdae-d5819581a56d service neutron] REQUEST_METHOD: `GET` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:19:17.828886 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8f06a1d-7e4e-45fa-bdae-d5819581a56d service neutron] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:19:17.829157 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8f06a1d-7e4e-45fa-bdae-d5819581a56d service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:19:17.837802 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e8f06a1d-7e4e-45fa-bdae-d5819581a56d service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:19:17.839314 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e8f06a1d-7e4e-45fa-bdae-d5819581a56d service neutron] RBAC: Authorization granted {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:19:17.848812 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 593/1207] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Oct 13 04:19:17 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 104 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 13 04:19:20.010229 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fdcfb124-be36-4436-9303-367ea29ed744 None None] REQUEST_METHOD: `GET` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:19:20.010792 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fdcfb124-be36-4436-9303-367ea29ed744 None None] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:19:20.010792 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fdcfb124-be36-4436-9303-367ea29ed744 None None] PATH_INFO: `/` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:19:20.012173 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 615/1208] 127.0.0.1 () {54 vars in 917 bytes} [Thu Oct 13 04:19:20 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 13 04:19:20.017541 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44043009-6519-4742-8707-33ceba358bde None None] REQUEST_METHOD: `POST` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:19:20.017947 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44043009-6519-4742-8707-33ceba358bde None None] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:19:20.018058 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44043009-6519-4742-8707-33ceba358bde None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:19:20.037479 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.auth.core [None req-44043009-6519-4742-8707-33ceba358bde None None] MFA Rules not processed for user `0e2d979563d6421995451a0e86bf3022`. Rule list: `[]` (Enabled: `True`). {{(pid=81408) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 13 04:19:20.038711 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.fernet_utils [None req-44043009-6519-4742-8707-33ceba358bde None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81408) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:19:20.047275 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 594/1209] 127.0.0.1 () {58 vars in 1014 bytes} [Thu Oct 13 04:19:20 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 31 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 13 04:19:20.339254 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fdcfb124-be36-4436-9303-367ea29ed744 None None] Authenticating user token {{(pid=81409) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:19:20.340953 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.fernet_utils [None req-fdcfb124-be36-4436-9303-367ea29ed744 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81409) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:19:20.354580 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fdcfb124-be36-4436-9303-367ea29ed744 None None] Validating token access rules against request {{(pid=81409) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:19:20.359097 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b958ee27-055d-4c12-bb1e-5382f41179e7 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd8812c3974f84871bb2d761cf0016257', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dcec0431fb6849499d6aec35a234daa6', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81409) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:19:20.359688 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b958ee27-055d-4c12-bb1e-5382f41179e7 service neutron] REQUEST_METHOD: `GET` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:19:20.359960 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b958ee27-055d-4c12-bb1e-5382f41179e7 service neutron] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:19:20.360238 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b958ee27-055d-4c12-bb1e-5382f41179e7 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:19:20.369049 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b958ee27-055d-4c12-bb1e-5382f41179e7 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:19:20.370493 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b958ee27-055d-4c12-bb1e-5382f41179e7 service neutron] RBAC: Authorization granted {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:19:20.380240 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 616/1210] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Oct 13 04:19:20 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 42 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 13 04:19:24.022487 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74fd36d8-041a-4b18-9a00-a0394b347f64 None None] REQUEST_METHOD: `GET` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:19:24.023022 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74fd36d8-041a-4b18-9a00-a0394b347f64 None None] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:19:24.023022 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74fd36d8-041a-4b18-9a00-a0394b347f64 None None] PATH_INFO: `/` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:19:24.024374 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 595/1211] 127.0.0.1 () {54 vars in 917 bytes} [Thu Oct 13 04:19:24 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 13 04:19:24.029517 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-354ebab2-9b97-4296-8df6-b645f2f0fbda None None] REQUEST_METHOD: `POST` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:19:24.029964 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-354ebab2-9b97-4296-8df6-b645f2f0fbda None None] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:19:24.030089 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-354ebab2-9b97-4296-8df6-b645f2f0fbda None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:19:24.049467 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.auth.core [None req-354ebab2-9b97-4296-8df6-b645f2f0fbda None None] MFA Rules not processed for user `0e2d979563d6421995451a0e86bf3022`. Rule list: `[]` (Enabled: `True`). {{(pid=81409) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 13 04:19:24.050694 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.fernet_utils [None req-354ebab2-9b97-4296-8df6-b645f2f0fbda None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81409) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:19:24.058509 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 617/1212] 127.0.0.1 () {58 vars in 1014 bytes} [Thu Oct 13 04:19:24 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 30 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 13 04:19:24.363545 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-74fd36d8-041a-4b18-9a00-a0394b347f64 None None] Authenticating user token {{(pid=81408) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:19:24.365407 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.fernet_utils [None req-74fd36d8-041a-4b18-9a00-a0394b347f64 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81408) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:19:24.433598 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG dbcounter [-] [81408] Writing DB stats keystone:SELECT=100 {{(pid=81408) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:19:24.447888 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-74fd36d8-041a-4b18-9a00-a0394b347f64 None None] Validating token access rules against request {{(pid=81408) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:19:24.451541 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-791d1c1c-3eb6-478c-9798-7e8a0c49d86c service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6141b8a6c3d54e179d6879bb019f4d12', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dcec0431fb6849499d6aec35a234daa6', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81408) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:19:24.452137 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-791d1c1c-3eb6-478c-9798-7e8a0c49d86c service cinder] REQUEST_METHOD: `GET` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:19:24.452415 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-791d1c1c-3eb6-478c-9798-7e8a0c49d86c service cinder] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:19:24.452684 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-791d1c1c-3eb6-478c-9798-7e8a0c49d86c service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:19:24.461503 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-791d1c1c-3eb6-478c-9798-7e8a0c49d86c service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:19:24.463018 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-791d1c1c-3eb6-478c-9798-7e8a0c49d86c service cinder] RBAC: Authorization granted {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:19:24.472481 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 596/1213] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Oct 13 04:19:24 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 109 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 13 04:19:26.626021 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43068000-c62d-44ea-b02d-5d593b678397 None None] REQUEST_METHOD: `GET` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:19:26.626575 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43068000-c62d-44ea-b02d-5d593b678397 None None] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:19:26.626694 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43068000-c62d-44ea-b02d-5d593b678397 None None] PATH_INFO: `/` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:19:26.627954 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 618/1214] 127.0.0.1 () {54 vars in 917 bytes} [Thu Oct 13 04:19:26 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 13 04:19:26.633375 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-276e12de-7157-405a-a8ce-ea166d57c3f4 None None] REQUEST_METHOD: `POST` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:19:26.633801 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-276e12de-7157-405a-a8ce-ea166d57c3f4 None None] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:19:26.633913 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-276e12de-7157-405a-a8ce-ea166d57c3f4 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:19:26.653956 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.auth.core [None req-276e12de-7157-405a-a8ce-ea166d57c3f4 None None] MFA Rules not processed for user `0e2d979563d6421995451a0e86bf3022`. Rule list: `[]` (Enabled: `True`). {{(pid=81408) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 13 04:19:26.655751 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.fernet_utils [None req-276e12de-7157-405a-a8ce-ea166d57c3f4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81408) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:19:26.663593 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 597/1215] 127.0.0.1 () {58 vars in 1014 bytes} [Thu Oct 13 04:19:26 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 31 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 13 04:19:26.966366 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-43068000-c62d-44ea-b02d-5d593b678397 None None] Authenticating user token {{(pid=81409) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:19:26.968173 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.fernet_utils [None req-43068000-c62d-44ea-b02d-5d593b678397 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81409) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:19:26.975665 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG dbcounter [-] [81409] Writing DB stats keystone:SELECT=84 {{(pid=81409) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:19:26.982799 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-43068000-c62d-44ea-b02d-5d593b678397 None None] Validating token access rules against request {{(pid=81409) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:19:26.986391 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-934822bb-ff81-4349-a15c-757b0bc7738a service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6141b8a6c3d54e179d6879bb019f4d12', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dcec0431fb6849499d6aec35a234daa6', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81409) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:19:26.986989 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-934822bb-ff81-4349-a15c-757b0bc7738a service cinder] REQUEST_METHOD: `GET` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:19:26.987266 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-934822bb-ff81-4349-a15c-757b0bc7738a service cinder] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:19:26.987537 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-934822bb-ff81-4349-a15c-757b0bc7738a service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:19:26.997174 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-934822bb-ff81-4349-a15c-757b0bc7738a service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:19:26.999423 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-934822bb-ff81-4349-a15c-757b0bc7738a service cinder] RBAC: Authorization granted {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:19:27.008750 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 619/1216] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Oct 13 04:19:26 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 43 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 13 04:19:36.649067 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG dbcounter [-] [81408] Writing DB stats keystone:SELECT=11 {{(pid=81408) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:19:37.004606 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG dbcounter [-] [81409] Writing DB stats keystone:SELECT=5 {{(pid=81409) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:20:47.400874 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25a1a402-fd76-4a10-918f-4652cf1d93d5 None None] REQUEST_METHOD: `POST` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:20:47.405738 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25a1a402-fd76-4a10-918f-4652cf1d93d5 None None] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:20:47.405738 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25a1a402-fd76-4a10-918f-4652cf1d93d5 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:20:47.439564 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.auth.core [None req-25a1a402-fd76-4a10-918f-4652cf1d93d5 None None] MFA Rules not processed for user `0e2d979563d6421995451a0e86bf3022`. Rule list: `[]` (Enabled: `True`). {{(pid=81408) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 13 04:20:47.440802 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.fernet_utils [None req-25a1a402-fd76-4a10-918f-4652cf1d93d5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81408) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:20:47.456791 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 598/1217] 10.222.0.27 () {58 vars in 958 bytes} [Thu Oct 13 04:20:47 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 57 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 13 04:20:47.468295 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7988ad6c-1ead-471b-8a9a-f6374f7cd2e5 None None] REQUEST_METHOD: `POST` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:20:47.468909 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7988ad6c-1ead-471b-8a9a-f6374f7cd2e5 None None] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:20:47.469190 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7988ad6c-1ead-471b-8a9a-f6374f7cd2e5 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:20:47.498707 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.auth.core [None req-7988ad6c-1ead-471b-8a9a-f6374f7cd2e5 None None] MFA Rules not processed for user `0e2d979563d6421995451a0e86bf3022`. Rule list: `[]` (Enabled: `True`). {{(pid=81409) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 13 04:20:47.500115 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.fernet_utils [None req-7988ad6c-1ead-471b-8a9a-f6374f7cd2e5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81409) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:20:47.513475 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 620/1218] 10.222.0.27 () {58 vars in 958 bytes} [Thu Oct 13 04:20:47 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 47 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 13 04:20:47.519003 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-25a1a402-fd76-4a10-918f-4652cf1d93d5 None None] Authenticating user token {{(pid=81408) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:20:47.529408 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-25a1a402-fd76-4a10-918f-4652cf1d93d5 None None] Validating token access rules against request {{(pid=81408) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:20:47.538582 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c5d1931e-a2ae-4ed1-9979-9d7a1e5ff505 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8ba823f313144da3b683bacef8fd62e0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81408) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:20:47.539208 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5d1931e-a2ae-4ed1-9979-9d7a1e5ff505 admin admin] REQUEST_METHOD: `GET` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:20:47.539486 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5d1931e-a2ae-4ed1-9979-9d7a1e5ff505 admin admin] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:20:47.539755 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5d1931e-a2ae-4ed1-9979-9d7a1e5ff505 admin admin] PATH_INFO: `/v3/domains` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:20:47.540410 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c5d1931e-a2ae-4ed1-9979-9d7a1e5ff505 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81408) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 13 04:20:47.540710 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c5d1931e-a2ae-4ed1-9979-9d7a1e5ff505 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:20:47.542421 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c5d1931e-a2ae-4ed1-9979-9d7a1e5ff505 admin admin] RBAC: Authorization granted {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:20:47.556256 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 599/1219] 10.222.0.27 () {58 vars in 1153 bytes} [Thu Oct 13 04:20:47 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 38 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 13 04:20:47.565903 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7988ad6c-1ead-471b-8a9a-f6374f7cd2e5 None None] Authenticating user token {{(pid=81409) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:20:47.576516 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7988ad6c-1ead-471b-8a9a-f6374f7cd2e5 None None] Validating token access rules against request {{(pid=81409) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:20:47.583542 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c00b8e0e-f6c7-4d67-90fb-718f1b0eac72 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8ba823f313144da3b683bacef8fd62e0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81409) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:20:47.584176 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c00b8e0e-f6c7-4d67-90fb-718f1b0eac72 admin admin] REQUEST_METHOD: `POST` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:20:47.584449 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c00b8e0e-f6c7-4d67-90fb-718f1b0eac72 admin admin] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:20:47.584719 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c00b8e0e-f6c7-4d67-90fb-718f1b0eac72 admin admin] PATH_INFO: `/v3/projects` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:20:47.585372 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c00b8e0e-f6c7-4d67-90fb-718f1b0eac72 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:20:47.587408 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c00b8e0e-f6c7-4d67-90fb-718f1b0eac72 admin admin] RBAC: Authorization granted {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:20:47.640882 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 621/1220] 10.222.0.27 () {60 vars in 1153 bytes} [Thu Oct 13 04:20:47 2022] POST /identity/v3/projects => generated 383 bytes in 75 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 13 04:20:47.645202 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c5d1931e-a2ae-4ed1-9979-9d7a1e5ff505 admin admin] Authenticating user token {{(pid=81408) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:20:47.656433 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c5d1931e-a2ae-4ed1-9979-9d7a1e5ff505 admin admin] Validating token access rules against request {{(pid=81408) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:20:47.741435 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ea9725f2-d919-4d8c-be8f-23e9ee663585 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8ba823f313144da3b683bacef8fd62e0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81408) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:20:47.742364 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea9725f2-d919-4d8c-be8f-23e9ee663585 admin admin] REQUEST_METHOD: `POST` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:20:47.742642 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea9725f2-d919-4d8c-be8f-23e9ee663585 admin admin] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:20:47.742912 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea9725f2-d919-4d8c-be8f-23e9ee663585 admin admin] PATH_INFO: `/v3/users` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:20:47.743568 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ea9725f2-d919-4d8c-be8f-23e9ee663585 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:20:47.745672 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ea9725f2-d919-4d8c-be8f-23e9ee663585 admin admin] RBAC: Authorization granted {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:20:47.775574 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 600/1221] 10.222.0.27 () {60 vars in 1144 bytes} [Thu Oct 13 04:20:47 2022] POST /identity/v3/users => generated 292 bytes in 130 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 13 04:20:47.780078 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c00b8e0e-f6c7-4d67-90fb-718f1b0eac72 admin admin] Authenticating user token {{(pid=81409) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:20:47.793792 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c00b8e0e-f6c7-4d67-90fb-718f1b0eac72 admin admin] Validating token access rules against request {{(pid=81409) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:20:47.803856 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-02a5d451-12ed-41bd-9985-fd217ade8119 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8ba823f313144da3b683bacef8fd62e0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81409) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:20:47.804483 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-02a5d451-12ed-41bd-9985-fd217ade8119 admin admin] REQUEST_METHOD: `GET` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:20:47.804760 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-02a5d451-12ed-41bd-9985-fd217ade8119 admin admin] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:20:47.805029 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-02a5d451-12ed-41bd-9985-fd217ade8119 admin admin] PATH_INFO: `/v3/roles` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:20:47.805675 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-02a5d451-12ed-41bd-9985-fd217ade8119 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:20:47.807299 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-02a5d451-12ed-41bd-9985-fd217ade8119 admin admin] RBAC: Authorization granted {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:20:47.821737 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 622/1222] 10.222.0.27 () {58 vars in 1122 bytes} [Thu Oct 13 04:20:47 2022] GET /identity/v3/roles => generated 1420 bytes in 42 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 13 04:20:47.826059 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ea9725f2-d919-4d8c-be8f-23e9ee663585 admin admin] Authenticating user token {{(pid=81408) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:20:47.838377 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ea9725f2-d919-4d8c-be8f-23e9ee663585 admin admin] Validating token access rules against request {{(pid=81408) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:20:47.846768 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3e37f727-a9e1-4c67-a478-d9691200ce46 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8ba823f313144da3b683bacef8fd62e0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81408) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:20:47.847379 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e37f727-a9e1-4c67-a478-d9691200ce46 admin admin] REQUEST_METHOD: `GET` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:20:47.847656 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e37f727-a9e1-4c67-a478-d9691200ce46 admin admin] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:20:47.847924 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e37f727-a9e1-4c67-a478-d9691200ce46 admin admin] PATH_INFO: `/v3/roles` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:20:47.848566 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3e37f727-a9e1-4c67-a478-d9691200ce46 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:20:47.850176 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3e37f727-a9e1-4c67-a478-d9691200ce46 admin admin] RBAC: Authorization granted {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:20:47.862861 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 601/1223] 10.222.0.27 () {58 vars in 1122 bytes} [Thu Oct 13 04:20:47 2022] GET /identity/v3/roles => generated 1420 bytes in 37 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 13 04:20:47.867478 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-02a5d451-12ed-41bd-9985-fd217ade8119 admin admin] Authenticating user token {{(pid=81409) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:20:47.878904 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-02a5d451-12ed-41bd-9985-fd217ade8119 admin admin] Validating token access rules against request {{(pid=81409) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:20:47.891232 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-40c0f604-8fd5-409f-a4f4-5fc56428f216 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8ba823f313144da3b683bacef8fd62e0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81409) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:20:47.891940 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40c0f604-8fd5-409f-a4f4-5fc56428f216 admin admin] REQUEST_METHOD: `PUT` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:20:47.892217 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40c0f604-8fd5-409f-a4f4-5fc56428f216 admin admin] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:20:47.892492 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40c0f604-8fd5-409f-a4f4-5fc56428f216 admin admin] PATH_INFO: `/v3/projects/7397177414dd4ed5baed133cc89e7299/users/999c89e1bc7a426886fc802e28d38596/roles/cc56c4ae13a7424ea771c15da4b153c1` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:20:47.914307 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-40c0f604-8fd5-409f-a4f4-5fc56428f216 admin admin] RBAC: Authorizing `identity:create_grant(project_id=7397177414dd4ed5baed133cc89e7299, user_id=999c89e1bc7a426886fc802e28d38596, role_id=cc56c4ae13a7424ea771c15da4b153c1)` {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:20:47.917423 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-40c0f604-8fd5-409f-a4f4-5fc56428f216 admin admin] RBAC: Authorization granted {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:20:47.938961 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 623/1224] 10.222.0.27 () {60 vars in 1483 bytes} [Thu Oct 13 04:20:47 2022] PUT /identity/v3/projects/7397177414dd4ed5baed133cc89e7299/users/999c89e1bc7a426886fc802e28d38596/roles/cc56c4ae13a7424ea771c15da4b153c1 => generated 0 bytes in 72 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 13 04:20:47.944472 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d69d965-1a18-4182-8f15-ae95ad322e01 None None] REQUEST_METHOD: `POST` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:20:47.944764 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d69d965-1a18-4182-8f15-ae95ad322e01 None None] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:20:47.945040 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d69d965-1a18-4182-8f15-ae95ad322e01 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:20:47.971829 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.auth.core [None req-8d69d965-1a18-4182-8f15-ae95ad322e01 None None] MFA Rules not processed for user `0e2d979563d6421995451a0e86bf3022`. Rule list: `[]` (Enabled: `True`). {{(pid=81408) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 13 04:20:47.973210 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.fernet_utils [None req-8d69d965-1a18-4182-8f15-ae95ad322e01 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81408) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:20:48.054998 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 602/1225] 10.222.0.27 () {58 vars in 958 bytes} [Thu Oct 13 04:20:47 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 111 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 13 04:20:48.074263 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-40c0f604-8fd5-409f-a4f4-5fc56428f216 admin admin] Authenticating user token {{(pid=81409) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:20:48.076495 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-199e6a01-8ddd-4169-9946-230ca2034ce2 None None] REQUEST_METHOD: `POST` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:20:48.076776 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-199e6a01-8ddd-4169-9946-230ca2034ce2 None None] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:20:48.077747 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-199e6a01-8ddd-4169-9946-230ca2034ce2 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:20:48.126807 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.auth.core [None req-199e6a01-8ddd-4169-9946-230ca2034ce2 None None] MFA Rules not processed for user `0e2d979563d6421995451a0e86bf3022`. Rule list: `[]` (Enabled: `True`). {{(pid=81408) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 13 04:20:48.128829 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.fernet_utils [None req-199e6a01-8ddd-4169-9946-230ca2034ce2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81408) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:20:48.142374 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 603/1226] 10.222.0.27 () {58 vars in 958 bytes} [Thu Oct 13 04:20:48 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 67 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 13 04:20:48.152509 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff83cd2f-ca9e-4691-93d7-69b71d47186c None None] REQUEST_METHOD: `POST` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:20:48.152839 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff83cd2f-ca9e-4691-93d7-69b71d47186c None None] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:20:48.153057 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff83cd2f-ca9e-4691-93d7-69b71d47186c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:20:48.188404 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.auth.core [None req-ff83cd2f-ca9e-4691-93d7-69b71d47186c None None] MFA Rules not processed for user `0e2d979563d6421995451a0e86bf3022`. Rule list: `[]` (Enabled: `True`). {{(pid=81408) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 13 04:20:48.189536 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.fernet_utils [None req-ff83cd2f-ca9e-4691-93d7-69b71d47186c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81408) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:20:48.204154 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 604/1227] 10.222.0.27 () {58 vars in 958 bytes} [Thu Oct 13 04:20:48 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 53 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 13 04:20:48.206796 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01f7f888-d5bd-4750-b501-e088a5811179 None None] REQUEST_METHOD: `POST` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:20:48.208140 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-40c0f604-8fd5-409f-a4f4-5fc56428f216 admin admin] Validating token access rules against request {{(pid=81409) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:20:48.209334 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01f7f888-d5bd-4750-b501-e088a5811179 None None] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:20:48.209611 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01f7f888-d5bd-4750-b501-e088a5811179 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:20:48.219886 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-adc1d9e7-d7bb-4c08-97e0-b5b7f0c8d339 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd8812c3974f84871bb2d761cf0016257', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dcec0431fb6849499d6aec35a234daa6', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81409) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:20:48.220530 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-adc1d9e7-d7bb-4c08-97e0-b5b7f0c8d339 service neutron] REQUEST_METHOD: `GET` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:20:48.220806 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-adc1d9e7-d7bb-4c08-97e0-b5b7f0c8d339 service neutron] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:20:48.221076 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-adc1d9e7-d7bb-4c08-97e0-b5b7f0c8d339 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:20:48.247295 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-adc1d9e7-d7bb-4c08-97e0-b5b7f0c8d339 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:20:48.248956 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-adc1d9e7-d7bb-4c08-97e0-b5b7f0c8d339 service neutron] RBAC: Authorization granted {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:20:48.251364 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.auth.core [None req-01f7f888-d5bd-4750-b501-e088a5811179 None None] MFA Rules not processed for user `0e2d979563d6421995451a0e86bf3022`. Rule list: `[]` (Enabled: `True`). {{(pid=81408) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 13 04:20:48.252585 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.fernet_utils [None req-01f7f888-d5bd-4750-b501-e088a5811179 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81408) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:20:48.270016 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 624/1228] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Oct 13 04:20:48 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 202 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 13 04:20:48.271346 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd993e82-c4e5-472a-8377-4bf5a312b030 None None] REQUEST_METHOD: `POST` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:20:48.271549 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 605/1229] 10.222.0.27 () {58 vars in 958 bytes} [Thu Oct 13 04:20:48 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 67 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 13 04:20:48.271917 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd993e82-c4e5-472a-8377-4bf5a312b030 None None] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:20:48.272198 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd993e82-c4e5-472a-8377-4bf5a312b030 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:20:48.272537 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-01f7f888-d5bd-4750-b501-e088a5811179 None None] Authenticating user token {{(pid=81408) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:20:48.298116 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-01f7f888-d5bd-4750-b501-e088a5811179 None None] Validating token access rules against request {{(pid=81408) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:20:48.313934 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.auth.core [None req-dd993e82-c4e5-472a-8377-4bf5a312b030 None None] MFA Rules not processed for user `0e2d979563d6421995451a0e86bf3022`. Rule list: `[]` (Enabled: `True`). {{(pid=81409) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 13 04:20:48.315321 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.fernet_utils [None req-dd993e82-c4e5-472a-8377-4bf5a312b030 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81409) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:20:48.327701 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f0fdd98d-0fc9-45f1-9cce-199c8bea7c50 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8ba823f313144da3b683bacef8fd62e0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81408) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:20:48.328366 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0fdd98d-0fc9-45f1-9cce-199c8bea7c50 admin admin] REQUEST_METHOD: `GET` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:20:48.328678 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0fdd98d-0fc9-45f1-9cce-199c8bea7c50 admin admin] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:20:48.328915 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0fdd98d-0fc9-45f1-9cce-199c8bea7c50 admin admin] PATH_INFO: `/v3/domains` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:20:48.329792 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f0fdd98d-0fc9-45f1-9cce-199c8bea7c50 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81408) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 13 04:20:48.330103 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f0fdd98d-0fc9-45f1-9cce-199c8bea7c50 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:20:48.331761 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f0fdd98d-0fc9-45f1-9cce-199c8bea7c50 admin admin] RBAC: Authorization granted {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:20:48.346271 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 625/1230] 10.222.0.27 () {58 vars in 958 bytes} [Thu Oct 13 04:20:48 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 71 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 13 04:20:48.346271 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c6ec46e-775c-4da7-96c9-19610c1dbcf0 None None] REQUEST_METHOD: `POST` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:20:48.346271 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c6ec46e-775c-4da7-96c9-19610c1dbcf0 None None] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:20:48.346271 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c6ec46e-775c-4da7-96c9-19610c1dbcf0 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:20:48.369196 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 606/1231] 10.222.0.27 () {58 vars in 1153 bytes} [Thu Oct 13 04:20:48 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 94 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 13 04:20:48.376455 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e604ac96-677b-4da4-92fa-a914805ae02e None None] REQUEST_METHOD: `POST` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:20:48.376739 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e604ac96-677b-4da4-92fa-a914805ae02e None None] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:20:48.376956 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e604ac96-677b-4da4-92fa-a914805ae02e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:20:48.410144 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.auth.core [None req-7c6ec46e-775c-4da7-96c9-19610c1dbcf0 None None] MFA Rules not processed for user `0e2d979563d6421995451a0e86bf3022`. Rule list: `[]` (Enabled: `True`). {{(pid=81409) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 13 04:20:48.410398 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.fernet_utils [None req-7c6ec46e-775c-4da7-96c9-19610c1dbcf0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81409) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:20:48.430437 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.auth.core [None req-e604ac96-677b-4da4-92fa-a914805ae02e None None] MFA Rules not processed for user `0e2d979563d6421995451a0e86bf3022`. Rule list: `[]` (Enabled: `True`). {{(pid=81408) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 13 04:20:48.431663 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.fernet_utils [None req-e604ac96-677b-4da4-92fa-a914805ae02e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81408) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:20:48.445667 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 626/1232] 10.222.0.27 () {58 vars in 958 bytes} [Thu Oct 13 04:20:48 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 103 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 13 04:20:48.446916 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31456a4b-c0fb-480c-8e39-84667be059c3 None None] REQUEST_METHOD: `POST` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:20:48.447194 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31456a4b-c0fb-480c-8e39-84667be059c3 None None] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:20:48.447466 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31456a4b-c0fb-480c-8e39-84667be059c3 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:20:48.468552 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 607/1233] 10.222.0.27 () {58 vars in 958 bytes} [Thu Oct 13 04:20:48 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 97 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 13 04:20:48.469675 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e604ac96-677b-4da4-92fa-a914805ae02e None None] Authenticating user token {{(pid=81408) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:20:48.488282 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e604ac96-677b-4da4-92fa-a914805ae02e None None] Validating token access rules against request {{(pid=81408) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:20:48.493506 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b6478b52-ccfa-4380-8cac-dec2d71064fd admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8ba823f313144da3b683bacef8fd62e0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81408) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:20:48.494173 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6478b52-ccfa-4380-8cac-dec2d71064fd admin admin] REQUEST_METHOD: `POST` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:20:48.494484 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6478b52-ccfa-4380-8cac-dec2d71064fd admin admin] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:20:48.494617 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.auth.core [None req-31456a4b-c0fb-480c-8e39-84667be059c3 None None] MFA Rules not processed for user `0e2d979563d6421995451a0e86bf3022`. Rule list: `[]` (Enabled: `True`). {{(pid=81409) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 13 04:20:48.495071 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6478b52-ccfa-4380-8cac-dec2d71064fd admin admin] PATH_INFO: `/v3/projects` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:20:48.495391 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b6478b52-ccfa-4380-8cac-dec2d71064fd admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:20:48.495791 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.fernet_utils [None req-31456a4b-c0fb-480c-8e39-84667be059c3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81409) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:20:48.496417 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: WARNING py.warnings [None req-b6478b52-ccfa-4380-8cac-dec2d71064fd admin admin] /usr/local/lib/python3.8/dist-packages/oslo_policy/policy.py:1129: UserWarning: Policy "identity:create_project": "(role:admin and system_scope:all) or (role:admin and domain_id:%(target.project.domain_id)s)" failed scope check. The token used to make the request was project scoped but the policy requires ['system', 'domain'] scope. This behavior may change in the future where using the intended scope is required Oct 13 04:20:48.496417 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: warnings.warn(msg) Oct 13 04:20:48.496417 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: Oct 13 04:20:48.497962 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b6478b52-ccfa-4380-8cac-dec2d71064fd admin admin] RBAC: Authorization granted {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:20:48.511711 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 627/1234] 10.222.0.27 () {58 vars in 958 bytes} [Thu Oct 13 04:20:48 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 66 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 13 04:20:48.512608 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-31456a4b-c0fb-480c-8e39-84667be059c3 None None] Authenticating user token {{(pid=81409) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:20:48.522975 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG dbcounter [-] [81408] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=81408) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:20:48.525117 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-31456a4b-c0fb-480c-8e39-84667be059c3 None None] Validating token access rules against request {{(pid=81409) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:20:48.531760 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-65773485-2824-4c74-ae0e-14ddb279f3d1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8ba823f313144da3b683bacef8fd62e0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81409) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:20:48.535882 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65773485-2824-4c74-ae0e-14ddb279f3d1 admin admin] REQUEST_METHOD: `GET` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:20:48.536041 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 608/1235] 10.222.0.27 () {60 vars in 1153 bytes} [Thu Oct 13 04:20:48 2022] POST /identity/v3/projects => generated 381 bytes in 63 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 13 04:20:48.536144 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65773485-2824-4c74-ae0e-14ddb279f3d1 admin admin] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:20:48.536144 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65773485-2824-4c74-ae0e-14ddb279f3d1 admin admin] PATH_INFO: `/v3/domains` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:20:48.536144 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-65773485-2824-4c74-ae0e-14ddb279f3d1 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81409) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 13 04:20:48.536378 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8bf27731-980f-4d8a-9d93-4dfb64a2a861 None None] REQUEST_METHOD: `POST` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:20:48.536455 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-65773485-2824-4c74-ae0e-14ddb279f3d1 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:20:48.536535 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8bf27731-980f-4d8a-9d93-4dfb64a2a861 None None] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:20:48.536535 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8bf27731-980f-4d8a-9d93-4dfb64a2a861 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:20:48.536681 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: WARNING py.warnings [None req-65773485-2824-4c74-ae0e-14ddb279f3d1 admin admin] /usr/local/lib/python3.8/dist-packages/oslo_policy/policy.py:1129: UserWarning: Policy "identity:list_domains": "role:reader and system_scope:all" failed scope check. The token used to make the request was project scoped but the policy requires ['system'] scope. This behavior may change in the future where using the intended scope is required Oct 13 04:20:48.536681 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: warnings.warn(msg) Oct 13 04:20:48.536681 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: Oct 13 04:20:48.536681 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-65773485-2824-4c74-ae0e-14ddb279f3d1 admin admin] RBAC: Authorization granted {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:20:48.554244 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 628/1236] 10.222.0.27 () {58 vars in 1153 bytes} [Thu Oct 13 04:20:48 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 37 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 13 04:20:48.554244 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3079465-1e61-431c-9ff7-8608cb4a2023 None None] REQUEST_METHOD: `POST` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:20:48.554244 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3079465-1e61-431c-9ff7-8608cb4a2023 None None] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:20:48.554244 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3079465-1e61-431c-9ff7-8608cb4a2023 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:20:48.563230 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.auth.core [None req-8bf27731-980f-4d8a-9d93-4dfb64a2a861 None None] MFA Rules not processed for user `0e2d979563d6421995451a0e86bf3022`. Rule list: `[]` (Enabled: `True`). {{(pid=81408) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 13 04:20:48.564817 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.fernet_utils [None req-8bf27731-980f-4d8a-9d93-4dfb64a2a861 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81408) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:20:48.580108 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.auth.core [None req-d3079465-1e61-431c-9ff7-8608cb4a2023 None None] MFA Rules not processed for user `0e2d979563d6421995451a0e86bf3022`. Rule list: `[]` (Enabled: `True`). {{(pid=81409) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 13 04:20:48.584585 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.fernet_utils [None req-d3079465-1e61-431c-9ff7-8608cb4a2023 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81409) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:20:48.646557 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 609/1237] 10.222.0.27 () {58 vars in 958 bytes} [Thu Oct 13 04:20:48 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 113 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 13 04:20:48.647798 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1df9722d-ee9f-4f9d-9694-83a64e3dcf77 None None] REQUEST_METHOD: `POST` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:20:48.648152 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1df9722d-ee9f-4f9d-9694-83a64e3dcf77 None None] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:20:48.648420 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1df9722d-ee9f-4f9d-9694-83a64e3dcf77 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:20:48.648964 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG dbcounter [-] [81409] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=81409) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:20:48.666321 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 629/1238] 10.222.0.27 () {58 vars in 958 bytes} [Thu Oct 13 04:20:48 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 116 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 13 04:20:48.668423 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a171188-4154-484b-97a0-9d62df9aae19 None None] REQUEST_METHOD: `POST` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:20:48.668774 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a171188-4154-484b-97a0-9d62df9aae19 None None] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:20:48.669216 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a171188-4154-484b-97a0-9d62df9aae19 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:20:48.669887 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.auth.core [None req-1df9722d-ee9f-4f9d-9694-83a64e3dcf77 None None] MFA Rules not processed for user `0e2d979563d6421995451a0e86bf3022`. Rule list: `[]` (Enabled: `True`). {{(pid=81408) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 13 04:20:48.671812 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.fernet_utils [None req-1df9722d-ee9f-4f9d-9694-83a64e3dcf77 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81408) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:20:48.686552 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 610/1239] 10.222.0.27 () {58 vars in 958 bytes} [Thu Oct 13 04:20:48 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 36 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 13 04:20:48.686552 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1df9722d-ee9f-4f9d-9694-83a64e3dcf77 None None] Authenticating user token {{(pid=81408) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:20:48.691291 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.auth.core [None req-3a171188-4154-484b-97a0-9d62df9aae19 None None] MFA Rules not processed for user `0e2d979563d6421995451a0e86bf3022`. Rule list: `[]` (Enabled: `True`). {{(pid=81409) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 13 04:20:48.692508 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.fernet_utils [None req-3a171188-4154-484b-97a0-9d62df9aae19 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81409) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:20:48.697434 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1df9722d-ee9f-4f9d-9694-83a64e3dcf77 None None] Validating token access rules against request {{(pid=81408) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:20:48.701467 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 630/1240] 10.222.0.27 () {58 vars in 958 bytes} [Thu Oct 13 04:20:48 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 34 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 13 04:20:48.702640 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3a171188-4154-484b-97a0-9d62df9aae19 None None] Authenticating user token {{(pid=81409) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:20:48.707818 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-99df464a-775e-4ef3-9472-2d831ac31af2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8ba823f313144da3b683bacef8fd62e0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81408) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:20:48.709212 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99df464a-775e-4ef3-9472-2d831ac31af2 admin admin] REQUEST_METHOD: `GET` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:20:48.709678 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99df464a-775e-4ef3-9472-2d831ac31af2 admin admin] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:20:48.710152 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99df464a-775e-4ef3-9472-2d831ac31af2 admin admin] PATH_INFO: `/v3/domains` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:20:48.711481 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-99df464a-775e-4ef3-9472-2d831ac31af2 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81408) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 13 04:20:48.711795 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-99df464a-775e-4ef3-9472-2d831ac31af2 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:20:48.713650 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-99df464a-775e-4ef3-9472-2d831ac31af2 admin admin] RBAC: Authorization granted {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:20:48.716762 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3a171188-4154-484b-97a0-9d62df9aae19 None None] Validating token access rules against request {{(pid=81409) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:20:48.722350 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f1017947-91c8-4c60-9248-58ab8ae6efb1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8ba823f313144da3b683bacef8fd62e0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81409) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:20:48.722976 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1017947-91c8-4c60-9248-58ab8ae6efb1 admin admin] REQUEST_METHOD: `POST` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:20:48.723251 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1017947-91c8-4c60-9248-58ab8ae6efb1 admin admin] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:20:48.723519 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1017947-91c8-4c60-9248-58ab8ae6efb1 admin admin] PATH_INFO: `/v3/users` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:20:48.724156 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f1017947-91c8-4c60-9248-58ab8ae6efb1 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:20:48.725244 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: WARNING py.warnings [None req-f1017947-91c8-4c60-9248-58ab8ae6efb1 admin admin] /usr/local/lib/python3.8/dist-packages/oslo_policy/policy.py:1129: UserWarning: Policy "identity:create_user": "(role:admin and system_scope:all) or (role:admin and token.domain.id:%(target.user.domain_id)s)" failed scope check. The token used to make the request was project scoped but the policy requires ['system', 'domain'] scope. This behavior may change in the future where using the intended scope is required Oct 13 04:20:48.725244 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: warnings.warn(msg) Oct 13 04:20:48.725244 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: Oct 13 04:20:48.726411 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f1017947-91c8-4c60-9248-58ab8ae6efb1 admin admin] RBAC: Authorization granted {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:20:48.729173 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 611/1241] 10.222.0.27 () {58 vars in 1153 bytes} [Thu Oct 13 04:20:48 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 45 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 13 04:20:48.737850 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-99df464a-775e-4ef3-9472-2d831ac31af2 admin admin] Authenticating user token {{(pid=81408) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:20:48.749805 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-99df464a-775e-4ef3-9472-2d831ac31af2 admin admin] Validating token access rules against request {{(pid=81408) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:20:48.751748 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 631/1242] 10.222.0.27 () {60 vars in 1144 bytes} [Thu Oct 13 04:20:48 2022] POST /identity/v3/users => generated 291 bytes in 50 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 13 04:20:48.752903 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f06c34a2-4edc-4905-b4be-1e320b9abe75 None None] REQUEST_METHOD: `POST` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:20:48.753199 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f06c34a2-4edc-4905-b4be-1e320b9abe75 None None] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:20:48.753456 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f06c34a2-4edc-4905-b4be-1e320b9abe75 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:20:48.758038 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4f0aeab0-b392-44cd-8fb7-83d2efadf834 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8ba823f313144da3b683bacef8fd62e0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81408) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:20:48.758763 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f0aeab0-b392-44cd-8fb7-83d2efadf834 admin admin] REQUEST_METHOD: `POST` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:20:48.758991 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f0aeab0-b392-44cd-8fb7-83d2efadf834 admin admin] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:20:48.759266 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f0aeab0-b392-44cd-8fb7-83d2efadf834 admin admin] PATH_INFO: `/v3/projects` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:20:48.759912 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4f0aeab0-b392-44cd-8fb7-83d2efadf834 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:20:48.761855 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4f0aeab0-b392-44cd-8fb7-83d2efadf834 admin admin] RBAC: Authorization granted {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:20:48.776154 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.auth.core [None req-f06c34a2-4edc-4905-b4be-1e320b9abe75 None None] MFA Rules not processed for user `0e2d979563d6421995451a0e86bf3022`. Rule list: `[]` (Enabled: `True`). {{(pid=81409) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 13 04:20:48.777403 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.fernet_utils [None req-f06c34a2-4edc-4905-b4be-1e320b9abe75 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81409) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:20:48.787530 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 632/1243] 10.222.0.27 () {58 vars in 958 bytes} [Thu Oct 13 04:20:48 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 35 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 13 04:20:48.788857 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1bade158-165e-471d-bc76-608278d8cd48 None None] REQUEST_METHOD: `POST` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:20:48.789157 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1bade158-165e-471d-bc76-608278d8cd48 None None] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:20:48.789430 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1bade158-165e-471d-bc76-608278d8cd48 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:20:48.798696 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 612/1244] 10.222.0.27 () {60 vars in 1153 bytes} [Thu Oct 13 04:20:48 2022] POST /identity/v3/projects => generated 383 bytes in 61 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 13 04:20:48.799538 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4f0aeab0-b392-44cd-8fb7-83d2efadf834 admin admin] Authenticating user token {{(pid=81408) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:20:48.815840 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.auth.core [None req-1bade158-165e-471d-bc76-608278d8cd48 None None] MFA Rules not processed for user `0e2d979563d6421995451a0e86bf3022`. Rule list: `[]` (Enabled: `True`). {{(pid=81409) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 13 04:20:48.817096 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.fernet_utils [None req-1bade158-165e-471d-bc76-608278d8cd48 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81409) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:20:48.882683 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4f0aeab0-b392-44cd-8fb7-83d2efadf834 admin admin] Validating token access rules against request {{(pid=81408) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:20:48.888217 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b729807e-fee8-4589-9da4-b6b2c0794e54 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8ba823f313144da3b683bacef8fd62e0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81408) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:20:48.889100 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b729807e-fee8-4589-9da4-b6b2c0794e54 admin admin] REQUEST_METHOD: `GET` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:20:48.889100 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b729807e-fee8-4589-9da4-b6b2c0794e54 admin admin] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:20:48.889360 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b729807e-fee8-4589-9da4-b6b2c0794e54 admin admin] PATH_INFO: `/v3/domains` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:20:48.890564 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b729807e-fee8-4589-9da4-b6b2c0794e54 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81408) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 13 04:20:48.890876 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b729807e-fee8-4589-9da4-b6b2c0794e54 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:20:48.892518 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b729807e-fee8-4589-9da4-b6b2c0794e54 admin admin] RBAC: Authorization granted {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:20:48.897452 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 633/1245] 10.222.0.27 () {58 vars in 958 bytes} [Thu Oct 13 04:20:48 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 109 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 13 04:20:48.899156 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54f8b2b7-ba2a-4c37-a366-443124ae3ff5 None None] REQUEST_METHOD: `POST` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:20:48.899619 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54f8b2b7-ba2a-4c37-a366-443124ae3ff5 None None] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:20:48.900050 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54f8b2b7-ba2a-4c37-a366-443124ae3ff5 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:20:48.906908 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 613/1246] 10.222.0.27 () {58 vars in 1153 bytes} [Thu Oct 13 04:20:48 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 108 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 13 04:20:48.908009 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b729807e-fee8-4589-9da4-b6b2c0794e54 admin admin] Authenticating user token {{(pid=81408) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:20:48.923210 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b729807e-fee8-4589-9da4-b6b2c0794e54 admin admin] Validating token access rules against request {{(pid=81408) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:20:48.928094 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.auth.core [None req-54f8b2b7-ba2a-4c37-a366-443124ae3ff5 None None] MFA Rules not processed for user `0e2d979563d6421995451a0e86bf3022`. Rule list: `[]` (Enabled: `True`). {{(pid=81409) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 13 04:20:48.928319 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-726dc11b-299a-4d16-899c-ba2a87a19481 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8ba823f313144da3b683bacef8fd62e0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81408) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:20:48.929044 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-726dc11b-299a-4d16-899c-ba2a87a19481 admin admin] REQUEST_METHOD: `POST` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:20:48.929423 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-726dc11b-299a-4d16-899c-ba2a87a19481 admin admin] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:20:48.929791 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-726dc11b-299a-4d16-899c-ba2a87a19481 admin admin] PATH_INFO: `/v3/projects` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:20:48.930533 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.fernet_utils [None req-54f8b2b7-ba2a-4c37-a366-443124ae3ff5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81409) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:20:48.930677 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-726dc11b-299a-4d16-899c-ba2a87a19481 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:20:48.932937 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-726dc11b-299a-4d16-899c-ba2a87a19481 admin admin] RBAC: Authorization granted {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:20:48.940470 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 634/1247] 10.222.0.27 () {58 vars in 958 bytes} [Thu Oct 13 04:20:48 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 42 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 13 04:20:48.941721 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-54f8b2b7-ba2a-4c37-a366-443124ae3ff5 None None] Authenticating user token {{(pid=81409) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:20:48.955393 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-54f8b2b7-ba2a-4c37-a366-443124ae3ff5 None None] Validating token access rules against request {{(pid=81409) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:20:48.961519 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ded0291a-c5b1-44af-b8f6-70cefc426518 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8ba823f313144da3b683bacef8fd62e0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81409) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:20:48.962368 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ded0291a-c5b1-44af-b8f6-70cefc426518 admin admin] REQUEST_METHOD: `GET` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:20:48.962663 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ded0291a-c5b1-44af-b8f6-70cefc426518 admin admin] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:20:48.963000 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ded0291a-c5b1-44af-b8f6-70cefc426518 admin admin] PATH_INFO: `/v3/roles` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:20:48.963853 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ded0291a-c5b1-44af-b8f6-70cefc426518 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:20:48.966302 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ded0291a-c5b1-44af-b8f6-70cefc426518 admin admin] RBAC: Authorization granted {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:20:48.969801 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 614/1248] 10.222.0.27 () {60 vars in 1153 bytes} [Thu Oct 13 04:20:48 2022] POST /identity/v3/projects => generated 389 bytes in 62 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 13 04:20:48.971034 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-726dc11b-299a-4d16-899c-ba2a87a19481 admin admin] Authenticating user token {{(pid=81408) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:20:48.984412 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 635/1249] 10.222.0.27 () {58 vars in 1122 bytes} [Thu Oct 13 04:20:48 2022] GET /identity/v3/roles => generated 1420 bytes in 43 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 13 04:20:48.987311 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ded0291a-c5b1-44af-b8f6-70cefc426518 admin admin] Authenticating user token {{(pid=81409) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:20:49.000264 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ded0291a-c5b1-44af-b8f6-70cefc426518 admin admin] Validating token access rules against request {{(pid=81409) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:20:49.019168 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG dbcounter [-] [81408] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=81408) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:20:49.062833 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-726dc11b-299a-4d16-899c-ba2a87a19481 admin admin] Validating token access rules against request {{(pid=81408) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:20:49.068938 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8a277c34-f063-4b40-afd4-212216b195a9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8ba823f313144da3b683bacef8fd62e0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81408) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:20:49.069162 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-46a40e46-abc7-4660-8a58-711c191d45a9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8ba823f313144da3b683bacef8fd62e0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81409) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:20:49.069571 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a277c34-f063-4b40-afd4-212216b195a9 admin admin] REQUEST_METHOD: `GET` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:20:49.069851 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a277c34-f063-4b40-afd4-212216b195a9 admin admin] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:20:49.070137 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46a40e46-abc7-4660-8a58-711c191d45a9 admin admin] REQUEST_METHOD: `POST` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:20:49.070310 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a277c34-f063-4b40-afd4-212216b195a9 admin admin] PATH_INFO: `/v3/domains` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:20:49.070531 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46a40e46-abc7-4660-8a58-711c191d45a9 admin admin] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:20:49.070872 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46a40e46-abc7-4660-8a58-711c191d45a9 admin admin] PATH_INFO: `/v3/users` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:20:49.070975 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8a277c34-f063-4b40-afd4-212216b195a9 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81408) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 13 04:20:49.071175 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8a277c34-f063-4b40-afd4-212216b195a9 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:20:49.071647 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-46a40e46-abc7-4660-8a58-711c191d45a9 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:20:49.072827 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8a277c34-f063-4b40-afd4-212216b195a9 admin admin] RBAC: Authorization granted {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:20:49.073734 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-46a40e46-abc7-4660-8a58-711c191d45a9 admin admin] RBAC: Authorization granted {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:20:49.085984 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 615/1250] 10.222.0.27 () {58 vars in 1153 bytes} [Thu Oct 13 04:20:48 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 115 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 13 04:20:49.086815 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8a277c34-f063-4b40-afd4-212216b195a9 admin admin] Authenticating user token {{(pid=81408) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:20:49.099319 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8a277c34-f063-4b40-afd4-212216b195a9 admin admin] Validating token access rules against request {{(pid=81408) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:20:49.104601 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-81c7501e-b26d-4d50-b3da-d7bc376ca0ab admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8ba823f313144da3b683bacef8fd62e0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81408) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:20:49.105196 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81c7501e-b26d-4d50-b3da-d7bc376ca0ab admin admin] REQUEST_METHOD: `GET` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:20:49.105475 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81c7501e-b26d-4d50-b3da-d7bc376ca0ab admin admin] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:20:49.105749 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81c7501e-b26d-4d50-b3da-d7bc376ca0ab admin admin] PATH_INFO: `/v3/domains` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:20:49.106430 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 636/1251] 10.222.0.27 () {60 vars in 1144 bytes} [Thu Oct 13 04:20:48 2022] POST /identity/v3/users => generated 292 bytes in 122 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 13 04:20:49.107245 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-46a40e46-abc7-4660-8a58-711c191d45a9 admin admin] Authenticating user token {{(pid=81409) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:20:49.110784 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-81c7501e-b26d-4d50-b3da-d7bc376ca0ab admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81408) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 13 04:20:49.111113 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-81c7501e-b26d-4d50-b3da-d7bc376ca0ab admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:20:49.112762 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-81c7501e-b26d-4d50-b3da-d7bc376ca0ab admin admin] RBAC: Authorization granted {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:20:49.120941 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-46a40e46-abc7-4660-8a58-711c191d45a9 admin admin] Validating token access rules against request {{(pid=81409) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:20:49.133739 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 616/1252] 10.222.0.27 () {58 vars in 1153 bytes} [Thu Oct 13 04:20:49 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 41 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 13 04:20:49.133739 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-81c7501e-b26d-4d50-b3da-d7bc376ca0ab admin admin] Authenticating user token {{(pid=81408) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:20:49.137568 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-19857338-c375-4985-b62d-ca97c45b2b16 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8ba823f313144da3b683bacef8fd62e0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81409) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:20:49.138247 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19857338-c375-4985-b62d-ca97c45b2b16 admin admin] REQUEST_METHOD: `POST` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:20:49.138528 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19857338-c375-4985-b62d-ca97c45b2b16 admin admin] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:20:49.138799 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19857338-c375-4985-b62d-ca97c45b2b16 admin admin] PATH_INFO: `/v3/projects` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:20:49.139753 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-19857338-c375-4985-b62d-ca97c45b2b16 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:20:49.141726 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-19857338-c375-4985-b62d-ca97c45b2b16 admin admin] RBAC: Authorization granted {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:20:49.144883 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-81c7501e-b26d-4d50-b3da-d7bc376ca0ab admin admin] Validating token access rules against request {{(pid=81408) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:20:49.150765 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-51a0d851-79f1-4cfd-a96f-f88d29e2b543 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8ba823f313144da3b683bacef8fd62e0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81408) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:20:49.151500 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51a0d851-79f1-4cfd-a96f-f88d29e2b543 admin admin] REQUEST_METHOD: `GET` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:20:49.151882 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51a0d851-79f1-4cfd-a96f-f88d29e2b543 admin admin] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:20:49.152732 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51a0d851-79f1-4cfd-a96f-f88d29e2b543 admin admin] PATH_INFO: `/v3/domains` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:20:49.153119 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-51a0d851-79f1-4cfd-a96f-f88d29e2b543 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81408) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 13 04:20:49.153425 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-51a0d851-79f1-4cfd-a96f-f88d29e2b543 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:20:49.155653 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-51a0d851-79f1-4cfd-a96f-f88d29e2b543 admin admin] RBAC: Authorization granted {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:20:49.169953 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 617/1253] 10.222.0.27 () {58 vars in 1153 bytes} [Thu Oct 13 04:20:49 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 43 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 13 04:20:49.171186 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-51a0d851-79f1-4cfd-a96f-f88d29e2b543 admin admin] Authenticating user token {{(pid=81408) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:20:49.176964 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 637/1254] 10.222.0.27 () {60 vars in 1153 bytes} [Thu Oct 13 04:20:49 2022] POST /identity/v3/projects => generated 371 bytes in 67 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 13 04:20:49.176964 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-19857338-c375-4985-b62d-ca97c45b2b16 admin admin] Authenticating user token {{(pid=81409) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:20:49.183122 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-51a0d851-79f1-4cfd-a96f-f88d29e2b543 admin admin] Validating token access rules against request {{(pid=81408) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:20:49.198583 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG dbcounter [-] [81409] Writing DB stats keystone:SELECT=93,keystone:INSERT=7 {{(pid=81409) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:20:49.244363 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-19857338-c375-4985-b62d-ca97c45b2b16 admin admin] Validating token access rules against request {{(pid=81409) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:20:49.249885 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-26f42af7-86d1-46c9-b7cf-0fbf945a00ea admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8ba823f313144da3b683bacef8fd62e0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81408) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:20:49.250491 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1382a145-f469-4256-b8ba-4afd51718b11 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8ba823f313144da3b683bacef8fd62e0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81409) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:20:49.250868 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26f42af7-86d1-46c9-b7cf-0fbf945a00ea admin admin] REQUEST_METHOD: `POST` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:20:49.250994 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1382a145-f469-4256-b8ba-4afd51718b11 admin admin] REQUEST_METHOD: `GET` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:20:49.251300 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26f42af7-86d1-46c9-b7cf-0fbf945a00ea admin admin] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:20:49.251427 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1382a145-f469-4256-b8ba-4afd51718b11 admin admin] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:20:49.251636 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1382a145-f469-4256-b8ba-4afd51718b11 admin admin] PATH_INFO: `/v3/roles` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:20:49.251749 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26f42af7-86d1-46c9-b7cf-0fbf945a00ea admin admin] PATH_INFO: `/v3/users` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:20:49.256473 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1382a145-f469-4256-b8ba-4afd51718b11 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:20:49.256624 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-26f42af7-86d1-46c9-b7cf-0fbf945a00ea admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:20:49.256924 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1382a145-f469-4256-b8ba-4afd51718b11 admin admin] RBAC: Authorization granted {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:20:49.258195 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-26f42af7-86d1-46c9-b7cf-0fbf945a00ea admin admin] RBAC: Authorization granted {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:20:49.265197 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 638/1255] 10.222.0.27 () {58 vars in 1122 bytes} [Thu Oct 13 04:20:49 2022] GET /identity/v3/roles => generated 1420 bytes in 90 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 13 04:20:49.266364 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1382a145-f469-4256-b8ba-4afd51718b11 admin admin] Authenticating user token {{(pid=81409) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:20:49.278366 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1382a145-f469-4256-b8ba-4afd51718b11 admin admin] Validating token access rules against request {{(pid=81409) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:20:49.284787 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e113a84f-c50a-4255-8735-4a8b3421b826 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8ba823f313144da3b683bacef8fd62e0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81409) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:20:49.285546 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e113a84f-c50a-4255-8735-4a8b3421b826 admin admin] REQUEST_METHOD: `POST` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:20:49.285908 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e113a84f-c50a-4255-8735-4a8b3421b826 admin admin] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:20:49.286281 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e113a84f-c50a-4255-8735-4a8b3421b826 admin admin] PATH_INFO: `/v3/projects` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:20:49.287084 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e113a84f-c50a-4255-8735-4a8b3421b826 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:20:49.289147 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e113a84f-c50a-4255-8735-4a8b3421b826 admin admin] RBAC: Authorization granted {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:20:49.290371 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 618/1256] 10.222.0.27 () {60 vars in 1144 bytes} [Thu Oct 13 04:20:49 2022] POST /identity/v3/users => generated 295 bytes in 120 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 13 04:20:49.291372 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-26f42af7-86d1-46c9-b7cf-0fbf945a00ea admin admin] Authenticating user token {{(pid=81408) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:20:49.305691 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-26f42af7-86d1-46c9-b7cf-0fbf945a00ea admin admin] Validating token access rules against request {{(pid=81408) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:20:49.312596 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-297b1bd2-f2b9-4efd-b0bb-ff70a3846c12 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8ba823f313144da3b683bacef8fd62e0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81408) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:20:49.313511 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-297b1bd2-f2b9-4efd-b0bb-ff70a3846c12 admin admin] REQUEST_METHOD: `GET` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:20:49.313896 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-297b1bd2-f2b9-4efd-b0bb-ff70a3846c12 admin admin] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:20:49.314512 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-297b1bd2-f2b9-4efd-b0bb-ff70a3846c12 admin admin] PATH_INFO: `/v3/roles` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:20:49.315313 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-297b1bd2-f2b9-4efd-b0bb-ff70a3846c12 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:20:49.317656 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-297b1bd2-f2b9-4efd-b0bb-ff70a3846c12 admin admin] RBAC: Authorization granted {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:20:49.330116 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 619/1257] 10.222.0.27 () {58 vars in 1122 bytes} [Thu Oct 13 04:20:49 2022] GET /identity/v3/roles => generated 1420 bytes in 39 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 13 04:20:49.331053 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-297b1bd2-f2b9-4efd-b0bb-ff70a3846c12 admin admin] Authenticating user token {{(pid=81408) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:20:49.340102 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 639/1258] 10.222.0.27 () {60 vars in 1153 bytes} [Thu Oct 13 04:20:49 2022] POST /identity/v3/projects => generated 379 bytes in 74 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 13 04:20:49.341194 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e113a84f-c50a-4255-8735-4a8b3421b826 admin admin] Authenticating user token {{(pid=81409) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:20:49.342295 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-297b1bd2-f2b9-4efd-b0bb-ff70a3846c12 admin admin] Validating token access rules against request {{(pid=81408) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:20:49.424562 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fa841eb2-d0e9-40c0-bbeb-28a3087ace00 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8ba823f313144da3b683bacef8fd62e0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81408) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:20:49.425185 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa841eb2-d0e9-40c0-bbeb-28a3087ace00 admin admin] REQUEST_METHOD: `POST` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:20:49.425464 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa841eb2-d0e9-40c0-bbeb-28a3087ace00 admin admin] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:20:49.425737 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa841eb2-d0e9-40c0-bbeb-28a3087ace00 admin admin] PATH_INFO: `/v3/projects` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:20:49.426411 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fa841eb2-d0e9-40c0-bbeb-28a3087ace00 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:20:49.428353 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fa841eb2-d0e9-40c0-bbeb-28a3087ace00 admin admin] RBAC: Authorization granted {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:20:49.438242 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e113a84f-c50a-4255-8735-4a8b3421b826 admin admin] Validating token access rules against request {{(pid=81409) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:20:49.447563 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff32e7a5-4cc2-4a1b-aadb-b83aa68b834b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8ba823f313144da3b683bacef8fd62e0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81409) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:20:49.448372 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff32e7a5-4cc2-4a1b-aadb-b83aa68b834b admin admin] REQUEST_METHOD: `POST` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:20:49.448786 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff32e7a5-4cc2-4a1b-aadb-b83aa68b834b admin admin] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:20:49.449434 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff32e7a5-4cc2-4a1b-aadb-b83aa68b834b admin admin] PATH_INFO: `/v3/users` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:20:49.450380 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ff32e7a5-4cc2-4a1b-aadb-b83aa68b834b admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:20:49.452943 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ff32e7a5-4cc2-4a1b-aadb-b83aa68b834b admin admin] RBAC: Authorization granted {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:20:49.455270 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG dbcounter [-] [81408] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=81408) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:20:49.463595 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 620/1259] 10.222.0.27 () {60 vars in 1153 bytes} [Thu Oct 13 04:20:49 2022] POST /identity/v3/projects => generated 393 bytes in 133 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 13 04:20:49.464598 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fa841eb2-d0e9-40c0-bbeb-28a3087ace00 admin admin] Authenticating user token {{(pid=81408) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:20:49.488743 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 640/1260] 10.222.0.27 () {60 vars in 1144 bytes} [Thu Oct 13 04:20:49 2022] POST /identity/v3/users => generated 286 bytes in 148 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 13 04:20:49.489971 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff32e7a5-4cc2-4a1b-aadb-b83aa68b834b admin admin] Authenticating user token {{(pid=81409) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:20:49.503181 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff32e7a5-4cc2-4a1b-aadb-b83aa68b834b admin admin] Validating token access rules against request {{(pid=81409) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:20:49.548030 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fa841eb2-d0e9-40c0-bbeb-28a3087ace00 admin admin] Validating token access rules against request {{(pid=81408) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:20:49.553654 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-104a7b1b-2656-4a69-8aae-70877f43cfa5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8ba823f313144da3b683bacef8fd62e0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81408) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:20:49.554920 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-104a7b1b-2656-4a69-8aae-70877f43cfa5 admin admin] REQUEST_METHOD: `POST` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:20:49.557442 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-104a7b1b-2656-4a69-8aae-70877f43cfa5 admin admin] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:20:49.557442 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-104a7b1b-2656-4a69-8aae-70877f43cfa5 admin admin] PATH_INFO: `/v3/projects` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:20:49.557442 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-104a7b1b-2656-4a69-8aae-70877f43cfa5 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:20:49.559225 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-104a7b1b-2656-4a69-8aae-70877f43cfa5 admin admin] RBAC: Authorization granted {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:20:49.578078 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1b131b76-4669-498c-a9d8-ad18c5e4e1d3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8ba823f313144da3b683bacef8fd62e0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81409) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:20:49.578906 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b131b76-4669-498c-a9d8-ad18c5e4e1d3 admin admin] REQUEST_METHOD: `PUT` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:20:49.579220 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b131b76-4669-498c-a9d8-ad18c5e4e1d3 admin admin] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:20:49.579580 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b131b76-4669-498c-a9d8-ad18c5e4e1d3 admin admin] PATH_INFO: `/v3/projects/ebafd8e9b47244bb9c8da2f570b41b3a/users/3525fa9be3fe4959af41d28cd3e4335b/roles/cc56c4ae13a7424ea771c15da4b153c1` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:20:49.599385 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1b131b76-4669-498c-a9d8-ad18c5e4e1d3 admin admin] RBAC: Authorizing `identity:create_grant(project_id=ebafd8e9b47244bb9c8da2f570b41b3a, user_id=3525fa9be3fe4959af41d28cd3e4335b, role_id=cc56c4ae13a7424ea771c15da4b153c1)` {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:20:49.601718 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1b131b76-4669-498c-a9d8-ad18c5e4e1d3 admin admin] RBAC: Authorization granted {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:20:49.605378 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 621/1261] 10.222.0.27 () {60 vars in 1153 bytes} [Thu Oct 13 04:20:49 2022] POST /identity/v3/projects => generated 377 bytes in 141 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 13 04:20:49.606517 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-104a7b1b-2656-4a69-8aae-70877f43cfa5 admin admin] Authenticating user token {{(pid=81408) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:20:49.626301 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 641/1262] 10.222.0.27 () {60 vars in 1483 bytes} [Thu Oct 13 04:20:49 2022] PUT /identity/v3/projects/ebafd8e9b47244bb9c8da2f570b41b3a/users/3525fa9be3fe4959af41d28cd3e4335b/roles/cc56c4ae13a7424ea771c15da4b153c1 => generated 0 bytes in 135 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 13 04:20:49.627517 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1b131b76-4669-498c-a9d8-ad18c5e4e1d3 admin admin] Authenticating user token {{(pid=81409) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:20:49.647565 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG dbcounter [-] [81409] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=81409) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:20:49.701450 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-104a7b1b-2656-4a69-8aae-70877f43cfa5 admin admin] Validating token access rules against request {{(pid=81408) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:20:49.719363 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1b131b76-4669-498c-a9d8-ad18c5e4e1d3 admin admin] Validating token access rules against request {{(pid=81409) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:20:49.728610 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f97773fc-308f-4c26-8ef0-063f05812bc5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8ba823f313144da3b683bacef8fd62e0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81409) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:20:49.729237 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f97773fc-308f-4c26-8ef0-063f05812bc5 admin admin] REQUEST_METHOD: `GET` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:20:49.729550 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f97773fc-308f-4c26-8ef0-063f05812bc5 admin admin] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:20:49.729828 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f97773fc-308f-4c26-8ef0-063f05812bc5 admin admin] PATH_INFO: `/v3/roles` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:20:49.731267 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f97773fc-308f-4c26-8ef0-063f05812bc5 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:20:49.733027 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f97773fc-308f-4c26-8ef0-063f05812bc5 admin admin] RBAC: Authorization granted {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:20:49.747605 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 642/1263] 10.222.0.27 () {58 vars in 1122 bytes} [Thu Oct 13 04:20:49 2022] GET /identity/v3/roles => generated 1420 bytes in 121 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 13 04:20:49.748568 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f97773fc-308f-4c26-8ef0-063f05812bc5 admin admin] Authenticating user token {{(pid=81409) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:20:49.766054 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f97773fc-308f-4c26-8ef0-063f05812bc5 admin admin] Validating token access rules against request {{(pid=81409) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:20:49.771548 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-58d8aee7-ad88-4c89-8951-f95f0843d5a6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8ba823f313144da3b683bacef8fd62e0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81409) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:20:49.772158 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58d8aee7-ad88-4c89-8951-f95f0843d5a6 admin admin] REQUEST_METHOD: `POST` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:20:49.773040 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58d8aee7-ad88-4c89-8951-f95f0843d5a6 admin admin] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:20:49.773367 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58d8aee7-ad88-4c89-8951-f95f0843d5a6 admin admin] PATH_INFO: `/v3/users` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:20:49.774253 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-58d8aee7-ad88-4c89-8951-f95f0843d5a6 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:20:49.776318 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-58d8aee7-ad88-4c89-8951-f95f0843d5a6 admin admin] RBAC: Authorization granted {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:20:49.777175 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5d02890d-f702-4ca5-92b1-65345bf3cbf3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8ba823f313144da3b683bacef8fd62e0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81408) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:20:49.777894 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d02890d-f702-4ca5-92b1-65345bf3cbf3 admin admin] REQUEST_METHOD: `GET` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:20:49.778745 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d02890d-f702-4ca5-92b1-65345bf3cbf3 admin admin] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:20:49.778745 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d02890d-f702-4ca5-92b1-65345bf3cbf3 admin admin] PATH_INFO: `/v3/roles` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:20:49.779406 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5d02890d-f702-4ca5-92b1-65345bf3cbf3 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:20:49.781530 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5d02890d-f702-4ca5-92b1-65345bf3cbf3 admin admin] RBAC: Authorization granted {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:20:49.798248 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 622/1264] 10.222.0.27 () {58 vars in 1122 bytes} [Thu Oct 13 04:20:49 2022] GET /identity/v3/roles => generated 1420 bytes in 192 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 13 04:20:49.802475 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5d02890d-f702-4ca5-92b1-65345bf3cbf3 admin admin] Authenticating user token {{(pid=81408) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:20:49.806896 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 643/1265] 10.222.0.27 () {60 vars in 1144 bytes} [Thu Oct 13 04:20:49 2022] POST /identity/v3/users => generated 290 bytes in 59 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 13 04:20:49.807946 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-58d8aee7-ad88-4c89-8951-f95f0843d5a6 admin admin] Authenticating user token {{(pid=81409) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:20:49.820118 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5d02890d-f702-4ca5-92b1-65345bf3cbf3 admin admin] Validating token access rules against request {{(pid=81408) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:20:49.820339 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-58d8aee7-ad88-4c89-8951-f95f0843d5a6 admin admin] Validating token access rules against request {{(pid=81409) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:20:49.826571 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0d9102fc-94d7-4a84-adb3-c11bc9875ae6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8ba823f313144da3b683bacef8fd62e0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81409) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:20:49.827494 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d9102fc-94d7-4a84-adb3-c11bc9875ae6 admin admin] REQUEST_METHOD: `GET` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:20:49.827769 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d9102fc-94d7-4a84-adb3-c11bc9875ae6 admin admin] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:20:49.828036 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d9102fc-94d7-4a84-adb3-c11bc9875ae6 admin admin] PATH_INFO: `/v3/roles` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:20:49.828521 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aa8ddc22-cbae-4a85-9257-557d8310d88b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8ba823f313144da3b683bacef8fd62e0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81408) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:20:49.828750 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0d9102fc-94d7-4a84-adb3-c11bc9875ae6 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:20:49.829440 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa8ddc22-cbae-4a85-9257-557d8310d88b admin admin] REQUEST_METHOD: `POST` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:20:49.829820 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa8ddc22-cbae-4a85-9257-557d8310d88b admin admin] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:20:49.830189 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa8ddc22-cbae-4a85-9257-557d8310d88b admin admin] PATH_INFO: `/v3/users` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:20:49.830432 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0d9102fc-94d7-4a84-adb3-c11bc9875ae6 admin admin] RBAC: Authorization granted {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:20:49.831102 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aa8ddc22-cbae-4a85-9257-557d8310d88b admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:20:49.833525 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aa8ddc22-cbae-4a85-9257-557d8310d88b admin admin] RBAC: Authorization granted {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:20:49.846755 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 644/1266] 10.222.0.27 () {58 vars in 1122 bytes} [Thu Oct 13 04:20:49 2022] GET /identity/v3/roles => generated 1420 bytes in 39 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 13 04:20:49.848047 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0d9102fc-94d7-4a84-adb3-c11bc9875ae6 admin admin] Authenticating user token {{(pid=81409) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:20:49.862226 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0d9102fc-94d7-4a84-adb3-c11bc9875ae6 admin admin] Validating token access rules against request {{(pid=81409) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:20:49.866845 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 623/1267] 10.222.0.27 () {60 vars in 1144 bytes} [Thu Oct 13 04:20:49 2022] POST /identity/v3/users => generated 297 bytes in 65 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 13 04:20:49.868226 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16e2e035-e522-4ad4-89de-9a5d2b0210dc None None] REQUEST_METHOD: `POST` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:20:49.868549 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16e2e035-e522-4ad4-89de-9a5d2b0210dc None None] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:20:49.868883 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16e2e035-e522-4ad4-89de-9a5d2b0210dc None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:20:49.870901 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-da433cdf-c3d7-4f10-8f3c-f13547b1cfd2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8ba823f313144da3b683bacef8fd62e0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81409) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:20:49.871577 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da433cdf-c3d7-4f10-8f3c-f13547b1cfd2 admin admin] REQUEST_METHOD: `POST` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:20:49.871974 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da433cdf-c3d7-4f10-8f3c-f13547b1cfd2 admin admin] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:20:49.872334 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da433cdf-c3d7-4f10-8f3c-f13547b1cfd2 admin admin] PATH_INFO: `/v3/users` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:20:49.873268 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-da433cdf-c3d7-4f10-8f3c-f13547b1cfd2 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:20:49.876083 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-da433cdf-c3d7-4f10-8f3c-f13547b1cfd2 admin admin] RBAC: Authorization granted {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:20:49.898904 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG dbcounter [-] [81408] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=81408) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:20:49.908586 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 645/1268] 10.222.0.27 () {60 vars in 1144 bytes} [Thu Oct 13 04:20:49 2022] POST /identity/v3/users => generated 289 bytes in 61 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 13 04:20:49.909421 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-da433cdf-c3d7-4f10-8f3c-f13547b1cfd2 admin admin] Authenticating user token {{(pid=81409) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:20:49.914973 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.auth.core [None req-16e2e035-e522-4ad4-89de-9a5d2b0210dc None None] MFA Rules not processed for user `3525fa9be3fe4959af41d28cd3e4335b`. Rule list: `[]` (Enabled: `True`). {{(pid=81408) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 13 04:20:49.916454 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.fernet_utils [None req-16e2e035-e522-4ad4-89de-9a5d2b0210dc None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81408) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:20:49.923348 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-da433cdf-c3d7-4f10-8f3c-f13547b1cfd2 admin admin] Validating token access rules against request {{(pid=81409) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:20:49.928178 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7ef465bc-1a33-4800-b3f0-de2ca95576b2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8ba823f313144da3b683bacef8fd62e0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81409) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:20:49.928846 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ef465bc-1a33-4800-b3f0-de2ca95576b2 admin admin] REQUEST_METHOD: `PUT` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:20:49.929120 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ef465bc-1a33-4800-b3f0-de2ca95576b2 admin admin] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:20:49.929389 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ef465bc-1a33-4800-b3f0-de2ca95576b2 admin admin] PATH_INFO: `/v3/projects/f9a0de5ed456439092b335e5239a2df8/users/874fc04c47164bafb9ed3ab86f248dca/roles/cc56c4ae13a7424ea771c15da4b153c1` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:20:49.946430 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7ef465bc-1a33-4800-b3f0-de2ca95576b2 admin admin] RBAC: Authorizing `identity:create_grant(project_id=f9a0de5ed456439092b335e5239a2df8, user_id=874fc04c47164bafb9ed3ab86f248dca, role_id=cc56c4ae13a7424ea771c15da4b153c1)` {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:20:49.949106 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7ef465bc-1a33-4800-b3f0-de2ca95576b2 admin admin] RBAC: Authorization granted {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:20:49.971237 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 646/1269] 10.222.0.27 () {60 vars in 1483 bytes} [Thu Oct 13 04:20:49 2022] PUT /identity/v3/projects/f9a0de5ed456439092b335e5239a2df8/users/874fc04c47164bafb9ed3ab86f248dca/roles/cc56c4ae13a7424ea771c15da4b153c1 => generated 0 bytes in 62 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 13 04:20:49.972818 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7ef465bc-1a33-4800-b3f0-de2ca95576b2 admin admin] Authenticating user token {{(pid=81409) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:20:50.004357 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 624/1270] 10.222.0.27 () {58 vars in 958 bytes} [Thu Oct 13 04:20:49 2022] POST /identity/v3/auth/tokens => generated 3258 bytes in 133 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 13 04:20:50.004357 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-16e2e035-e522-4ad4-89de-9a5d2b0210dc None None] Authenticating user token {{(pid=81408) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:20:50.016697 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-16e2e035-e522-4ad4-89de-9a5d2b0210dc None None] Validating token access rules against request {{(pid=81408) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:20:50.078216 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7ef465bc-1a33-4800-b3f0-de2ca95576b2 admin admin] Validating token access rules against request {{(pid=81409) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:20:50.087087 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c08ba156-c131-4208-92e1-a5617131c687 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8ba823f313144da3b683bacef8fd62e0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81409) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:20:50.088241 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c08ba156-c131-4208-92e1-a5617131c687 admin admin] REQUEST_METHOD: `GET` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:20:50.088241 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c08ba156-c131-4208-92e1-a5617131c687 admin admin] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:20:50.088446 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c08ba156-c131-4208-92e1-a5617131c687 admin admin] PATH_INFO: `/v3/roles` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:20:50.088985 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c08ba156-c131-4208-92e1-a5617131c687 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:20:50.090888 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c08ba156-c131-4208-92e1-a5617131c687 admin admin] RBAC: Authorization granted {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:20:50.102505 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 647/1271] 10.222.0.27 () {58 vars in 1122 bytes} [Thu Oct 13 04:20:49 2022] GET /identity/v3/roles => generated 1420 bytes in 130 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 13 04:20:50.103512 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c08ba156-c131-4208-92e1-a5617131c687 admin admin] Authenticating user token {{(pid=81409) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:20:50.111326 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b53a3db3-7f87-41f2-a6e7-aa603f6094fd admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8ba823f313144da3b683bacef8fd62e0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81408) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:20:50.112340 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b53a3db3-7f87-41f2-a6e7-aa603f6094fd admin admin] REQUEST_METHOD: `GET` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:20:50.112700 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b53a3db3-7f87-41f2-a6e7-aa603f6094fd admin admin] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:20:50.113500 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b53a3db3-7f87-41f2-a6e7-aa603f6094fd admin admin] PATH_INFO: `/v3/roles` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:20:50.114962 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b53a3db3-7f87-41f2-a6e7-aa603f6094fd admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:20:50.116543 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c08ba156-c131-4208-92e1-a5617131c687 admin admin] Validating token access rules against request {{(pid=81409) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:20:50.116902 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b53a3db3-7f87-41f2-a6e7-aa603f6094fd admin admin] RBAC: Authorization granted {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:20:50.121817 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d61981e9-61b6-4a20-af8c-1547f30d8b88 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8ba823f313144da3b683bacef8fd62e0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81409) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:20:50.122462 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d61981e9-61b6-4a20-af8c-1547f30d8b88 admin admin] REQUEST_METHOD: `GET` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:20:50.122733 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d61981e9-61b6-4a20-af8c-1547f30d8b88 admin admin] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:20:50.123073 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d61981e9-61b6-4a20-af8c-1547f30d8b88 admin admin] PATH_INFO: `/v3/roles` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:20:50.123940 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d61981e9-61b6-4a20-af8c-1547f30d8b88 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:20:50.126444 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d61981e9-61b6-4a20-af8c-1547f30d8b88 admin admin] RBAC: Authorization granted {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:20:50.135794 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 625/1272] 10.222.0.27 () {58 vars in 1122 bytes} [Thu Oct 13 04:20:50 2022] GET /identity/v3/roles => generated 1420 bytes in 134 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 13 04:20:50.136652 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b53a3db3-7f87-41f2-a6e7-aa603f6094fd admin admin] Authenticating user token {{(pid=81408) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:20:50.145439 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 648/1273] 10.222.0.27 () {58 vars in 1122 bytes} [Thu Oct 13 04:20:50 2022] GET /identity/v3/roles => generated 1420 bytes in 42 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 13 04:20:50.146713 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d61981e9-61b6-4a20-af8c-1547f30d8b88 admin admin] Authenticating user token {{(pid=81409) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:20:50.148593 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b53a3db3-7f87-41f2-a6e7-aa603f6094fd admin admin] Validating token access rules against request {{(pid=81408) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:20:50.158256 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9fc2d8b3-7e8c-472a-9f00-284cbb0fd761 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8ba823f313144da3b683bacef8fd62e0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81408) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:20:50.158496 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9fc2d8b3-7e8c-472a-9f00-284cbb0fd761 admin admin] REQUEST_METHOD: `GET` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:20:50.158644 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9fc2d8b3-7e8c-472a-9f00-284cbb0fd761 admin admin] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:20:50.158913 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9fc2d8b3-7e8c-472a-9f00-284cbb0fd761 admin admin] PATH_INFO: `/v3/roles` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:20:50.159558 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9fc2d8b3-7e8c-472a-9f00-284cbb0fd761 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:20:50.162888 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d61981e9-61b6-4a20-af8c-1547f30d8b88 admin admin] Validating token access rules against request {{(pid=81409) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:20:50.164298 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9fc2d8b3-7e8c-472a-9f00-284cbb0fd761 admin admin] RBAC: Authorization granted {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:20:50.168458 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-39c93ed7-9c69-4498-8b91-f5a8f3f3f8e7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8ba823f313144da3b683bacef8fd62e0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81409) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:20:50.169441 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39c93ed7-9c69-4498-8b91-f5a8f3f3f8e7 admin admin] REQUEST_METHOD: `GET` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:20:50.169915 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39c93ed7-9c69-4498-8b91-f5a8f3f3f8e7 admin admin] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:20:50.170441 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39c93ed7-9c69-4498-8b91-f5a8f3f3f8e7 admin admin] PATH_INFO: `/v3/roles` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:20:50.172109 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-39c93ed7-9c69-4498-8b91-f5a8f3f3f8e7 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:20:50.174093 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-39c93ed7-9c69-4498-8b91-f5a8f3f3f8e7 admin admin] RBAC: Authorization granted {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:20:50.183620 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 626/1274] 10.222.0.27 () {58 vars in 1122 bytes} [Thu Oct 13 04:20:50 2022] GET /identity/v3/roles => generated 1420 bytes in 44 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 13 04:20:50.183620 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e482500-e691-4ade-bc15-bece6308cfeb None None] REQUEST_METHOD: `POST` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:20:50.183620 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e482500-e691-4ade-bc15-bece6308cfeb None None] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:20:50.183620 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e482500-e691-4ade-bc15-bece6308cfeb None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:20:50.202562 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 649/1275] 10.222.0.27 () {58 vars in 1122 bytes} [Thu Oct 13 04:20:50 2022] GET /identity/v3/roles => generated 1420 bytes in 53 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 13 04:20:50.202562 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-39c93ed7-9c69-4498-8b91-f5a8f3f3f8e7 admin admin] Authenticating user token {{(pid=81409) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:20:50.217099 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG dbcounter [-] [81409] Writing DB stats keystone:SELECT=93,keystone:INSERT=7 {{(pid=81409) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:20:50.218403 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.auth.core [None req-8e482500-e691-4ade-bc15-bece6308cfeb None None] MFA Rules not processed for user `874fc04c47164bafb9ed3ab86f248dca`. Rule list: `[]` (Enabled: `True`). {{(pid=81408) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 13 04:20:50.219550 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.fernet_utils [None req-8e482500-e691-4ade-bc15-bece6308cfeb None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81408) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:20:50.226795 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-39c93ed7-9c69-4498-8b91-f5a8f3f3f8e7 admin admin] Validating token access rules against request {{(pid=81409) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:20:50.239971 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-49e73c06-e707-47a8-8caf-893cfbd74880 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8ba823f313144da3b683bacef8fd62e0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81409) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:20:50.240599 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49e73c06-e707-47a8-8caf-893cfbd74880 admin admin] REQUEST_METHOD: `POST` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:20:50.240875 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49e73c06-e707-47a8-8caf-893cfbd74880 admin admin] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:20:50.241141 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49e73c06-e707-47a8-8caf-893cfbd74880 admin admin] PATH_INFO: `/v3/projects` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:20:50.241777 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-49e73c06-e707-47a8-8caf-893cfbd74880 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:20:50.243740 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-49e73c06-e707-47a8-8caf-893cfbd74880 admin admin] RBAC: Authorization granted {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:20:50.285788 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 650/1276] 10.222.0.27 () {60 vars in 1153 bytes} [Thu Oct 13 04:20:50 2022] POST /identity/v3/projects => generated 381 bytes in 86 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 13 04:20:50.287202 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-49e73c06-e707-47a8-8caf-893cfbd74880 admin admin] Authenticating user token {{(pid=81409) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:20:50.315028 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 627/1277] 10.222.0.27 () {58 vars in 958 bytes} [Thu Oct 13 04:20:50 2022] POST /identity/v3/auth/tokens => generated 3260 bytes in 133 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 13 04:20:50.316149 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8e482500-e691-4ade-bc15-bece6308cfeb None None] Authenticating user token {{(pid=81408) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:20:50.335040 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8e482500-e691-4ade-bc15-bece6308cfeb None None] Validating token access rules against request {{(pid=81408) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:20:50.350259 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG dbcounter [-] [81408] Writing DB stats keystone:SELECT=100 {{(pid=81408) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:20:50.385327 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-49e73c06-e707-47a8-8caf-893cfbd74880 admin admin] Validating token access rules against request {{(pid=81409) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:20:50.393111 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e6e31821-bb93-4794-9399-8df2ed7571ed admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8ba823f313144da3b683bacef8fd62e0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81409) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:20:50.393794 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6e31821-bb93-4794-9399-8df2ed7571ed admin admin] REQUEST_METHOD: `PUT` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:20:50.394069 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6e31821-bb93-4794-9399-8df2ed7571ed admin admin] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:20:50.394368 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6e31821-bb93-4794-9399-8df2ed7571ed admin admin] PATH_INFO: `/v3/projects/75387c76ea384e5c96a336751c613e6e/users/35a058c849104a9699ecfcd0c81d398e/roles/cc56c4ae13a7424ea771c15da4b153c1` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:20:50.419619 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1f4203fb-20f2-44b5-81b0-ba9050f712ce admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8ba823f313144da3b683bacef8fd62e0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81408) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:20:50.420276 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f4203fb-20f2-44b5-81b0-ba9050f712ce admin admin] REQUEST_METHOD: `GET` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:20:50.420566 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f4203fb-20f2-44b5-81b0-ba9050f712ce admin admin] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:20:50.420898 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f4203fb-20f2-44b5-81b0-ba9050f712ce admin admin] PATH_INFO: `/v3/roles` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:20:50.421740 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1f4203fb-20f2-44b5-81b0-ba9050f712ce admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:20:50.421950 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e6e31821-bb93-4794-9399-8df2ed7571ed admin admin] RBAC: Authorizing `identity:create_grant(project_id=75387c76ea384e5c96a336751c613e6e, user_id=35a058c849104a9699ecfcd0c81d398e, role_id=cc56c4ae13a7424ea771c15da4b153c1)` {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:20:50.423430 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1f4203fb-20f2-44b5-81b0-ba9050f712ce admin admin] RBAC: Authorization granted {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:20:50.425028 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e6e31821-bb93-4794-9399-8df2ed7571ed admin admin] RBAC: Authorization granted {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:20:50.540755 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 651/1278] 10.222.0.27 () {60 vars in 1483 bytes} [Thu Oct 13 04:20:50 2022] PUT /identity/v3/projects/75387c76ea384e5c96a336751c613e6e/users/35a058c849104a9699ecfcd0c81d398e/roles/cc56c4ae13a7424ea771c15da4b153c1 => generated 0 bytes in 166 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 13 04:20:50.540755 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e6e31821-bb93-4794-9399-8df2ed7571ed admin admin] Authenticating user token {{(pid=81409) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:20:50.542463 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 628/1279] 10.222.0.27 () {58 vars in 1122 bytes} [Thu Oct 13 04:20:50 2022] GET /identity/v3/roles => generated 1420 bytes in 146 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 13 04:20:50.542463 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1f4203fb-20f2-44b5-81b0-ba9050f712ce admin admin] Authenticating user token {{(pid=81408) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:20:50.542463 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1f4203fb-20f2-44b5-81b0-ba9050f712ce admin admin] Validating token access rules against request {{(pid=81408) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:20:50.567402 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e6e31821-bb93-4794-9399-8df2ed7571ed admin admin] Validating token access rules against request {{(pid=81409) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:20:50.570128 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d8e3fcb2-0b9b-4568-b98d-f31204afeaeb admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8ba823f313144da3b683bacef8fd62e0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81408) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:20:50.570838 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8e3fcb2-0b9b-4568-b98d-f31204afeaeb admin admin] REQUEST_METHOD: `GET` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:20:50.571119 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8e3fcb2-0b9b-4568-b98d-f31204afeaeb admin admin] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:20:50.571389 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8e3fcb2-0b9b-4568-b98d-f31204afeaeb admin admin] PATH_INFO: `/v3/roles` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:20:50.572038 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d8e3fcb2-0b9b-4568-b98d-f31204afeaeb admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:20:50.573382 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7360ad84-8687-46de-bb90-f1921157a314 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8ba823f313144da3b683bacef8fd62e0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81409) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:20:50.573844 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d8e3fcb2-0b9b-4568-b98d-f31204afeaeb admin admin] RBAC: Authorization granted {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:20:50.574456 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7360ad84-8687-46de-bb90-f1921157a314 admin admin] REQUEST_METHOD: `PUT` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:20:50.574730 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7360ad84-8687-46de-bb90-f1921157a314 admin admin] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:20:50.575139 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7360ad84-8687-46de-bb90-f1921157a314 admin admin] PATH_INFO: `/v3/projects/bb1bd95475ad40029bd402081f61a70c/users/c870f42503d0471881b812c024fddb7f/roles/cc56c4ae13a7424ea771c15da4b153c1` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:20:50.586456 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 629/1280] 10.222.0.27 () {58 vars in 1122 bytes} [Thu Oct 13 04:20:50 2022] GET /identity/v3/roles => generated 1420 bytes in 124 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 13 04:20:50.587422 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d8e3fcb2-0b9b-4568-b98d-f31204afeaeb admin admin] Authenticating user token {{(pid=81408) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:20:50.594394 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7360ad84-8687-46de-bb90-f1921157a314 admin admin] RBAC: Authorizing `identity:create_grant(project_id=bb1bd95475ad40029bd402081f61a70c, user_id=c870f42503d0471881b812c024fddb7f, role_id=cc56c4ae13a7424ea771c15da4b153c1)` {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:20:50.597195 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7360ad84-8687-46de-bb90-f1921157a314 admin admin] RBAC: Authorization granted {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:20:50.604561 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d8e3fcb2-0b9b-4568-b98d-f31204afeaeb admin admin] Validating token access rules against request {{(pid=81408) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:20:50.614764 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-98395ab4-e49c-4444-9be4-5389138ae654 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8ba823f313144da3b683bacef8fd62e0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81408) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:20:50.615447 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98395ab4-e49c-4444-9be4-5389138ae654 admin admin] REQUEST_METHOD: `GET` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:20:50.615715 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98395ab4-e49c-4444-9be4-5389138ae654 admin admin] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:20:50.615982 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98395ab4-e49c-4444-9be4-5389138ae654 admin admin] PATH_INFO: `/v3/roles` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:20:50.616781 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-98395ab4-e49c-4444-9be4-5389138ae654 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:20:50.618504 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-98395ab4-e49c-4444-9be4-5389138ae654 admin admin] RBAC: Authorization granted {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:20:50.622295 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 652/1281] 10.222.0.27 () {60 vars in 1483 bytes} [Thu Oct 13 04:20:50 2022] PUT /identity/v3/projects/bb1bd95475ad40029bd402081f61a70c/users/c870f42503d0471881b812c024fddb7f/roles/cc56c4ae13a7424ea771c15da4b153c1 => generated 0 bytes in 164 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 13 04:20:50.624639 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7360ad84-8687-46de-bb90-f1921157a314 admin admin] Authenticating user token {{(pid=81409) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:20:50.630869 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 630/1282] 10.222.0.27 () {58 vars in 1122 bytes} [Thu Oct 13 04:20:50 2022] GET /identity/v3/roles => generated 1420 bytes in 44 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 13 04:20:50.632297 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95582ddf-dd12-4229-a964-e86211b2eeef None None] REQUEST_METHOD: `GET` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:20:50.632566 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95582ddf-dd12-4229-a964-e86211b2eeef None None] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:20:50.632840 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95582ddf-dd12-4229-a964-e86211b2eeef None None] PATH_INFO: `/` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:20:50.634282 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 631/1283] 10.222.0.27 () {54 vars in 961 bytes} [Thu Oct 13 04:20:50 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 13 04:20:50.635100 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-302b65ad-1f00-4a96-94f8-8daf7680e09d None None] REQUEST_METHOD: `POST` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:20:50.635374 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-302b65ad-1f00-4a96-94f8-8daf7680e09d None None] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:20:50.635639 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-302b65ad-1f00-4a96-94f8-8daf7680e09d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:20:50.657807 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.auth.core [None req-302b65ad-1f00-4a96-94f8-8daf7680e09d None None] MFA Rules not processed for user `0e2d979563d6421995451a0e86bf3022`. Rule list: `[]` (Enabled: `True`). {{(pid=81408) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 13 04:20:50.659092 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.fernet_utils [None req-302b65ad-1f00-4a96-94f8-8daf7680e09d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81408) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:20:50.692519 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG dbcounter [-] [81409] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81409) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:20:50.709033 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7360ad84-8687-46de-bb90-f1921157a314 admin admin] Validating token access rules against request {{(pid=81409) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:20:50.715051 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5add0ad1-b6ab-45ad-bad2-35c281474d43 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8ba823f313144da3b683bacef8fd62e0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81409) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:20:50.715991 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5add0ad1-b6ab-45ad-bad2-35c281474d43 admin admin] REQUEST_METHOD: `POST` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:20:50.716549 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5add0ad1-b6ab-45ad-bad2-35c281474d43 admin admin] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:20:50.716832 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5add0ad1-b6ab-45ad-bad2-35c281474d43 admin admin] PATH_INFO: `/v3/users` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:20:50.718000 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5add0ad1-b6ab-45ad-bad2-35c281474d43 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:20:50.720289 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5add0ad1-b6ab-45ad-bad2-35c281474d43 admin admin] RBAC: Authorization granted {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:20:50.724945 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 632/1284] 10.222.0.27 () {58 vars in 958 bytes} [Thu Oct 13 04:20:50 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 90 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 13 04:20:50.725667 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-302b65ad-1f00-4a96-94f8-8daf7680e09d None None] Authenticating user token {{(pid=81408) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:20:50.738879 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-302b65ad-1f00-4a96-94f8-8daf7680e09d None None] Validating token access rules against request {{(pid=81408) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:20:50.745524 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3c39bfc5-08f5-43ac-a083-d4eff97abee1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8ba823f313144da3b683bacef8fd62e0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81408) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:20:50.746592 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c39bfc5-08f5-43ac-a083-d4eff97abee1 admin admin] REQUEST_METHOD: `PUT` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:20:50.746592 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c39bfc5-08f5-43ac-a083-d4eff97abee1 admin admin] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:20:50.746903 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c39bfc5-08f5-43ac-a083-d4eff97abee1 admin admin] PATH_INFO: `/v3/projects/2413ab4744c345bc94f68b6510ca3b44/users/70ff2e3f5a0f4e0da74af8d0ea6b1506/roles/cc56c4ae13a7424ea771c15da4b153c1` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:20:50.749704 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 653/1285] 10.222.0.27 () {60 vars in 1144 bytes} [Thu Oct 13 04:20:50 2022] POST /identity/v3/users => generated 297 bytes in 124 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 13 04:20:50.750670 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5add0ad1-b6ab-45ad-bad2-35c281474d43 admin admin] Authenticating user token {{(pid=81409) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:20:50.760334 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5add0ad1-b6ab-45ad-bad2-35c281474d43 admin admin] Validating token access rules against request {{(pid=81409) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:20:50.766024 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f089e623-a38f-4772-b15f-78d86dd14926 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8ba823f313144da3b683bacef8fd62e0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81409) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:20:50.766831 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f089e623-a38f-4772-b15f-78d86dd14926 admin admin] REQUEST_METHOD: `PUT` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:20:50.767181 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f089e623-a38f-4772-b15f-78d86dd14926 admin admin] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:20:50.767536 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f089e623-a38f-4772-b15f-78d86dd14926 admin admin] PATH_INFO: `/v3/projects/0913f8eb06164012b366d62c553d261c/users/da9c75191a3b4ef49438aae9e35fcf13/roles/cc56c4ae13a7424ea771c15da4b153c1` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:20:50.768561 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3c39bfc5-08f5-43ac-a083-d4eff97abee1 admin admin] RBAC: Authorizing `identity:create_grant(project_id=2413ab4744c345bc94f68b6510ca3b44, user_id=70ff2e3f5a0f4e0da74af8d0ea6b1506, role_id=cc56c4ae13a7424ea771c15da4b153c1)` {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:20:50.772618 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3c39bfc5-08f5-43ac-a083-d4eff97abee1 admin admin] RBAC: Authorization granted {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:20:50.786815 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f089e623-a38f-4772-b15f-78d86dd14926 admin admin] RBAC: Authorizing `identity:create_grant(project_id=0913f8eb06164012b366d62c553d261c, user_id=da9c75191a3b4ef49438aae9e35fcf13, role_id=cc56c4ae13a7424ea771c15da4b153c1)` {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:20:50.789495 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f089e623-a38f-4772-b15f-78d86dd14926 admin admin] RBAC: Authorization granted {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:20:50.789896 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 633/1286] 10.222.0.27 () {60 vars in 1483 bytes} [Thu Oct 13 04:20:50 2022] PUT /identity/v3/projects/2413ab4744c345bc94f68b6510ca3b44/users/70ff2e3f5a0f4e0da74af8d0ea6b1506/roles/cc56c4ae13a7424ea771c15da4b153c1 => generated 0 bytes in 65 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 13 04:20:50.791410 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b0aca69-b46b-4048-b73a-35e0e144e559 None None] REQUEST_METHOD: `POST` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:20:50.791831 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b0aca69-b46b-4048-b73a-35e0e144e559 None None] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:20:50.792023 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b0aca69-b46b-4048-b73a-35e0e144e559 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:20:50.808307 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 654/1287] 10.222.0.27 () {60 vars in 1483 bytes} [Thu Oct 13 04:20:50 2022] PUT /identity/v3/projects/0913f8eb06164012b366d62c553d261c/users/da9c75191a3b4ef49438aae9e35fcf13/roles/cc56c4ae13a7424ea771c15da4b153c1 => generated 0 bytes in 58 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 13 04:20:50.809154 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f089e623-a38f-4772-b15f-78d86dd14926 admin admin] Authenticating user token {{(pid=81409) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:20:50.826734 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG dbcounter [-] [81408] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81408) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:20:50.831383 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.auth.core [None req-2b0aca69-b46b-4048-b73a-35e0e144e559 None None] MFA Rules not processed for user `c870f42503d0471881b812c024fddb7f`. Rule list: `[]` (Enabled: `True`). {{(pid=81408) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 13 04:20:50.833891 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.fernet_utils [None req-2b0aca69-b46b-4048-b73a-35e0e144e559 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81408) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:20:50.878935 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f089e623-a38f-4772-b15f-78d86dd14926 admin admin] Validating token access rules against request {{(pid=81409) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:20:50.883920 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d0d05a97-4772-4bb4-8832-090840786360 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8ba823f313144da3b683bacef8fd62e0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81409) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:20:50.884686 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0d05a97-4772-4bb4-8832-090840786360 admin admin] REQUEST_METHOD: `PUT` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:20:50.885029 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0d05a97-4772-4bb4-8832-090840786360 admin admin] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:20:50.885409 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0d05a97-4772-4bb4-8832-090840786360 admin admin] PATH_INFO: `/v3/projects/73d6c8ffc0354e9d889b1abc471602ac/users/1f6cf725908041ccaceb1ffa694ad161/roles/cc56c4ae13a7424ea771c15da4b153c1` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:20:50.905816 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d0d05a97-4772-4bb4-8832-090840786360 admin admin] RBAC: Authorizing `identity:create_grant(project_id=73d6c8ffc0354e9d889b1abc471602ac, user_id=1f6cf725908041ccaceb1ffa694ad161, role_id=cc56c4ae13a7424ea771c15da4b153c1)` {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:20:50.912927 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 634/1288] 10.222.0.27 () {58 vars in 958 bytes} [Thu Oct 13 04:20:50 2022] POST /identity/v3/auth/tokens => generated 3248 bytes in 118 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 13 04:20:50.913112 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d0d05a97-4772-4bb4-8832-090840786360 admin admin] RBAC: Authorization granted {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:20:50.913226 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fdcc4518-83dc-4e3b-a9b1-736d979fe27f None None] REQUEST_METHOD: `POST` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:20:50.913226 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fdcc4518-83dc-4e3b-a9b1-736d979fe27f None None] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:20:50.913226 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fdcc4518-83dc-4e3b-a9b1-736d979fe27f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:20:50.928829 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 655/1289] 10.222.0.27 () {60 vars in 1483 bytes} [Thu Oct 13 04:20:50 2022] PUT /identity/v3/projects/73d6c8ffc0354e9d889b1abc471602ac/users/1f6cf725908041ccaceb1ffa694ad161/roles/cc56c4ae13a7424ea771c15da4b153c1 => generated 0 bytes in 120 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 13 04:20:50.933710 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d0d05a97-4772-4bb4-8832-090840786360 admin admin] Authenticating user token {{(pid=81409) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:20:50.963386 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.auth.core [None req-fdcc4518-83dc-4e3b-a9b1-736d979fe27f None None] MFA Rules not processed for user `cb740ed7f7684b079e45f887be83f795`. Rule list: `[]` (Enabled: `True`). {{(pid=81408) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 13 04:20:50.965556 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.fernet_utils [None req-fdcc4518-83dc-4e3b-a9b1-736d979fe27f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81408) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:20:51.023543 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d0d05a97-4772-4bb4-8832-090840786360 admin admin] Validating token access rules against request {{(pid=81409) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:20:51.031551 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4236d780-d13c-48e4-80a1-115e84789850 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd8812c3974f84871bb2d761cf0016257', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dcec0431fb6849499d6aec35a234daa6', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81409) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:20:51.033890 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4236d780-d13c-48e4-80a1-115e84789850 service neutron] REQUEST_METHOD: `GET` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:20:51.034379 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4236d780-d13c-48e4-80a1-115e84789850 service neutron] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:20:51.034717 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4236d780-d13c-48e4-80a1-115e84789850 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:20:51.069731 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 635/1290] 10.222.0.27 () {58 vars in 1058 bytes} [Thu Oct 13 04:20:50 2022] POST /identity/v3/auth/tokens => generated 3372 bytes in 160 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 13 04:20:51.070895 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fdcc4518-83dc-4e3b-a9b1-736d979fe27f None None] Authenticating user token {{(pid=81408) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:20:51.110885 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG dbcounter [-] [81409] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=81409) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:20:51.129981 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4236d780-d13c-48e4-80a1-115e84789850 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:20:51.131645 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4236d780-d13c-48e4-80a1-115e84789850 service neutron] RBAC: Authorization granted {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:20:51.143286 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 656/1291] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Oct 13 04:20:50 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 214 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 13 04:20:51.144815 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cbd33c3f-98bd-4fb0-a26c-df64685b9fad None None] REQUEST_METHOD: `POST` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:20:51.145204 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cbd33c3f-98bd-4fb0-a26c-df64685b9fad None None] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:20:51.145584 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cbd33c3f-98bd-4fb0-a26c-df64685b9fad None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:20:51.164857 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fdcc4518-83dc-4e3b-a9b1-736d979fe27f None None] Validating token access rules against request {{(pid=81408) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:20:51.167177 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.auth.core [None req-cbd33c3f-98bd-4fb0-a26c-df64685b9fad None None] MFA Rules not processed for user `0e2d979563d6421995451a0e86bf3022`. Rule list: `[]` (Enabled: `True`). {{(pid=81409) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 13 04:20:51.168554 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.fernet_utils [None req-cbd33c3f-98bd-4fb0-a26c-df64685b9fad None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81409) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:20:51.169825 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-74eb0ff7-c369-4489-888f-1591b11bb9b6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8ba823f313144da3b683bacef8fd62e0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81408) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:20:51.170475 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74eb0ff7-c369-4489-888f-1591b11bb9b6 admin admin] REQUEST_METHOD: `GET` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:20:51.171874 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74eb0ff7-c369-4489-888f-1591b11bb9b6 admin admin] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:20:51.172162 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74eb0ff7-c369-4489-888f-1591b11bb9b6 admin admin] PATH_INFO: `/v3/roles` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:20:51.172923 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-74eb0ff7-c369-4489-888f-1591b11bb9b6 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:20:51.175080 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-74eb0ff7-c369-4489-888f-1591b11bb9b6 admin admin] RBAC: Authorization granted {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:20:51.179251 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 657/1292] 10.222.0.27 () {58 vars in 958 bytes} [Thu Oct 13 04:20:51 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 35 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 13 04:20:51.181778 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c173cf60-ea7e-413d-a875-331c9ff94ace None None] REQUEST_METHOD: `POST` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:20:51.182409 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c173cf60-ea7e-413d-a875-331c9ff94ace None None] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:20:51.183130 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c173cf60-ea7e-413d-a875-331c9ff94ace None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:20:51.197650 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 636/1293] 10.222.0.27 () {58 vars in 1122 bytes} [Thu Oct 13 04:20:51 2022] GET /identity/v3/roles => generated 1420 bytes in 127 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 13 04:20:51.203351 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f795b528-6d44-430e-9b71-9f7401329292 None None] REQUEST_METHOD: `POST` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:20:51.203642 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f795b528-6d44-430e-9b71-9f7401329292 None None] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:20:51.203919 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f795b528-6d44-430e-9b71-9f7401329292 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:20:51.209781 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.auth.core [None req-c173cf60-ea7e-413d-a875-331c9ff94ace None None] MFA Rules not processed for user `0e2d979563d6421995451a0e86bf3022`. Rule list: `[]` (Enabled: `True`). {{(pid=81409) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 13 04:20:51.211387 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.fernet_utils [None req-c173cf60-ea7e-413d-a875-331c9ff94ace None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81409) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:20:51.221727 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 658/1294] 10.222.0.27 () {58 vars in 958 bytes} [Thu Oct 13 04:20:51 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 42 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 13 04:20:51.224932 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f991bb30-12e6-47f5-99e4-cacc3368ffa6 None None] REQUEST_METHOD: `GET` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:20:51.224932 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f991bb30-12e6-47f5-99e4-cacc3368ffa6 None None] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:20:51.224932 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f991bb30-12e6-47f5-99e4-cacc3368ffa6 None None] PATH_INFO: `/` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:20:51.228971 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 659/1295] 10.222.0.27 () {54 vars in 961 bytes} [Thu Oct 13 04:20:51 2022] GET /identity => generated 270 bytes in 6 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 13 04:20:51.231369 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f991bb30-12e6-47f5-99e4-cacc3368ffa6 None None] Authenticating user token {{(pid=81409) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:20:51.246078 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.auth.core [None req-f795b528-6d44-430e-9b71-9f7401329292 None None] MFA Rules not processed for user `0e2d979563d6421995451a0e86bf3022`. Rule list: `[]` (Enabled: `True`). {{(pid=81408) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 13 04:20:51.247382 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.fernet_utils [None req-f795b528-6d44-430e-9b71-9f7401329292 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81408) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:20:51.261053 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 637/1296] 10.222.0.27 () {58 vars in 958 bytes} [Thu Oct 13 04:20:51 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 60 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 13 04:20:51.261053 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f795b528-6d44-430e-9b71-9f7401329292 None None] Authenticating user token {{(pid=81408) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:20:51.262491 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f991bb30-12e6-47f5-99e4-cacc3368ffa6 None None] Validating token access rules against request {{(pid=81409) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:20:51.277383 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f795b528-6d44-430e-9b71-9f7401329292 None None] Validating token access rules against request {{(pid=81408) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:20:51.278364 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a549ab00-87ed-45a0-be3a-35fe04988157 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cb740ed7f7684b079e45f887be83f795', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dcec0431fb6849499d6aec35a234daa6', 'project_domain_id': 'default', 'roles': ['ResellerAdmin', 'admin', 'reader', 'service', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81409) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:20:51.279205 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a549ab00-87ed-45a0-be3a-35fe04988157 service nova] REQUEST_METHOD: `GET` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:20:51.279552 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a549ab00-87ed-45a0-be3a-35fe04988157 service nova] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:20:51.280138 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a549ab00-87ed-45a0-be3a-35fe04988157 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:20:51.285223 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bb4cfa60-5f2b-40d4-af71-9cd2f673feb8 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd8812c3974f84871bb2d761cf0016257', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dcec0431fb6849499d6aec35a234daa6', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81408) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:20:51.286219 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb4cfa60-5f2b-40d4-af71-9cd2f673feb8 service neutron] REQUEST_METHOD: `GET` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:20:51.286594 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb4cfa60-5f2b-40d4-af71-9cd2f673feb8 service neutron] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:20:51.287038 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb4cfa60-5f2b-40d4-af71-9cd2f673feb8 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:20:51.301729 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bb4cfa60-5f2b-40d4-af71-9cd2f673feb8 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:20:51.312171 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bb4cfa60-5f2b-40d4-af71-9cd2f673feb8 service neutron] RBAC: Authorization granted {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:20:51.317449 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG dbcounter [-] [81408] Writing DB stats keystone:SELECT=100 {{(pid=81408) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:20:51.329974 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 638/1297] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Oct 13 04:20:51 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 71 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 13 04:20:51.332298 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bb4cfa60-5f2b-40d4-af71-9cd2f673feb8 service neutron] Authenticating user token {{(pid=81408) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:20:51.344738 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bb4cfa60-5f2b-40d4-af71-9cd2f673feb8 service neutron] Validating token access rules against request {{(pid=81408) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:20:51.360886 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-79feed59-0fd2-4c72-8b00-ec3645862b02 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8ba823f313144da3b683bacef8fd62e0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81408) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:20:51.362236 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79feed59-0fd2-4c72-8b00-ec3645862b02 admin admin] REQUEST_METHOD: `PUT` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:20:51.363252 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79feed59-0fd2-4c72-8b00-ec3645862b02 admin admin] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:20:51.363633 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79feed59-0fd2-4c72-8b00-ec3645862b02 admin admin] PATH_INFO: `/v3/projects/9e491fd308fb4073a62b71c4e0d190d8/users/25c7b42411704d48bb1a25778a16d208/roles/ef55fbd94e1040be83695754075a7ed5` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:20:51.387533 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a549ab00-87ed-45a0-be3a-35fe04988157 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:20:51.389495 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a549ab00-87ed-45a0-be3a-35fe04988157 service nova] RBAC: Authorization granted {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:20:51.401867 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 660/1298] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Oct 13 04:20:51 2022] GET /identity/v3/auth/tokens => generated 3260 bytes in 172 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 13 04:20:51.403319 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-79feed59-0fd2-4c72-8b00-ec3645862b02 admin admin] RBAC: Authorizing `identity:create_grant(project_id=9e491fd308fb4073a62b71c4e0d190d8, user_id=25c7b42411704d48bb1a25778a16d208, role_id=ef55fbd94e1040be83695754075a7ed5)` {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:20:51.405822 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1bf48b14-34e7-405a-9896-f59002051773 None None] REQUEST_METHOD: `POST` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:20:51.406293 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1bf48b14-34e7-405a-9896-f59002051773 None None] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:20:51.406571 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1bf48b14-34e7-405a-9896-f59002051773 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:20:51.414235 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-79feed59-0fd2-4c72-8b00-ec3645862b02 admin admin] RBAC: Authorization granted {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:20:51.435993 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.auth.core [None req-1bf48b14-34e7-405a-9896-f59002051773 None None] MFA Rules not processed for user `cb740ed7f7684b079e45f887be83f795`. Rule list: `[]` (Enabled: `True`). {{(pid=81409) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 13 04:20:51.437547 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.fernet_utils [None req-1bf48b14-34e7-405a-9896-f59002051773 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81409) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:20:51.442812 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 639/1299] 10.222.0.27 () {60 vars in 1483 bytes} [Thu Oct 13 04:20:51 2022] PUT /identity/v3/projects/9e491fd308fb4073a62b71c4e0d190d8/users/25c7b42411704d48bb1a25778a16d208/roles/ef55fbd94e1040be83695754075a7ed5 => generated 0 bytes in 112 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 13 04:20:51.448496 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-79feed59-0fd2-4c72-8b00-ec3645862b02 admin admin] Authenticating user token {{(pid=81408) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:20:51.601485 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 661/1300] 10.222.0.27 () {58 vars in 1058 bytes} [Thu Oct 13 04:20:51 2022] POST /identity/v3/auth/tokens => generated 3372 bytes in 199 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 13 04:20:51.602479 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1bf48b14-34e7-405a-9896-f59002051773 None None] Authenticating user token {{(pid=81409) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:20:51.613134 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-79feed59-0fd2-4c72-8b00-ec3645862b02 admin admin] Validating token access rules against request {{(pid=81408) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:20:51.620903 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-244b5b6d-55ed-4ebb-a8f1-26e127890b94 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd8812c3974f84871bb2d761cf0016257', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dcec0431fb6849499d6aec35a234daa6', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81408) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:20:51.621594 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-244b5b6d-55ed-4ebb-a8f1-26e127890b94 service neutron] REQUEST_METHOD: `GET` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:20:51.621877 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-244b5b6d-55ed-4ebb-a8f1-26e127890b94 service neutron] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:20:51.622152 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-244b5b6d-55ed-4ebb-a8f1-26e127890b94 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:20:51.623218 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1bf48b14-34e7-405a-9896-f59002051773 None None] Validating token access rules against request {{(pid=81409) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:20:51.635388 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-91692978-d9a1-408a-a9b8-3579f506869a service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd8812c3974f84871bb2d761cf0016257', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dcec0431fb6849499d6aec35a234daa6', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81409) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:20:51.636441 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91692978-d9a1-408a-a9b8-3579f506869a service neutron] REQUEST_METHOD: `GET` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:20:51.636874 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91692978-d9a1-408a-a9b8-3579f506869a service neutron] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:20:51.639264 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91692978-d9a1-408a-a9b8-3579f506869a service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:20:51.743708 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-244b5b6d-55ed-4ebb-a8f1-26e127890b94 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:20:51.746401 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-244b5b6d-55ed-4ebb-a8f1-26e127890b94 service neutron] RBAC: Authorization granted {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:20:51.761646 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 640/1301] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Oct 13 04:20:51 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 318 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 13 04:20:51.762858 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-244b5b6d-55ed-4ebb-a8f1-26e127890b94 service neutron] Authenticating user token {{(pid=81408) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:20:51.778347 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-244b5b6d-55ed-4ebb-a8f1-26e127890b94 service neutron] Validating token access rules against request {{(pid=81408) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:20:51.786882 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f9c7df36-247c-4481-b706-cf9c0a132890 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8ba823f313144da3b683bacef8fd62e0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81408) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:20:51.787361 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9c7df36-247c-4481-b706-cf9c0a132890 admin admin] REQUEST_METHOD: `GET` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:20:51.787644 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9c7df36-247c-4481-b706-cf9c0a132890 admin admin] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:20:51.788725 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9c7df36-247c-4481-b706-cf9c0a132890 admin admin] PATH_INFO: `/v3/roles` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:20:51.788725 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f9c7df36-247c-4481-b706-cf9c0a132890 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:20:51.792116 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f9c7df36-247c-4481-b706-cf9c0a132890 admin admin] RBAC: Authorization granted {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:20:51.819348 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-91692978-d9a1-408a-a9b8-3579f506869a service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:20:51.821083 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-91692978-d9a1-408a-a9b8-3579f506869a service neutron] RBAC: Authorization granted {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:20:51.822730 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 641/1302] 10.222.0.27 () {58 vars in 1122 bytes} [Thu Oct 13 04:20:51 2022] GET /identity/v3/roles => generated 1420 bytes in 61 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 13 04:20:51.826256 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f9c7df36-247c-4481-b706-cf9c0a132890 admin admin] Authenticating user token {{(pid=81408) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:20:51.833244 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG dbcounter [-] [81409] Writing DB stats keystone:SELECT=100 {{(pid=81409) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:20:51.854077 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 662/1303] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Oct 13 04:20:51 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 252 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 13 04:20:51.859873 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-91692978-d9a1-408a-a9b8-3579f506869a service neutron] Authenticating user token {{(pid=81409) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:20:51.864294 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f9c7df36-247c-4481-b706-cf9c0a132890 admin admin] Validating token access rules against request {{(pid=81408) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:20:51.882654 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e53219e6-5580-40e7-8231-ee6ddad45bf6 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cb740ed7f7684b079e45f887be83f795', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dcec0431fb6849499d6aec35a234daa6', 'project_domain_id': 'default', 'roles': ['ResellerAdmin', 'admin', 'reader', 'service', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81408) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:20:51.883637 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e53219e6-5580-40e7-8231-ee6ddad45bf6 service nova] REQUEST_METHOD: `GET` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:20:51.883637 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e53219e6-5580-40e7-8231-ee6ddad45bf6 service nova] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:20:51.885375 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e53219e6-5580-40e7-8231-ee6ddad45bf6 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:20:51.890822 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-91692978-d9a1-408a-a9b8-3579f506869a service neutron] Validating token access rules against request {{(pid=81409) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:20:51.914504 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a0f54a2a-eb48-4ae7-ba51-e7299913d928 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8ba823f313144da3b683bacef8fd62e0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81409) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:20:51.917696 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0f54a2a-eb48-4ae7-ba51-e7299913d928 admin admin] REQUEST_METHOD: `PUT` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:20:51.917856 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0f54a2a-eb48-4ae7-ba51-e7299913d928 admin admin] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:20:51.918012 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0f54a2a-eb48-4ae7-ba51-e7299913d928 admin admin] PATH_INFO: `/v3/projects/9e491fd308fb4073a62b71c4e0d190d8/users/25c7b42411704d48bb1a25778a16d208/roles/ef55fbd94e1040be83695754075a7ed5` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:20:51.925559 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a0f54a2a-eb48-4ae7-ba51-e7299913d928 admin admin] RBAC: Authorizing `identity:create_grant(project_id=9e491fd308fb4073a62b71c4e0d190d8, user_id=25c7b42411704d48bb1a25778a16d208, role_id=ef55fbd94e1040be83695754075a7ed5)` {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:20:51.928365 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a0f54a2a-eb48-4ae7-ba51-e7299913d928 admin admin] RBAC: Authorization granted {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:20:51.975552 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 663/1304] 10.222.0.27 () {60 vars in 1483 bytes} [Thu Oct 13 04:20:51 2022] PUT /identity/v3/projects/9e491fd308fb4073a62b71c4e0d190d8/users/25c7b42411704d48bb1a25778a16d208/roles/ef55fbd94e1040be83695754075a7ed5 => generated 0 bytes in 116 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 13 04:20:51.995427 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56f1b420-3508-47ce-8060-46860c24a12a None None] REQUEST_METHOD: `POST` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:20:51.995738 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56f1b420-3508-47ce-8060-46860c24a12a None None] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:20:51.995975 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56f1b420-3508-47ce-8060-46860c24a12a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:20:52.096675 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.auth.core [None req-56f1b420-3508-47ce-8060-46860c24a12a None None] MFA Rules not processed for user `25c7b42411704d48bb1a25778a16d208`. Rule list: `[]` (Enabled: `True`). {{(pid=81409) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 13 04:20:52.098003 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.fernet_utils [None req-56f1b420-3508-47ce-8060-46860c24a12a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81409) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:20:52.110234 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e53219e6-5580-40e7-8231-ee6ddad45bf6 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:20:52.112065 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e53219e6-5580-40e7-8231-ee6ddad45bf6 service nova] RBAC: Authorization granted {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:20:52.140203 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 642/1305] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Oct 13 04:20:51 2022] GET /identity/v3/auth/tokens => generated 3248 bytes in 317 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 13 04:20:52.164998 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78056b32-471d-4a09-9ad9-0c67b147483d None None] REQUEST_METHOD: `GET` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:20:52.165291 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78056b32-471d-4a09-9ad9-0c67b147483d None None] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:20:52.166781 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78056b32-471d-4a09-9ad9-0c67b147483d None None] PATH_INFO: `/` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:20:52.179129 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 643/1306] 10.222.0.27 () {54 vars in 912 bytes} [Thu Oct 13 04:20:52 2022] GET /identity => generated 270 bytes in 24 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 13 04:20:52.185395 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-843fed01-958b-4ea0-ac11-080375f872eb None None] REQUEST_METHOD: `POST` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:20:52.187936 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-843fed01-958b-4ea0-ac11-080375f872eb None None] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:20:52.189092 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-843fed01-958b-4ea0-ac11-080375f872eb None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:20:52.290591 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.auth.core [None req-843fed01-958b-4ea0-ac11-080375f872eb None None] MFA Rules not processed for user `cb740ed7f7684b079e45f887be83f795`. Rule list: `[]` (Enabled: `True`). {{(pid=81408) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 13 04:20:52.303396 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.fernet_utils [None req-843fed01-958b-4ea0-ac11-080375f872eb None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81408) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:20:52.326327 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG dbcounter [-] [81408] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81408) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:20:52.342930 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 664/1307] 10.222.0.27 () {58 vars in 958 bytes} [Thu Oct 13 04:20:51 2022] POST /identity/v3/auth/tokens => generated 3325 bytes in 350 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 13 04:20:52.357388 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-56f1b420-3508-47ce-8060-46860c24a12a None None] Authenticating user token {{(pid=81409) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:20:52.491448 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-56f1b420-3508-47ce-8060-46860c24a12a None None] Validating token access rules against request {{(pid=81409) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:20:52.501516 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-88fb9a03-e701-4387-8d21-51badfefdfaa service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6141b8a6c3d54e179d6879bb019f4d12', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dcec0431fb6849499d6aec35a234daa6', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81409) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:20:52.502163 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88fb9a03-e701-4387-8d21-51badfefdfaa service cinder] REQUEST_METHOD: `GET` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:20:52.502514 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88fb9a03-e701-4387-8d21-51badfefdfaa service cinder] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:20:52.502889 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88fb9a03-e701-4387-8d21-51badfefdfaa service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:20:52.524022 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-88fb9a03-e701-4387-8d21-51badfefdfaa service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:20:52.525540 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-88fb9a03-e701-4387-8d21-51badfefdfaa service cinder] RBAC: Authorization granted {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:20:52.553269 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 665/1308] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Oct 13 04:20:52 2022] GET /identity/v3/auth/tokens => generated 3325 bytes in 197 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Oct 13 04:20:52.555979 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 644/1309] 10.222.0.27 () {58 vars in 1009 bytes} [Thu Oct 13 04:20:52 2022] POST /identity/v3/auth/tokens => generated 3372 bytes in 376 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 13 04:20:52.580259 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-88fb9a03-e701-4387-8d21-51badfefdfaa service cinder] Authenticating user token {{(pid=81409) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:20:52.604715 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-88fb9a03-e701-4387-8d21-51badfefdfaa service cinder] Validating token access rules against request {{(pid=81409) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:20:52.623194 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5ad214fa-678c-4508-8385-389908d85f40 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6141b8a6c3d54e179d6879bb019f4d12', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dcec0431fb6849499d6aec35a234daa6', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81409) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:20:52.624462 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ad214fa-678c-4508-8385-389908d85f40 service cinder] REQUEST_METHOD: `GET` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:20:52.624844 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ad214fa-678c-4508-8385-389908d85f40 service cinder] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:20:52.625115 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ad214fa-678c-4508-8385-389908d85f40 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:20:52.651052 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f40869dd-f27d-4c0e-b0e7-5856c25f5b74 None None] REQUEST_METHOD: `GET` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:20:52.651645 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f40869dd-f27d-4c0e-b0e7-5856c25f5b74 None None] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:20:52.651645 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f40869dd-f27d-4c0e-b0e7-5856c25f5b74 None None] PATH_INFO: `/` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:20:52.653275 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 645/1310] 10.222.0.27 () {54 vars in 912 bytes} [Thu Oct 13 04:20:52 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 13 04:20:52.661846 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2596f147-2a43-4d05-97e9-2fda760bbe10 None None] REQUEST_METHOD: `POST` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:20:52.662240 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2596f147-2a43-4d05-97e9-2fda760bbe10 None None] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:20:52.662606 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2596f147-2a43-4d05-97e9-2fda760bbe10 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:20:52.664069 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5ad214fa-678c-4508-8385-389908d85f40 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:20:52.665501 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5ad214fa-678c-4508-8385-389908d85f40 service cinder] RBAC: Authorization granted {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:20:52.725904 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 666/1311] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Oct 13 04:20:52 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 146 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 13 04:20:52.727105 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.auth.core [None req-2596f147-2a43-4d05-97e9-2fda760bbe10 None None] MFA Rules not processed for user `cb740ed7f7684b079e45f887be83f795`. Rule list: `[]` (Enabled: `True`). {{(pid=81408) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 13 04:20:52.728666 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.fernet_utils [None req-2596f147-2a43-4d05-97e9-2fda760bbe10 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81408) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:20:52.766418 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 646/1312] 10.222.0.27 () {58 vars in 1009 bytes} [Thu Oct 13 04:20:52 2022] POST /identity/v3/auth/tokens => generated 3372 bytes in 105 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 13 04:20:52.810047 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5ad214fa-678c-4508-8385-389908d85f40 service cinder] Authenticating user token {{(pid=81409) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:20:52.830019 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5ad214fa-678c-4508-8385-389908d85f40 service cinder] Validating token access rules against request {{(pid=81409) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:20:52.840344 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-75a64729-27c7-4b5d-b61f-ab1a345154e5 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6141b8a6c3d54e179d6879bb019f4d12', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dcec0431fb6849499d6aec35a234daa6', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81409) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:20:52.841223 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75a64729-27c7-4b5d-b61f-ab1a345154e5 service cinder] REQUEST_METHOD: `GET` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:20:52.841665 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75a64729-27c7-4b5d-b61f-ab1a345154e5 service cinder] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:20:52.842099 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75a64729-27c7-4b5d-b61f-ab1a345154e5 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:20:52.875153 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-75a64729-27c7-4b5d-b61f-ab1a345154e5 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:20:52.882562 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-75a64729-27c7-4b5d-b61f-ab1a345154e5 service cinder] RBAC: Authorization granted {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:20:52.926385 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 667/1313] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Oct 13 04:20:52 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 117 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 13 04:20:54.281461 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2596f147-2a43-4d05-97e9-2fda760bbe10 None None] Authenticating user token {{(pid=81408) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:20:54.307493 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2596f147-2a43-4d05-97e9-2fda760bbe10 None None] Validating token access rules against request {{(pid=81408) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:20:54.315853 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-209c70ee-6e47-45af-8b24-b41c32b317a4 tempest-VolumeQuotaClassesTest-1841142944 tempest-VolumeQuotaClassesTest-1841142944-project-admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '25c7b42411704d48bb1a25778a16d208', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9e491fd308fb4073a62b71c4e0d190d8', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81408) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:20:54.317443 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-209c70ee-6e47-45af-8b24-b41c32b317a4 tempest-VolumeQuotaClassesTest-1841142944 tempest-VolumeQuotaClassesTest-1841142944-project-admin] REQUEST_METHOD: `GET` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:20:54.317950 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-209c70ee-6e47-45af-8b24-b41c32b317a4 tempest-VolumeQuotaClassesTest-1841142944 tempest-VolumeQuotaClassesTest-1841142944-project-admin] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:20:54.319031 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-209c70ee-6e47-45af-8b24-b41c32b317a4 tempest-VolumeQuotaClassesTest-1841142944 tempest-VolumeQuotaClassesTest-1841142944-project-admin] PATH_INFO: `/v3/domains` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:20:54.319906 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-209c70ee-6e47-45af-8b24-b41c32b317a4 tempest-VolumeQuotaClassesTest-1841142944 tempest-VolumeQuotaClassesTest-1841142944-project-admin] RBAC: Adding query filter params (name=Default) {{(pid=81408) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 13 04:20:54.320366 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-209c70ee-6e47-45af-8b24-b41c32b317a4 tempest-VolumeQuotaClassesTest-1841142944 tempest-VolumeQuotaClassesTest-1841142944-project-admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:20:54.325857 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-209c70ee-6e47-45af-8b24-b41c32b317a4 tempest-VolumeQuotaClassesTest-1841142944 tempest-VolumeQuotaClassesTest-1841142944-project-admin] RBAC: Authorization granted {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:20:54.352537 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 647/1314] 10.222.0.27 () {58 vars in 1153 bytes} [Thu Oct 13 04:20:54 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 72 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 13 04:20:54.356695 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-75a64729-27c7-4b5d-b61f-ab1a345154e5 service cinder] Authenticating user token {{(pid=81409) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:20:54.370417 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-75a64729-27c7-4b5d-b61f-ab1a345154e5 service cinder] Validating token access rules against request {{(pid=81409) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:20:54.376014 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bdedec36-a984-460a-a56b-74f8b321b5c9 tempest-VolumeQuotaClassesTest-1841142944 tempest-VolumeQuotaClassesTest-1841142944-project-admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '25c7b42411704d48bb1a25778a16d208', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9e491fd308fb4073a62b71c4e0d190d8', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81409) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:20:54.376885 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bdedec36-a984-460a-a56b-74f8b321b5c9 tempest-VolumeQuotaClassesTest-1841142944 tempest-VolumeQuotaClassesTest-1841142944-project-admin] REQUEST_METHOD: `POST` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:20:54.377309 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bdedec36-a984-460a-a56b-74f8b321b5c9 tempest-VolumeQuotaClassesTest-1841142944 tempest-VolumeQuotaClassesTest-1841142944-project-admin] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:20:54.377724 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bdedec36-a984-460a-a56b-74f8b321b5c9 tempest-VolumeQuotaClassesTest-1841142944 tempest-VolumeQuotaClassesTest-1841142944-project-admin] PATH_INFO: `/v3/projects` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:20:54.378601 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bdedec36-a984-460a-a56b-74f8b321b5c9 tempest-VolumeQuotaClassesTest-1841142944 tempest-VolumeQuotaClassesTest-1841142944-project-admin] RBAC: Authorizing `identity:create_project()` {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:20:54.381497 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bdedec36-a984-460a-a56b-74f8b321b5c9 tempest-VolumeQuotaClassesTest-1841142944 tempest-VolumeQuotaClassesTest-1841142944-project-admin] RBAC: Authorization granted {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:20:54.434839 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 668/1315] 10.222.0.27 () {60 vars in 1153 bytes} [Thu Oct 13 04:20:54 2022] POST /identity/v3/projects => generated 354 bytes in 76 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 13 04:20:54.437966 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-209c70ee-6e47-45af-8b24-b41c32b317a4 tempest-VolumeQuotaClassesTest-1841142944 tempest-VolumeQuotaClassesTest-1841142944-project-admin] Authenticating user token {{(pid=81408) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:20:54.468630 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-209c70ee-6e47-45af-8b24-b41c32b317a4 tempest-VolumeQuotaClassesTest-1841142944 tempest-VolumeQuotaClassesTest-1841142944-project-admin] Validating token access rules against request {{(pid=81408) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:20:54.618135 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5d831438-560a-4697-9294-44f71b8e996e tempest-VolumeQuotaClassesTest-1841142944 tempest-VolumeQuotaClassesTest-1841142944-project-admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '25c7b42411704d48bb1a25778a16d208', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9e491fd308fb4073a62b71c4e0d190d8', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81408) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:20:54.620120 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d831438-560a-4697-9294-44f71b8e996e tempest-VolumeQuotaClassesTest-1841142944 tempest-VolumeQuotaClassesTest-1841142944-project-admin] REQUEST_METHOD: `GET` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:20:54.629392 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d831438-560a-4697-9294-44f71b8e996e tempest-VolumeQuotaClassesTest-1841142944 tempest-VolumeQuotaClassesTest-1841142944-project-admin] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:20:54.630005 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d831438-560a-4697-9294-44f71b8e996e tempest-VolumeQuotaClassesTest-1841142944 tempest-VolumeQuotaClassesTest-1841142944-project-admin] PATH_INFO: `/v3/domains` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:20:54.632479 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5d831438-560a-4697-9294-44f71b8e996e tempest-VolumeQuotaClassesTest-1841142944 tempest-VolumeQuotaClassesTest-1841142944-project-admin] RBAC: Adding query filter params (name=Default) {{(pid=81408) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 13 04:20:54.632929 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5d831438-560a-4697-9294-44f71b8e996e tempest-VolumeQuotaClassesTest-1841142944 tempest-VolumeQuotaClassesTest-1841142944-project-admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:20:54.635801 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5d831438-560a-4697-9294-44f71b8e996e tempest-VolumeQuotaClassesTest-1841142944 tempest-VolumeQuotaClassesTest-1841142944-project-admin] RBAC: Authorization granted {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:20:54.699665 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 648/1316] 10.222.0.27 () {58 vars in 1153 bytes} [Thu Oct 13 04:20:54 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 261 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 13 04:20:54.868228 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bdedec36-a984-460a-a56b-74f8b321b5c9 tempest-VolumeQuotaClassesTest-1841142944 tempest-VolumeQuotaClassesTest-1841142944-project-admin] Authenticating user token {{(pid=81409) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:20:54.897458 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bdedec36-a984-460a-a56b-74f8b321b5c9 tempest-VolumeQuotaClassesTest-1841142944 tempest-VolumeQuotaClassesTest-1841142944-project-admin] Validating token access rules against request {{(pid=81409) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:20:54.907999 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2776dd37-f2be-4363-92ab-078539bca859 tempest-VolumeQuotaClassesTest-1841142944 tempest-VolumeQuotaClassesTest-1841142944-project-admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '25c7b42411704d48bb1a25778a16d208', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9e491fd308fb4073a62b71c4e0d190d8', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81409) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:20:54.909670 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2776dd37-f2be-4363-92ab-078539bca859 tempest-VolumeQuotaClassesTest-1841142944 tempest-VolumeQuotaClassesTest-1841142944-project-admin] REQUEST_METHOD: `DELETE` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:20:54.911952 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2776dd37-f2be-4363-92ab-078539bca859 tempest-VolumeQuotaClassesTest-1841142944 tempest-VolumeQuotaClassesTest-1841142944-project-admin] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:20:54.912699 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2776dd37-f2be-4363-92ab-078539bca859 tempest-VolumeQuotaClassesTest-1841142944 tempest-VolumeQuotaClassesTest-1841142944-project-admin] PATH_INFO: `/v3/projects/c49ab08567b345598d788e04831bb07e` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:20:54.916071 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2776dd37-f2be-4363-92ab-078539bca859 tempest-VolumeQuotaClassesTest-1841142944 tempest-VolumeQuotaClassesTest-1841142944-project-admin] RBAC: Authorizing `identity:delete_project(project_id=c49ab08567b345598d788e04831bb07e)` {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:20:54.918927 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: WARNING py.warnings [None req-2776dd37-f2be-4363-92ab-078539bca859 tempest-VolumeQuotaClassesTest-1841142944 tempest-VolumeQuotaClassesTest-1841142944-project-admin] /usr/local/lib/python3.8/dist-packages/oslo_policy/policy.py:1129: UserWarning: Policy "identity:delete_project": "(role:admin and system_scope:all) or (role:admin and domain_id:%(target.project.domain_id)s)" failed scope check. The token used to make the request was project scoped but the policy requires ['system', 'domain'] scope. This behavior may change in the future where using the intended scope is required Oct 13 04:20:54.918927 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: warnings.warn(msg) Oct 13 04:20:54.918927 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: Oct 13 04:20:54.924402 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2776dd37-f2be-4363-92ab-078539bca859 tempest-VolumeQuotaClassesTest-1841142944 tempest-VolumeQuotaClassesTest-1841142944-project-admin] RBAC: Authorization granted {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:20:54.976198 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG dbcounter [-] [81409] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=81409) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:20:55.058887 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.notifications [None req-2776dd37-f2be-4363-92ab-078539bca859 tempest-VolumeQuotaClassesTest-1841142944 tempest-VolumeQuotaClassesTest-1841142944-project-admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'c49ab08567b345598d788e04831bb07e', 'request_id': 'req-2776dd37-f2be-4363-92ab-078539bca859'} {{(pid=81409) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 13 04:20:55.077580 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.notifications [None req-2776dd37-f2be-4363-92ab-078539bca859 tempest-VolumeQuotaClassesTest-1841142944 tempest-VolumeQuotaClassesTest-1841142944-project-admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'c49ab08567b345598d788e04831bb07e', 'request_id': 'req-2776dd37-f2be-4363-92ab-078539bca859'} {{(pid=81409) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 13 04:20:55.110420 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.notifications [None req-2776dd37-f2be-4363-92ab-078539bca859 tempest-VolumeQuotaClassesTest-1841142944 tempest-VolumeQuotaClassesTest-1841142944-project-admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'c49ab08567b345598d788e04831bb07e', 'request_id': 'req-2776dd37-f2be-4363-92ab-078539bca859'} {{(pid=81409) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 13 04:20:55.134715 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.notifications [None req-2776dd37-f2be-4363-92ab-078539bca859 tempest-VolumeQuotaClassesTest-1841142944 tempest-VolumeQuotaClassesTest-1841142944-project-admin] The token cache is being invalidate because project c49ab08567b345598d788e04831bb07e was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81409) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 13 04:20:55.134952 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.notifications [None req-2776dd37-f2be-4363-92ab-078539bca859 tempest-VolumeQuotaClassesTest-1841142944 tempest-VolumeQuotaClassesTest-1841142944-project-admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81409) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 13 04:20:55.137125 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.notifications [None req-2776dd37-f2be-4363-92ab-078539bca859 tempest-VolumeQuotaClassesTest-1841142944 tempest-VolumeQuotaClassesTest-1841142944-project-admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81409) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 13 04:20:55.147167 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 669/1317] 10.222.0.27 () {58 vars in 1233 bytes} [Thu Oct 13 04:20:54 2022] DELETE /identity/v3/projects/c49ab08567b345598d788e04831bb07e => generated 0 bytes in 281 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 13 04:20:56.078947 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5d831438-560a-4697-9294-44f71b8e996e tempest-VolumeQuotaClassesTest-1841142944 tempest-VolumeQuotaClassesTest-1841142944-project-admin] Authenticating user token {{(pid=81408) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:20:56.081350 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.fernet_utils [None req-5d831438-560a-4697-9294-44f71b8e996e tempest-VolumeQuotaClassesTest-1841142944 tempest-VolumeQuotaClassesTest-1841142944-project-admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81408) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:20:56.258241 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5d831438-560a-4697-9294-44f71b8e996e tempest-VolumeQuotaClassesTest-1841142944 tempest-VolumeQuotaClassesTest-1841142944-project-admin] Validating token access rules against request {{(pid=81408) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:20:56.307457 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG dbcounter [-] [81408] Writing DB stats keystone:SELECT=100 {{(pid=81408) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:20:56.431916 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-163ca8d1-907f-49e2-b508-1686f0448736 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8ba823f313144da3b683bacef8fd62e0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81408) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:20:56.433017 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-163ca8d1-907f-49e2-b508-1686f0448736 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:20:56.433632 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-163ca8d1-907f-49e2-b508-1686f0448736 admin admin] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:20:56.434169 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-163ca8d1-907f-49e2-b508-1686f0448736 admin admin] PATH_INFO: `/v3/users/3525fa9be3fe4959af41d28cd3e4335b` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:20:56.436560 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-163ca8d1-907f-49e2-b508-1686f0448736 admin admin] RBAC: Authorizing `identity:delete_user(user_id=3525fa9be3fe4959af41d28cd3e4335b)` {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:20:56.438341 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: WARNING py.warnings [None req-163ca8d1-907f-49e2-b508-1686f0448736 admin admin] /usr/local/lib/python3.8/dist-packages/oslo_policy/policy.py:1129: UserWarning: Policy "identity:delete_user": "(role:admin and system_scope:all) or (role:admin and token.domain.id:%(target.user.domain_id)s)" failed scope check. The token used to make the request was project scoped but the policy requires ['system', 'domain'] scope. This behavior may change in the future where using the intended scope is required Oct 13 04:20:56.438341 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: warnings.warn(msg) Oct 13 04:20:56.438341 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: Oct 13 04:20:56.441079 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-163ca8d1-907f-49e2-b508-1686f0448736 admin admin] RBAC: Authorization granted {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:20:56.570699 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.notifications [None req-163ca8d1-907f-49e2-b508-1686f0448736 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '3525fa9be3fe4959af41d28cd3e4335b'} {{(pid=81408) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 13 04:20:56.590191 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.notifications [None req-163ca8d1-907f-49e2-b508-1686f0448736 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '3525fa9be3fe4959af41d28cd3e4335b'} {{(pid=81408) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 13 04:20:56.591392 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.notifications [None req-163ca8d1-907f-49e2-b508-1686f0448736 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '3525fa9be3fe4959af41d28cd3e4335b'} {{(pid=81408) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 13 04:20:56.591966 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.notifications [None req-163ca8d1-907f-49e2-b508-1686f0448736 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '3525fa9be3fe4959af41d28cd3e4335b'} {{(pid=81408) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 13 04:20:56.605794 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.notifications [None req-163ca8d1-907f-49e2-b508-1686f0448736 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '3525fa9be3fe4959af41d28cd3e4335b'} {{(pid=81408) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 13 04:20:56.654243 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 649/1318] 10.222.0.27 () {58 vars in 1224 bytes} [Thu Oct 13 04:20:56 2022] DELETE /identity/v3/users/3525fa9be3fe4959af41d28cd3e4335b => generated 0 bytes in 576 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 13 04:20:56.659701 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6138e47-33b6-4aec-8412-e17e3f7e08cd None None] REQUEST_METHOD: `POST` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:20:56.660558 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6138e47-33b6-4aec-8412-e17e3f7e08cd None None] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:20:56.661025 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6138e47-33b6-4aec-8412-e17e3f7e08cd None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:20:56.693273 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.auth.core [None req-b6138e47-33b6-4aec-8412-e17e3f7e08cd None None] MFA Rules not processed for user `0e2d979563d6421995451a0e86bf3022`. Rule list: `[]` (Enabled: `True`). {{(pid=81409) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 13 04:20:56.694556 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.fernet_utils [None req-b6138e47-33b6-4aec-8412-e17e3f7e08cd None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81409) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:20:56.777527 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 670/1319] 10.222.0.27 () {58 vars in 958 bytes} [Thu Oct 13 04:20:56 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 120 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 13 04:20:56.791464 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-163ca8d1-907f-49e2-b508-1686f0448736 admin admin] Authenticating user token {{(pid=81408) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:20:56.797032 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.fernet_utils [None req-163ca8d1-907f-49e2-b508-1686f0448736 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81408) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:20:56.877334 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-163ca8d1-907f-49e2-b508-1686f0448736 admin admin] Validating token access rules against request {{(pid=81408) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:20:56.881571 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6f64b217-17a2-4dd8-974d-55936434a13d service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd8812c3974f84871bb2d761cf0016257', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dcec0431fb6849499d6aec35a234daa6', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81408) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:20:56.882725 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f64b217-17a2-4dd8-974d-55936434a13d service neutron] REQUEST_METHOD: `GET` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:20:56.883219 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f64b217-17a2-4dd8-974d-55936434a13d service neutron] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:20:56.883483 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f64b217-17a2-4dd8-974d-55936434a13d service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:20:56.898978 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6f64b217-17a2-4dd8-974d-55936434a13d service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:20:56.900568 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6f64b217-17a2-4dd8-974d-55936434a13d service neutron] RBAC: Authorization granted {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:20:56.912635 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 650/1320] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Oct 13 04:20:56 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 122 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 13 04:20:57.160282 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b6138e47-33b6-4aec-8412-e17e3f7e08cd None None] Authenticating user token {{(pid=81409) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:20:57.160905 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.fernet_utils [None req-b6138e47-33b6-4aec-8412-e17e3f7e08cd None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81409) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:20:57.175528 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b6138e47-33b6-4aec-8412-e17e3f7e08cd None None] Validating token access rules against request {{(pid=81409) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:20:57.180437 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6a0a8230-6c19-449f-b1fe-44f404ae5184 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8ba823f313144da3b683bacef8fd62e0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81409) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:20:57.181383 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a0a8230-6c19-449f-b1fe-44f404ae5184 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:20:57.181841 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a0a8230-6c19-449f-b1fe-44f404ae5184 admin admin] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:20:57.182317 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a0a8230-6c19-449f-b1fe-44f404ae5184 admin admin] PATH_INFO: `/v3/users/c870f42503d0471881b812c024fddb7f` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:20:57.183725 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6a0a8230-6c19-449f-b1fe-44f404ae5184 admin admin] RBAC: Authorizing `identity:delete_user(user_id=c870f42503d0471881b812c024fddb7f)` {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:20:57.185880 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6a0a8230-6c19-449f-b1fe-44f404ae5184 admin admin] RBAC: Authorization granted {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:20:57.255235 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.notifications [None req-6a0a8230-6c19-449f-b1fe-44f404ae5184 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'c870f42503d0471881b812c024fddb7f'} {{(pid=81409) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 13 04:20:57.256254 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.notifications [None req-6a0a8230-6c19-449f-b1fe-44f404ae5184 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'c870f42503d0471881b812c024fddb7f'} {{(pid=81409) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 13 04:20:57.266513 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.notifications [None req-6a0a8230-6c19-449f-b1fe-44f404ae5184 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'c870f42503d0471881b812c024fddb7f'} {{(pid=81409) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 13 04:20:57.268275 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.notifications [None req-6a0a8230-6c19-449f-b1fe-44f404ae5184 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'c870f42503d0471881b812c024fddb7f'} {{(pid=81409) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 13 04:20:57.280208 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.notifications [None req-6a0a8230-6c19-449f-b1fe-44f404ae5184 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'c870f42503d0471881b812c024fddb7f'} {{(pid=81409) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 13 04:20:57.312518 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 671/1321] 10.222.0.27 () {58 vars in 1224 bytes} [Thu Oct 13 04:20:57 2022] DELETE /identity/v3/users/c870f42503d0471881b812c024fddb7f => generated 0 bytes in 154 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 13 04:20:57.320016 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-026e85ff-333b-4bfb-bd86-5e0a21bd89e7 None None] REQUEST_METHOD: `POST` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:20:57.322695 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-026e85ff-333b-4bfb-bd86-5e0a21bd89e7 None None] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:20:57.323163 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-026e85ff-333b-4bfb-bd86-5e0a21bd89e7 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:20:57.354172 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.auth.core [None req-026e85ff-333b-4bfb-bd86-5e0a21bd89e7 None None] MFA Rules not processed for user `0e2d979563d6421995451a0e86bf3022`. Rule list: `[]` (Enabled: `True`). {{(pid=81408) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 13 04:20:57.359525 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.fernet_utils [None req-026e85ff-333b-4bfb-bd86-5e0a21bd89e7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81408) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:20:57.456063 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG dbcounter [-] [81408] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81408) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:20:57.496504 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 651/1322] 10.222.0.27 () {58 vars in 958 bytes} [Thu Oct 13 04:20:57 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 179 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 13 04:20:57.556190 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6a0a8230-6c19-449f-b1fe-44f404ae5184 admin admin] Authenticating user token {{(pid=81409) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:20:57.558646 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.fernet_utils [None req-6a0a8230-6c19-449f-b1fe-44f404ae5184 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81409) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:20:57.607547 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG dbcounter [-] [81409] Writing DB stats keystone:SELECT=81,keystone:DELETE=17,keystone:INSERT=2 {{(pid=81409) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:20:57.683333 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6a0a8230-6c19-449f-b1fe-44f404ae5184 admin admin] Validating token access rules against request {{(pid=81409) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:20:57.687688 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d3a62862-3333-468e-a89d-d78f3c1d31c8 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd8812c3974f84871bb2d761cf0016257', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dcec0431fb6849499d6aec35a234daa6', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81409) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:20:57.688502 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3a62862-3333-468e-a89d-d78f3c1d31c8 service neutron] REQUEST_METHOD: `GET` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:20:57.688882 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3a62862-3333-468e-a89d-d78f3c1d31c8 service neutron] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:20:57.689197 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3a62862-3333-468e-a89d-d78f3c1d31c8 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:20:57.707432 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d3a62862-3333-468e-a89d-d78f3c1d31c8 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:20:57.711164 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d3a62862-3333-468e-a89d-d78f3c1d31c8 service neutron] RBAC: Authorization granted {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:20:57.737943 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 672/1323] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Oct 13 04:20:57 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 182 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 13 04:20:57.772931 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-026e85ff-333b-4bfb-bd86-5e0a21bd89e7 None None] Authenticating user token {{(pid=81408) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:20:57.775567 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.fernet_utils [None req-026e85ff-333b-4bfb-bd86-5e0a21bd89e7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81408) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:20:57.794256 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-026e85ff-333b-4bfb-bd86-5e0a21bd89e7 None None] Validating token access rules against request {{(pid=81408) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:20:57.809071 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e04c5110-9319-4f1b-a172-d32f33bd74c3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8ba823f313144da3b683bacef8fd62e0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81408) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:20:57.809857 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e04c5110-9319-4f1b-a172-d32f33bd74c3 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:20:57.810253 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e04c5110-9319-4f1b-a172-d32f33bd74c3 admin admin] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:20:57.810624 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e04c5110-9319-4f1b-a172-d32f33bd74c3 admin admin] PATH_INFO: `/v3/projects/ebafd8e9b47244bb9c8da2f570b41b3a` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:20:57.814102 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e04c5110-9319-4f1b-a172-d32f33bd74c3 admin admin] RBAC: Authorizing `identity:delete_project(project_id=ebafd8e9b47244bb9c8da2f570b41b3a)` {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:20:57.816371 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e04c5110-9319-4f1b-a172-d32f33bd74c3 admin admin] RBAC: Authorization granted {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:20:57.908569 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.notifications [None req-e04c5110-9319-4f1b-a172-d32f33bd74c3 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'ebafd8e9b47244bb9c8da2f570b41b3a', 'request_id': 'req-e04c5110-9319-4f1b-a172-d32f33bd74c3'} {{(pid=81408) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 13 04:20:57.920479 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.notifications [None req-e04c5110-9319-4f1b-a172-d32f33bd74c3 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'ebafd8e9b47244bb9c8da2f570b41b3a', 'request_id': 'req-e04c5110-9319-4f1b-a172-d32f33bd74c3'} {{(pid=81408) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 13 04:20:57.934398 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.notifications [None req-e04c5110-9319-4f1b-a172-d32f33bd74c3 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'ebafd8e9b47244bb9c8da2f570b41b3a', 'request_id': 'req-e04c5110-9319-4f1b-a172-d32f33bd74c3'} {{(pid=81408) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 13 04:20:57.953842 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.notifications [None req-e04c5110-9319-4f1b-a172-d32f33bd74c3 admin admin] The token cache is being invalidate because project ebafd8e9b47244bb9c8da2f570b41b3a was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81408) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 13 04:20:57.954633 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.notifications [None req-e04c5110-9319-4f1b-a172-d32f33bd74c3 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81408) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 13 04:20:57.958267 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.notifications [None req-e04c5110-9319-4f1b-a172-d32f33bd74c3 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81408) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 13 04:20:57.961592 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 652/1324] 10.222.0.27 () {58 vars in 1233 bytes} [Thu Oct 13 04:20:57 2022] DELETE /identity/v3/projects/ebafd8e9b47244bb9c8da2f570b41b3a => generated 0 bytes in 189 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 13 04:20:57.965389 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d3a62862-3333-468e-a89d-d78f3c1d31c8 service neutron] Authenticating user token {{(pid=81409) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:20:57.977408 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d3a62862-3333-468e-a89d-d78f3c1d31c8 service neutron] Validating token access rules against request {{(pid=81409) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:20:58.057519 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0169fdb6-c3dc-4e37-b937-8182824f9585 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8ba823f313144da3b683bacef8fd62e0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81409) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:20:58.058360 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0169fdb6-c3dc-4e37-b937-8182824f9585 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:20:58.058738 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0169fdb6-c3dc-4e37-b937-8182824f9585 admin admin] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:20:58.059150 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0169fdb6-c3dc-4e37-b937-8182824f9585 admin admin] PATH_INFO: `/v3/users/25c7b42411704d48bb1a25778a16d208` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:20:58.060414 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0169fdb6-c3dc-4e37-b937-8182824f9585 admin admin] RBAC: Authorizing `identity:delete_user(user_id=25c7b42411704d48bb1a25778a16d208)` {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:20:58.064123 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0169fdb6-c3dc-4e37-b937-8182824f9585 admin admin] RBAC: Authorization granted {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:20:58.161615 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.notifications [None req-0169fdb6-c3dc-4e37-b937-8182824f9585 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '25c7b42411704d48bb1a25778a16d208'} {{(pid=81409) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 13 04:20:58.162371 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.notifications [None req-0169fdb6-c3dc-4e37-b937-8182824f9585 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '25c7b42411704d48bb1a25778a16d208'} {{(pid=81409) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 13 04:20:58.173165 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.notifications [None req-0169fdb6-c3dc-4e37-b937-8182824f9585 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '25c7b42411704d48bb1a25778a16d208'} {{(pid=81409) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 13 04:20:58.174492 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.notifications [None req-0169fdb6-c3dc-4e37-b937-8182824f9585 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '25c7b42411704d48bb1a25778a16d208'} {{(pid=81409) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 13 04:20:58.186886 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.notifications [None req-0169fdb6-c3dc-4e37-b937-8182824f9585 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '25c7b42411704d48bb1a25778a16d208'} {{(pid=81409) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 13 04:20:58.224635 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 673/1325] 10.222.0.27 () {58 vars in 1224 bytes} [Thu Oct 13 04:20:57 2022] DELETE /identity/v3/users/25c7b42411704d48bb1a25778a16d208 => generated 0 bytes in 256 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 13 04:20:58.229003 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05aadeb0-6889-496e-9557-af3e5734eb0d None None] REQUEST_METHOD: `POST` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:20:58.229320 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05aadeb0-6889-496e-9557-af3e5734eb0d None None] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:20:58.229545 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05aadeb0-6889-496e-9557-af3e5734eb0d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:20:58.273275 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.auth.core [None req-05aadeb0-6889-496e-9557-af3e5734eb0d None None] MFA Rules not processed for user `999c89e1bc7a426886fc802e28d38596`. Rule list: `[]` (Enabled: `True`). {{(pid=81408) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 13 04:20:58.274881 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.fernet_utils [None req-05aadeb0-6889-496e-9557-af3e5734eb0d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81408) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:20:58.409925 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 653/1326] 10.222.0.27 () {58 vars in 958 bytes} [Thu Oct 13 04:20:58 2022] POST /identity/v3/auth/tokens => generated 3260 bytes in 182 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 13 04:20:58.426176 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0169fdb6-c3dc-4e37-b937-8182824f9585 admin admin] Authenticating user token {{(pid=81409) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:20:58.427780 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.fernet_utils [None req-0169fdb6-c3dc-4e37-b937-8182824f9585 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81409) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:20:58.579686 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG dbcounter [-] [81409] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81409) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:20:58.696767 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0169fdb6-c3dc-4e37-b937-8182824f9585 admin admin] Validating token access rules against request {{(pid=81409) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:20:58.714503 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-02061d54-b623-4b31-95ae-29685ac7b010 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cb740ed7f7684b079e45f887be83f795', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dcec0431fb6849499d6aec35a234daa6', 'project_domain_id': 'default', 'roles': ['ResellerAdmin', 'admin', 'reader', 'service', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81409) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:20:58.714503 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-02061d54-b623-4b31-95ae-29685ac7b010 service nova] REQUEST_METHOD: `GET` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:20:58.714503 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-02061d54-b623-4b31-95ae-29685ac7b010 service nova] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:20:58.714503 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-02061d54-b623-4b31-95ae-29685ac7b010 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:20:58.728895 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-02061d54-b623-4b31-95ae-29685ac7b010 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:20:58.731313 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-02061d54-b623-4b31-95ae-29685ac7b010 service nova] RBAC: Authorization granted {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:20:58.768036 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 674/1327] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Oct 13 04:20:58 2022] GET /identity/v3/auth/tokens => generated 3260 bytes in 342 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 13 04:21:00.481954 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-05aadeb0-6889-496e-9557-af3e5734eb0d None None] Authenticating user token {{(pid=81408) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:21:00.484265 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.fernet_utils [None req-05aadeb0-6889-496e-9557-af3e5734eb0d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81408) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:21:00.688091 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-05aadeb0-6889-496e-9557-af3e5734eb0d None None] Validating token access rules against request {{(pid=81408) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:21:00.705662 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7f2d9f26-34ea-4fb9-9b05-657bcb13a69a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8ba823f313144da3b683bacef8fd62e0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81408) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:21:00.708339 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f2d9f26-34ea-4fb9-9b05-657bcb13a69a admin admin] REQUEST_METHOD: `DELETE` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:21:00.708920 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f2d9f26-34ea-4fb9-9b05-657bcb13a69a admin admin] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:21:00.709685 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f2d9f26-34ea-4fb9-9b05-657bcb13a69a admin admin] PATH_INFO: `/v3/projects/9e491fd308fb4073a62b71c4e0d190d8` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:21:00.713329 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7f2d9f26-34ea-4fb9-9b05-657bcb13a69a admin admin] RBAC: Authorizing `identity:delete_project(project_id=9e491fd308fb4073a62b71c4e0d190d8)` {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:21:00.721264 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7f2d9f26-34ea-4fb9-9b05-657bcb13a69a admin admin] RBAC: Authorization granted {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:21:00.728931 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG dbcounter [-] [81408] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=81408) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:21:00.851218 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-02061d54-b623-4b31-95ae-29685ac7b010 service nova] Authenticating user token {{(pid=81409) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:21:00.853429 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.fernet_utils [None req-02061d54-b623-4b31-95ae-29685ac7b010 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81409) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:21:00.883077 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-02061d54-b623-4b31-95ae-29685ac7b010 service nova] Validating token access rules against request {{(pid=81409) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:21:00.892512 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.notifications [None req-7f2d9f26-34ea-4fb9-9b05-657bcb13a69a admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '9e491fd308fb4073a62b71c4e0d190d8', 'request_id': 'req-7f2d9f26-34ea-4fb9-9b05-657bcb13a69a'} {{(pid=81408) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 13 04:21:00.914580 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.notifications [None req-7f2d9f26-34ea-4fb9-9b05-657bcb13a69a admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '9e491fd308fb4073a62b71c4e0d190d8', 'request_id': 'req-7f2d9f26-34ea-4fb9-9b05-657bcb13a69a'} {{(pid=81408) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 13 04:21:00.931940 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.notifications [None req-7f2d9f26-34ea-4fb9-9b05-657bcb13a69a admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '9e491fd308fb4073a62b71c4e0d190d8', 'request_id': 'req-7f2d9f26-34ea-4fb9-9b05-657bcb13a69a'} {{(pid=81408) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 13 04:21:00.965488 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.notifications [None req-7f2d9f26-34ea-4fb9-9b05-657bcb13a69a admin admin] The token cache is being invalidate because project 9e491fd308fb4073a62b71c4e0d190d8 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81408) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 13 04:21:00.965789 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.notifications [None req-7f2d9f26-34ea-4fb9-9b05-657bcb13a69a admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81408) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 13 04:21:00.970650 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.notifications [None req-7f2d9f26-34ea-4fb9-9b05-657bcb13a69a admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81408) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 13 04:21:00.976154 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 654/1328] 10.222.0.27 () {58 vars in 1233 bytes} [Thu Oct 13 04:21:00 2022] DELETE /identity/v3/projects/9e491fd308fb4073a62b71c4e0d190d8 => generated 0 bytes in 494 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 13 04:21:00.981476 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c53cdbc4-5900-40da-8ee2-e53911eabdc0 None None] REQUEST_METHOD: `POST` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:21:00.981763 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c53cdbc4-5900-40da-8ee2-e53911eabdc0 None None] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:21:00.982046 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c53cdbc4-5900-40da-8ee2-e53911eabdc0 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:21:01.041213 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c8af6a5e-806c-4800-b414-4181c4ffb17a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8ba823f313144da3b683bacef8fd62e0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81409) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:21:01.041952 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8af6a5e-806c-4800-b414-4181c4ffb17a admin admin] REQUEST_METHOD: `DELETE` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:21:01.042368 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8af6a5e-806c-4800-b414-4181c4ffb17a admin admin] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:21:01.042779 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8af6a5e-806c-4800-b414-4181c4ffb17a admin admin] PATH_INFO: `/v3/projects/bb1bd95475ad40029bd402081f61a70c` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:21:01.045197 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c8af6a5e-806c-4800-b414-4181c4ffb17a admin admin] RBAC: Authorizing `identity:delete_project(project_id=bb1bd95475ad40029bd402081f61a70c)` {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:21:01.054829 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c8af6a5e-806c-4800-b414-4181c4ffb17a admin admin] RBAC: Authorization granted {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:21:01.123862 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.auth.core [None req-c53cdbc4-5900-40da-8ee2-e53911eabdc0 None None] MFA Rules not processed for user `0e2d979563d6421995451a0e86bf3022`. Rule list: `[]` (Enabled: `True`). {{(pid=81408) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 13 04:21:01.125268 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.fernet_utils [None req-c53cdbc4-5900-40da-8ee2-e53911eabdc0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81408) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:21:01.163631 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.notifications [None req-c8af6a5e-806c-4800-b414-4181c4ffb17a admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'bb1bd95475ad40029bd402081f61a70c', 'request_id': 'req-c8af6a5e-806c-4800-b414-4181c4ffb17a'} {{(pid=81409) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 13 04:21:01.183561 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.notifications [None req-c8af6a5e-806c-4800-b414-4181c4ffb17a admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'bb1bd95475ad40029bd402081f61a70c', 'request_id': 'req-c8af6a5e-806c-4800-b414-4181c4ffb17a'} {{(pid=81409) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 13 04:21:01.200375 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.notifications [None req-c8af6a5e-806c-4800-b414-4181c4ffb17a admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'bb1bd95475ad40029bd402081f61a70c', 'request_id': 'req-c8af6a5e-806c-4800-b414-4181c4ffb17a'} {{(pid=81409) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 13 04:21:01.223695 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.notifications [None req-c8af6a5e-806c-4800-b414-4181c4ffb17a admin admin] The token cache is being invalidate because project bb1bd95475ad40029bd402081f61a70c was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81409) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 13 04:21:01.224138 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.notifications [None req-c8af6a5e-806c-4800-b414-4181c4ffb17a admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81409) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 13 04:21:01.225091 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.notifications [None req-c8af6a5e-806c-4800-b414-4181c4ffb17a admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81409) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 13 04:21:01.227653 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 675/1329] 10.222.0.27 () {58 vars in 1233 bytes} [Thu Oct 13 04:21:00 2022] DELETE /identity/v3/projects/bb1bd95475ad40029bd402081f61a70c => generated 0 bytes in 376 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 13 04:21:01.232205 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac2dd9d1-cc4a-4710-8534-56a40b36e32f None None] REQUEST_METHOD: `POST` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:21:01.234503 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac2dd9d1-cc4a-4710-8534-56a40b36e32f None None] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:21:01.234910 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac2dd9d1-cc4a-4710-8534-56a40b36e32f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:21:01.318179 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.auth.core [None req-ac2dd9d1-cc4a-4710-8534-56a40b36e32f None None] MFA Rules not processed for user `0e2d979563d6421995451a0e86bf3022`. Rule list: `[]` (Enabled: `True`). {{(pid=81409) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 13 04:21:01.345908 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.fernet_utils [None req-ac2dd9d1-cc4a-4710-8534-56a40b36e32f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81409) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:21:01.424416 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 655/1330] 10.222.0.27 () {58 vars in 958 bytes} [Thu Oct 13 04:21:00 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 445 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 13 04:21:01.437123 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9738cfb-2e75-4ae6-b02e-9049f185be69 None None] REQUEST_METHOD: `POST` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:21:01.437680 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9738cfb-2e75-4ae6-b02e-9049f185be69 None None] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:21:01.437876 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9738cfb-2e75-4ae6-b02e-9049f185be69 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:21:01.516474 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.auth.core [None req-f9738cfb-2e75-4ae6-b02e-9049f185be69 None None] MFA Rules not processed for user `0e2d979563d6421995451a0e86bf3022`. Rule list: `[]` (Enabled: `True`). {{(pid=81408) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 13 04:21:01.520217 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.fernet_utils [None req-f9738cfb-2e75-4ae6-b02e-9049f185be69 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81408) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:21:01.546505 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 656/1331] 10.222.0.27 () {58 vars in 958 bytes} [Thu Oct 13 04:21:01 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 112 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 13 04:21:01.550787 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f9738cfb-2e75-4ae6-b02e-9049f185be69 None None] Authenticating user token {{(pid=81408) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:21:01.581353 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f9738cfb-2e75-4ae6-b02e-9049f185be69 None None] Validating token access rules against request {{(pid=81408) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:21:01.598704 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-47f66745-95ae-4bcd-8cae-01af0b294670 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8ba823f313144da3b683bacef8fd62e0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81408) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:21:01.607475 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG dbcounter [-] [81409] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=81409) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:21:01.623132 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47f66745-95ae-4bcd-8cae-01af0b294670 admin admin] REQUEST_METHOD: `GET` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:21:01.625179 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47f66745-95ae-4bcd-8cae-01af0b294670 admin admin] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:21:01.628223 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47f66745-95ae-4bcd-8cae-01af0b294670 admin admin] PATH_INFO: `/v3/domains` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:21:01.630574 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-47f66745-95ae-4bcd-8cae-01af0b294670 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81408) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 13 04:21:01.641045 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 676/1332] 10.222.0.27 () {58 vars in 958 bytes} [Thu Oct 13 04:21:01 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 404 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 13 04:21:01.643037 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-47f66745-95ae-4bcd-8cae-01af0b294670 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:21:01.643372 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d40881a1-68b2-4c7b-b7d8-090cac624ffc None None] REQUEST_METHOD: `POST` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:21:01.643706 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d40881a1-68b2-4c7b-b7d8-090cac624ffc None None] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:21:01.643986 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d40881a1-68b2-4c7b-b7d8-090cac624ffc None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:21:01.648209 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-47f66745-95ae-4bcd-8cae-01af0b294670 admin admin] RBAC: Authorization granted {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:21:01.687765 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 657/1333] 10.222.0.27 () {58 vars in 1153 bytes} [Thu Oct 13 04:21:01 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 137 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 13 04:21:01.693145 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-47f66745-95ae-4bcd-8cae-01af0b294670 admin admin] Authenticating user token {{(pid=81408) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:21:01.698226 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.auth.core [None req-d40881a1-68b2-4c7b-b7d8-090cac624ffc None None] MFA Rules not processed for user `0e2d979563d6421995451a0e86bf3022`. Rule list: `[]` (Enabled: `True`). {{(pid=81409) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 13 04:21:01.699518 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.fernet_utils [None req-d40881a1-68b2-4c7b-b7d8-090cac624ffc None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81409) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:21:01.722522 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 677/1334] 10.222.0.27 () {58 vars in 958 bytes} [Thu Oct 13 04:21:01 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 78 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 13 04:21:01.724534 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d40881a1-68b2-4c7b-b7d8-090cac624ffc None None] Authenticating user token {{(pid=81409) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:21:01.738414 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-47f66745-95ae-4bcd-8cae-01af0b294670 admin admin] Validating token access rules against request {{(pid=81408) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:21:01.746656 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7c9b162a-e04d-4767-a78a-9cdb6ac07ee7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8ba823f313144da3b683bacef8fd62e0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81408) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:21:01.747299 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c9b162a-e04d-4767-a78a-9cdb6ac07ee7 admin admin] REQUEST_METHOD: `POST` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:21:01.747760 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c9b162a-e04d-4767-a78a-9cdb6ac07ee7 admin admin] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:21:01.747981 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c9b162a-e04d-4767-a78a-9cdb6ac07ee7 admin admin] PATH_INFO: `/v3/projects` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:21:01.749229 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7c9b162a-e04d-4767-a78a-9cdb6ac07ee7 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:21:01.753197 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7c9b162a-e04d-4767-a78a-9cdb6ac07ee7 admin admin] RBAC: Authorization granted {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:21:01.770302 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d40881a1-68b2-4c7b-b7d8-090cac624ffc None None] Validating token access rules against request {{(pid=81409) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:21:01.783176 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3347436a-30ec-40af-ad7e-ff66d215e667 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8ba823f313144da3b683bacef8fd62e0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81409) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:21:01.784133 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3347436a-30ec-40af-ad7e-ff66d215e667 admin admin] REQUEST_METHOD: `GET` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:21:01.784583 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3347436a-30ec-40af-ad7e-ff66d215e667 admin admin] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:21:01.785022 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3347436a-30ec-40af-ad7e-ff66d215e667 admin admin] PATH_INFO: `/v3/domains` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:21:01.786062 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3347436a-30ec-40af-ad7e-ff66d215e667 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81409) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 13 04:21:01.786396 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3347436a-30ec-40af-ad7e-ff66d215e667 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:21:01.788030 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3347436a-30ec-40af-ad7e-ff66d215e667 admin admin] RBAC: Authorization granted {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:21:01.851761 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 678/1335] 10.222.0.27 () {58 vars in 1153 bytes} [Thu Oct 13 04:21:01 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 128 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 13 04:21:01.856593 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3347436a-30ec-40af-ad7e-ff66d215e667 admin admin] Authenticating user token {{(pid=81409) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:21:01.862482 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 658/1336] 10.222.0.27 () {60 vars in 1153 bytes} [Thu Oct 13 04:21:01 2022] POST /identity/v3/projects => generated 391 bytes in 170 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 13 04:21:01.863944 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7c9b162a-e04d-4767-a78a-9cdb6ac07ee7 admin admin] Authenticating user token {{(pid=81408) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:21:01.871777 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3347436a-30ec-40af-ad7e-ff66d215e667 admin admin] Validating token access rules against request {{(pid=81409) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:21:01.988268 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG dbcounter [-] [81408] Writing DB stats keystone:SELECT=91,keystone:DELETE=7,keystone:INSERT=2 {{(pid=81408) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:21:02.005929 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-efcec3e7-9e7a-4506-b828-517fff9a15e3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8ba823f313144da3b683bacef8fd62e0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81409) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:21:02.006499 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-efcec3e7-9e7a-4506-b828-517fff9a15e3 admin admin] REQUEST_METHOD: `POST` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:21:02.006774 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-efcec3e7-9e7a-4506-b828-517fff9a15e3 admin admin] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:21:02.007038 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-efcec3e7-9e7a-4506-b828-517fff9a15e3 admin admin] PATH_INFO: `/v3/projects` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:21:02.007859 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-efcec3e7-9e7a-4506-b828-517fff9a15e3 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:21:02.009954 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-efcec3e7-9e7a-4506-b828-517fff9a15e3 admin admin] RBAC: Authorization granted {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:21:02.020579 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7c9b162a-e04d-4767-a78a-9cdb6ac07ee7 admin admin] Validating token access rules against request {{(pid=81408) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:21:02.055041 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b82bd626-5b4d-4164-b17e-538350d7b422 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8ba823f313144da3b683bacef8fd62e0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81408) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:21:02.055780 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b82bd626-5b4d-4164-b17e-538350d7b422 admin admin] REQUEST_METHOD: `POST` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:21:02.056049 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b82bd626-5b4d-4164-b17e-538350d7b422 admin admin] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:21:02.056308 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b82bd626-5b4d-4164-b17e-538350d7b422 admin admin] PATH_INFO: `/v3/users` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:21:02.056968 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b82bd626-5b4d-4164-b17e-538350d7b422 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:21:02.059059 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b82bd626-5b4d-4164-b17e-538350d7b422 admin admin] RBAC: Authorization granted {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:21:02.099074 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 679/1337] 10.222.0.27 () {60 vars in 1153 bytes} [Thu Oct 13 04:21:01 2022] POST /identity/v3/projects => generated 375 bytes in 243 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 13 04:21:02.111745 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-efcec3e7-9e7a-4506-b828-517fff9a15e3 admin admin] Authenticating user token {{(pid=81409) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:21:02.115568 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 659/1338] 10.222.0.27 () {60 vars in 1144 bytes} [Thu Oct 13 04:21:01 2022] POST /identity/v3/users => generated 296 bytes in 239 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 13 04:21:02.120028 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b82bd626-5b4d-4164-b17e-538350d7b422 admin admin] Authenticating user token {{(pid=81408) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:21:02.151981 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b82bd626-5b4d-4164-b17e-538350d7b422 admin admin] Validating token access rules against request {{(pid=81408) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:21:02.282295 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-11422433-5ee1-4070-8ac6-7cd52424641a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8ba823f313144da3b683bacef8fd62e0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81408) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:21:02.282927 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11422433-5ee1-4070-8ac6-7cd52424641a admin admin] REQUEST_METHOD: `GET` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:21:02.283206 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11422433-5ee1-4070-8ac6-7cd52424641a admin admin] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:21:02.283548 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11422433-5ee1-4070-8ac6-7cd52424641a admin admin] PATH_INFO: `/v3/roles` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:21:02.284177 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-11422433-5ee1-4070-8ac6-7cd52424641a admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:21:02.286808 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-11422433-5ee1-4070-8ac6-7cd52424641a admin admin] RBAC: Authorization granted {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:21:02.292159 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-efcec3e7-9e7a-4506-b828-517fff9a15e3 admin admin] Validating token access rules against request {{(pid=81409) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:21:02.304333 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 660/1339] 10.222.0.27 () {58 vars in 1122 bytes} [Thu Oct 13 04:21:02 2022] GET /identity/v3/roles => generated 1420 bytes in 185 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 13 04:21:02.304545 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-08e40083-eea0-427c-917c-46f0cbdccd85 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8ba823f313144da3b683bacef8fd62e0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81409) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:21:02.304545 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08e40083-eea0-427c-917c-46f0cbdccd85 admin admin] REQUEST_METHOD: `POST` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:21:02.305016 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08e40083-eea0-427c-917c-46f0cbdccd85 admin admin] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:21:02.305398 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08e40083-eea0-427c-917c-46f0cbdccd85 admin admin] PATH_INFO: `/v3/users` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:21:02.305513 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-11422433-5ee1-4070-8ac6-7cd52424641a admin admin] Authenticating user token {{(pid=81408) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:21:02.310378 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-08e40083-eea0-427c-917c-46f0cbdccd85 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:21:02.312921 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-08e40083-eea0-427c-917c-46f0cbdccd85 admin admin] RBAC: Authorization granted {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:21:02.318586 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-11422433-5ee1-4070-8ac6-7cd52424641a admin admin] Validating token access rules against request {{(pid=81408) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:21:02.324143 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-afd9639f-0d03-43a8-9b2e-9f8239495392 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8ba823f313144da3b683bacef8fd62e0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81408) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:21:02.324979 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-afd9639f-0d03-43a8-9b2e-9f8239495392 admin admin] REQUEST_METHOD: `GET` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:21:02.325431 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-afd9639f-0d03-43a8-9b2e-9f8239495392 admin admin] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:21:02.325853 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-afd9639f-0d03-43a8-9b2e-9f8239495392 admin admin] PATH_INFO: `/v3/roles` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:21:02.327863 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-afd9639f-0d03-43a8-9b2e-9f8239495392 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:21:02.329928 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-afd9639f-0d03-43a8-9b2e-9f8239495392 admin admin] RBAC: Authorization granted {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:21:02.349732 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 661/1340] 10.222.0.27 () {58 vars in 1122 bytes} [Thu Oct 13 04:21:02 2022] GET /identity/v3/roles => generated 1420 bytes in 44 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Oct 13 04:21:02.352045 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-afd9639f-0d03-43a8-9b2e-9f8239495392 admin admin] Authenticating user token {{(pid=81408) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:21:02.365725 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 680/1341] 10.222.0.27 () {60 vars in 1144 bytes} [Thu Oct 13 04:21:02 2022] POST /identity/v3/users => generated 288 bytes in 262 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 13 04:21:02.370216 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-08e40083-eea0-427c-917c-46f0cbdccd85 admin admin] Authenticating user token {{(pid=81409) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:21:02.370550 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-afd9639f-0d03-43a8-9b2e-9f8239495392 admin admin] Validating token access rules against request {{(pid=81408) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:21:02.377056 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-23843292-f7f7-498a-9084-83ce9c0c38dd admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8ba823f313144da3b683bacef8fd62e0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81408) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:21:02.377749 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23843292-f7f7-498a-9084-83ce9c0c38dd admin admin] REQUEST_METHOD: `PUT` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:21:02.378026 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23843292-f7f7-498a-9084-83ce9c0c38dd admin admin] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:21:02.378350 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23843292-f7f7-498a-9084-83ce9c0c38dd admin admin] PATH_INFO: `/v3/projects/862fb54ea0634004989eb055ce0f7300/users/d15c3f1e7b4f4f188466ccf02a1d33b4/roles/cc56c4ae13a7424ea771c15da4b153c1` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:21:02.394298 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-08e40083-eea0-427c-917c-46f0cbdccd85 admin admin] Validating token access rules against request {{(pid=81409) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:21:02.402513 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-23843292-f7f7-498a-9084-83ce9c0c38dd admin admin] RBAC: Authorizing `identity:create_grant(project_id=862fb54ea0634004989eb055ce0f7300, user_id=d15c3f1e7b4f4f188466ccf02a1d33b4, role_id=cc56c4ae13a7424ea771c15da4b153c1)` {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:21:02.403357 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5d54d45b-0ec6-41c9-934a-3e6582636f3b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8ba823f313144da3b683bacef8fd62e0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81409) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:21:02.404283 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d54d45b-0ec6-41c9-934a-3e6582636f3b admin admin] REQUEST_METHOD: `GET` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:21:02.418745 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d54d45b-0ec6-41c9-934a-3e6582636f3b admin admin] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:21:02.419906 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d54d45b-0ec6-41c9-934a-3e6582636f3b admin admin] PATH_INFO: `/v3/roles` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:21:02.421207 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5d54d45b-0ec6-41c9-934a-3e6582636f3b admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:21:02.423434 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5d54d45b-0ec6-41c9-934a-3e6582636f3b admin admin] RBAC: Authorization granted {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:21:02.425967 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-23843292-f7f7-498a-9084-83ce9c0c38dd admin admin] RBAC: Authorization granted {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:21:02.440478 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 681/1342] 10.222.0.27 () {58 vars in 1122 bytes} [Thu Oct 13 04:21:02 2022] GET /identity/v3/roles => generated 1420 bytes in 71 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 13 04:21:02.451279 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5d54d45b-0ec6-41c9-934a-3e6582636f3b admin admin] Authenticating user token {{(pid=81409) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:21:02.482125 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 662/1343] 10.222.0.27 () {60 vars in 1483 bytes} [Thu Oct 13 04:21:02 2022] PUT /identity/v3/projects/862fb54ea0634004989eb055ce0f7300/users/d15c3f1e7b4f4f188466ccf02a1d33b4/roles/cc56c4ae13a7424ea771c15da4b153c1 => generated 0 bytes in 131 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 13 04:21:02.487429 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5d54d45b-0ec6-41c9-934a-3e6582636f3b admin admin] Validating token access rules against request {{(pid=81409) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:21:02.515256 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-73f08ea2-5b4b-466d-bd24-ccf8ebbbde20 None None] REQUEST_METHOD: `POST` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:21:02.515256 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-73f08ea2-5b4b-466d-bd24-ccf8ebbbde20 None None] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:21:02.515256 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-73f08ea2-5b4b-466d-bd24-ccf8ebbbde20 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:21:02.582467 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.auth.core [None req-73f08ea2-5b4b-466d-bd24-ccf8ebbbde20 None None] MFA Rules not processed for user `d15c3f1e7b4f4f188466ccf02a1d33b4`. Rule list: `[]` (Enabled: `True`). {{(pid=81408) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 13 04:21:02.583999 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG dbcounter [-] [81409] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=81409) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:21:02.584992 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.fernet_utils [None req-73f08ea2-5b4b-466d-bd24-ccf8ebbbde20 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81408) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:21:02.609529 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8465ade7-8490-49d7-ab16-068620f7821d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8ba823f313144da3b683bacef8fd62e0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81409) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:21:02.610808 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8465ade7-8490-49d7-ab16-068620f7821d admin admin] REQUEST_METHOD: `GET` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:21:02.611213 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8465ade7-8490-49d7-ab16-068620f7821d admin admin] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:21:02.611587 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8465ade7-8490-49d7-ab16-068620f7821d admin admin] PATH_INFO: `/v3/roles` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:21:02.612520 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8465ade7-8490-49d7-ab16-068620f7821d admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:21:02.618922 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8465ade7-8490-49d7-ab16-068620f7821d admin admin] RBAC: Authorization granted {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:21:02.656582 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 682/1344] 10.222.0.27 () {58 vars in 1122 bytes} [Thu Oct 13 04:21:02 2022] GET /identity/v3/roles => generated 1420 bytes in 206 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 13 04:21:02.661703 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8465ade7-8490-49d7-ab16-068620f7821d admin admin] Authenticating user token {{(pid=81409) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:21:02.694735 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8465ade7-8490-49d7-ab16-068620f7821d admin admin] Validating token access rules against request {{(pid=81409) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:21:02.710870 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-df9bc30b-4537-46b3-915b-5c91e4d141aa admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8ba823f313144da3b683bacef8fd62e0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81409) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:21:02.711992 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df9bc30b-4537-46b3-915b-5c91e4d141aa admin admin] REQUEST_METHOD: `PUT` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:21:02.712376 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df9bc30b-4537-46b3-915b-5c91e4d141aa admin admin] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:21:02.712963 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df9bc30b-4537-46b3-915b-5c91e4d141aa admin admin] PATH_INFO: `/v3/projects/f63ab8de0cd04a4e9e46b4d9292faae6/users/450b79e9ba6e455aa07bcdd013b52d1f/roles/cc56c4ae13a7424ea771c15da4b153c1` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:21:02.738366 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-df9bc30b-4537-46b3-915b-5c91e4d141aa admin admin] RBAC: Authorizing `identity:create_grant(project_id=f63ab8de0cd04a4e9e46b4d9292faae6, user_id=450b79e9ba6e455aa07bcdd013b52d1f, role_id=cc56c4ae13a7424ea771c15da4b153c1)` {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:21:02.741543 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-df9bc30b-4537-46b3-915b-5c91e4d141aa admin admin] RBAC: Authorization granted {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:21:02.769016 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 683/1345] 10.222.0.27 () {60 vars in 1483 bytes} [Thu Oct 13 04:21:02 2022] PUT /identity/v3/projects/f63ab8de0cd04a4e9e46b4d9292faae6/users/450b79e9ba6e455aa07bcdd013b52d1f/roles/cc56c4ae13a7424ea771c15da4b153c1 => generated 0 bytes in 108 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 13 04:21:02.770140 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-df9bc30b-4537-46b3-915b-5c91e4d141aa admin admin] Authenticating user token {{(pid=81409) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:21:02.781172 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.fernet_utils [None req-df9bc30b-4537-46b3-915b-5c91e4d141aa admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81409) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:21:02.842888 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 663/1346] 10.222.0.27 () {58 vars in 958 bytes} [Thu Oct 13 04:21:02 2022] POST /identity/v3/auth/tokens => generated 3268 bytes in 334 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 13 04:21:02.854241 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6dab65a5-3a72-478d-8db8-3b3c19369816 None None] REQUEST_METHOD: `POST` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:21:02.854534 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6dab65a5-3a72-478d-8db8-3b3c19369816 None None] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:21:02.855760 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6dab65a5-3a72-478d-8db8-3b3c19369816 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:21:02.935351 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.auth.core [None req-6dab65a5-3a72-478d-8db8-3b3c19369816 None None] MFA Rules not processed for user `450b79e9ba6e455aa07bcdd013b52d1f`. Rule list: `[]` (Enabled: `True`). {{(pid=81408) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 13 04:21:02.937273 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.fernet_utils [None req-6dab65a5-3a72-478d-8db8-3b3c19369816 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81408) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:21:02.943809 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-df9bc30b-4537-46b3-915b-5c91e4d141aa admin admin] Validating token access rules against request {{(pid=81409) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:21:02.958457 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c5056909-4055-48c6-98f5-e05503193620 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8ba823f313144da3b683bacef8fd62e0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81409) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:21:02.959037 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5056909-4055-48c6-98f5-e05503193620 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:21:02.959315 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5056909-4055-48c6-98f5-e05503193620 admin admin] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:21:02.959587 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5056909-4055-48c6-98f5-e05503193620 admin admin] PATH_INFO: `/v3/users/874fc04c47164bafb9ed3ab86f248dca` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:21:02.967845 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c5056909-4055-48c6-98f5-e05503193620 admin admin] RBAC: Authorizing `identity:delete_user(user_id=874fc04c47164bafb9ed3ab86f248dca)` {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:21:02.972543 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c5056909-4055-48c6-98f5-e05503193620 admin admin] RBAC: Authorization granted {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:21:02.986506 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG dbcounter [-] [81408] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=81408) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:21:03.105345 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.notifications [None req-c5056909-4055-48c6-98f5-e05503193620 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '874fc04c47164bafb9ed3ab86f248dca'} {{(pid=81409) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 13 04:21:03.108660 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.notifications [None req-c5056909-4055-48c6-98f5-e05503193620 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '874fc04c47164bafb9ed3ab86f248dca'} {{(pid=81409) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 13 04:21:03.121094 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.notifications [None req-c5056909-4055-48c6-98f5-e05503193620 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '874fc04c47164bafb9ed3ab86f248dca'} {{(pid=81409) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 13 04:21:03.122289 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.notifications [None req-c5056909-4055-48c6-98f5-e05503193620 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '874fc04c47164bafb9ed3ab86f248dca'} {{(pid=81409) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 13 04:21:03.138254 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 664/1347] 10.222.0.27 () {58 vars in 958 bytes} [Thu Oct 13 04:21:02 2022] POST /identity/v3/auth/tokens => generated 3252 bytes in 295 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 13 04:21:03.156930 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6dab65a5-3a72-478d-8db8-3b3c19369816 None None] Authenticating user token {{(pid=81408) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:21:03.172365 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.notifications [None req-c5056909-4055-48c6-98f5-e05503193620 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '874fc04c47164bafb9ed3ab86f248dca'} {{(pid=81409) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 13 04:21:03.192719 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6dab65a5-3a72-478d-8db8-3b3c19369816 None None] Validating token access rules against request {{(pid=81408) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:21:03.205986 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4de23a8b-4a00-4ba8-bd58-8ecde5e8e13d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8ba823f313144da3b683bacef8fd62e0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81408) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:21:03.207443 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4de23a8b-4a00-4ba8-bd58-8ecde5e8e13d admin admin] REQUEST_METHOD: `POST` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:21:03.207733 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4de23a8b-4a00-4ba8-bd58-8ecde5e8e13d admin admin] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:21:03.208007 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4de23a8b-4a00-4ba8-bd58-8ecde5e8e13d admin admin] PATH_INFO: `/v3/projects` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:21:03.209722 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4de23a8b-4a00-4ba8-bd58-8ecde5e8e13d admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:21:03.211748 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4de23a8b-4a00-4ba8-bd58-8ecde5e8e13d admin admin] RBAC: Authorization granted {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:21:03.216946 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 684/1348] 10.222.0.27 () {58 vars in 1224 bytes} [Thu Oct 13 04:21:02 2022] DELETE /identity/v3/users/874fc04c47164bafb9ed3ab86f248dca => generated 0 bytes in 447 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 13 04:21:03.218266 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c5056909-4055-48c6-98f5-e05503193620 admin admin] Authenticating user token {{(pid=81409) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:21:03.219628 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.fernet_utils [None req-c5056909-4055-48c6-98f5-e05503193620 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81409) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:21:03.304260 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 665/1349] 10.222.0.27 () {60 vars in 1153 bytes} [Thu Oct 13 04:21:03 2022] POST /identity/v3/projects => generated 393 bytes in 165 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 13 04:21:03.307254 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f214e9d-d874-4bb4-96ec-6f0bc8db14f5 None None] REQUEST_METHOD: `POST` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:21:03.307539 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f214e9d-d874-4bb4-96ec-6f0bc8db14f5 None None] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:21:03.308890 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f214e9d-d874-4bb4-96ec-6f0bc8db14f5 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:21:03.339827 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG dbcounter [-] [81409] Writing DB stats keystone:SELECT=88,keystone:INSERT=2,keystone:DELETE=10 {{(pid=81409) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:21:03.355500 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.auth.core [None req-7f214e9d-d874-4bb4-96ec-6f0bc8db14f5 None None] MFA Rules not processed for user `0e2d979563d6421995451a0e86bf3022`. Rule list: `[]` (Enabled: `True`). {{(pid=81408) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 13 04:21:03.364471 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.fernet_utils [None req-7f214e9d-d874-4bb4-96ec-6f0bc8db14f5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81408) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:21:03.443231 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c5056909-4055-48c6-98f5-e05503193620 admin admin] Validating token access rules against request {{(pid=81409) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:21:03.487413 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 666/1350] 10.222.0.27 () {58 vars in 958 bytes} [Thu Oct 13 04:21:03 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 182 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 13 04:21:03.490298 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7f214e9d-d874-4bb4-96ec-6f0bc8db14f5 None None] Authenticating user token {{(pid=81408) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:21:03.492668 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.fernet_utils [None req-7f214e9d-d874-4bb4-96ec-6f0bc8db14f5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81408) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:21:03.517504 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7f214e9d-d874-4bb4-96ec-6f0bc8db14f5 None None] Validating token access rules against request {{(pid=81408) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:21:03.524525 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-99018146-b03f-447f-9ec8-e90404751ceb admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8ba823f313144da3b683bacef8fd62e0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81408) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:21:03.526984 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99018146-b03f-447f-9ec8-e90404751ceb admin admin] REQUEST_METHOD: `POST` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:21:03.527441 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99018146-b03f-447f-9ec8-e90404751ceb admin admin] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:21:03.527867 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99018146-b03f-447f-9ec8-e90404751ceb admin admin] PATH_INFO: `/v3/users` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:21:03.528701 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-99018146-b03f-447f-9ec8-e90404751ceb admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:21:03.532304 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-99018146-b03f-447f-9ec8-e90404751ceb admin admin] RBAC: Authorization granted {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:21:03.572609 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 667/1351] 10.222.0.27 () {60 vars in 1144 bytes} [Thu Oct 13 04:21:03 2022] POST /identity/v3/users => generated 303 bytes in 84 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 13 04:21:03.574470 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-99018146-b03f-447f-9ec8-e90404751ceb admin admin] Authenticating user token {{(pid=81408) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:21:03.576769 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-70e9dbfc-0fb6-4fc1-8708-d80098a20b63 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cb740ed7f7684b079e45f887be83f795', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dcec0431fb6849499d6aec35a234daa6', 'project_domain_id': 'default', 'roles': ['ResellerAdmin', 'admin', 'reader', 'service', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81409) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:21:03.577176 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.fernet_utils [None req-99018146-b03f-447f-9ec8-e90404751ceb admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81408) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:21:03.577898 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70e9dbfc-0fb6-4fc1-8708-d80098a20b63 service nova] REQUEST_METHOD: `GET` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:21:03.578175 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70e9dbfc-0fb6-4fc1-8708-d80098a20b63 service nova] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:21:03.578488 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70e9dbfc-0fb6-4fc1-8708-d80098a20b63 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:21:03.580393 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.fernet_utils [None req-70e9dbfc-0fb6-4fc1-8708-d80098a20b63 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81409) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:21:03.693187 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-99018146-b03f-447f-9ec8-e90404751ceb admin admin] Validating token access rules against request {{(pid=81408) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:21:03.706824 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-70e9dbfc-0fb6-4fc1-8708-d80098a20b63 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:21:03.708778 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-70e9dbfc-0fb6-4fc1-8708-d80098a20b63 service nova] RBAC: Authorization granted {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:21:03.714497 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-79ff077a-43bd-4585-b81f-095fb565c13a service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd8812c3974f84871bb2d761cf0016257', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dcec0431fb6849499d6aec35a234daa6', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81408) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:21:03.715285 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79ff077a-43bd-4585-b81f-095fb565c13a service neutron] REQUEST_METHOD: `GET` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:21:03.715568 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79ff077a-43bd-4585-b81f-095fb565c13a service neutron] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:21:03.715875 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79ff077a-43bd-4585-b81f-095fb565c13a service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:21:03.723964 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 685/1352] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Oct 13 04:21:03 2022] GET /identity/v3/auth/tokens => generated 3252 bytes in 506 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 13 04:21:03.724838 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-70e9dbfc-0fb6-4fc1-8708-d80098a20b63 service nova] Authenticating user token {{(pid=81409) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:21:03.739703 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-79ff077a-43bd-4585-b81f-095fb565c13a service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:21:03.741779 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-70e9dbfc-0fb6-4fc1-8708-d80098a20b63 service nova] Validating token access rules against request {{(pid=81409) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:21:03.747538 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-79ff077a-43bd-4585-b81f-095fb565c13a service neutron] RBAC: Authorization granted {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:21:03.770223 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-52aa0b01-754f-4e10-ad85-24a1d5d2a6d4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8ba823f313144da3b683bacef8fd62e0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81409) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:21:03.771365 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52aa0b01-754f-4e10-ad85-24a1d5d2a6d4 admin admin] REQUEST_METHOD: `GET` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:21:03.771835 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52aa0b01-754f-4e10-ad85-24a1d5d2a6d4 admin admin] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:21:03.772306 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52aa0b01-754f-4e10-ad85-24a1d5d2a6d4 admin admin] PATH_INFO: `/v3/roles` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:21:03.773162 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-52aa0b01-754f-4e10-ad85-24a1d5d2a6d4 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:21:03.775048 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-52aa0b01-754f-4e10-ad85-24a1d5d2a6d4 admin admin] RBAC: Authorization granted {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:21:03.778263 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 668/1353] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Oct 13 04:21:03 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 205 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 13 04:21:03.802464 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 686/1354] 10.222.0.27 () {58 vars in 1122 bytes} [Thu Oct 13 04:21:03 2022] GET /identity/v3/roles => generated 1420 bytes in 78 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 13 04:21:03.806264 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-79ff077a-43bd-4585-b81f-095fb565c13a service neutron] Authenticating user token {{(pid=81408) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:21:03.823028 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-79ff077a-43bd-4585-b81f-095fb565c13a service neutron] Validating token access rules against request {{(pid=81408) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:21:03.833439 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5ab11db7-09b9-4494-9843-e6c9323e933a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8ba823f313144da3b683bacef8fd62e0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81408) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:21:03.834128 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ab11db7-09b9-4494-9843-e6c9323e933a admin admin] REQUEST_METHOD: `PUT` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:21:03.834434 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ab11db7-09b9-4494-9843-e6c9323e933a admin admin] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:21:03.834709 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ab11db7-09b9-4494-9843-e6c9323e933a admin admin] PATH_INFO: `/v3/projects/49aa2543e5d44238a451f724b4219caa/users/6ab0dce872114681ac2b80714a6709e9/roles/ef55fbd94e1040be83695754075a7ed5` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:21:03.867595 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5ab11db7-09b9-4494-9843-e6c9323e933a admin admin] RBAC: Authorizing `identity:create_grant(project_id=49aa2543e5d44238a451f724b4219caa, user_id=6ab0dce872114681ac2b80714a6709e9, role_id=ef55fbd94e1040be83695754075a7ed5)` {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:21:03.870583 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5ab11db7-09b9-4494-9843-e6c9323e933a admin admin] RBAC: Authorization granted {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:21:03.892619 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG dbcounter [-] [81408] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=81408) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:21:03.915305 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-52aa0b01-754f-4e10-ad85-24a1d5d2a6d4 admin admin] Authenticating user token {{(pid=81409) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:21:03.918751 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 669/1355] 10.222.0.27 () {60 vars in 1483 bytes} [Thu Oct 13 04:21:03 2022] PUT /identity/v3/projects/49aa2543e5d44238a451f724b4219caa/users/6ab0dce872114681ac2b80714a6709e9/roles/ef55fbd94e1040be83695754075a7ed5 => generated 0 bytes in 106 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 13 04:21:03.934949 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-52aa0b01-754f-4e10-ad85-24a1d5d2a6d4 admin admin] Validating token access rules against request {{(pid=81409) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:21:04.086222 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5ab11db7-09b9-4494-9843-e6c9323e933a admin admin] Authenticating user token {{(pid=81408) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:21:04.088002 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.fernet_utils [None req-5ab11db7-09b9-4494-9843-e6c9323e933a admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81408) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:21:04.118103 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG dbcounter [-] [81409] Writing DB stats keystone:SELECT=100 {{(pid=81409) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:21:04.131420 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6f6a34a2-6d25-4d11-b0b8-f653f8bb7caa admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8ba823f313144da3b683bacef8fd62e0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81409) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:21:04.132053 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f6a34a2-6d25-4d11-b0b8-f653f8bb7caa admin admin] REQUEST_METHOD: `GET` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:21:04.132329 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f6a34a2-6d25-4d11-b0b8-f653f8bb7caa admin admin] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:21:04.132602 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f6a34a2-6d25-4d11-b0b8-f653f8bb7caa admin admin] PATH_INFO: `/v3/roles` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:21:04.133250 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6f6a34a2-6d25-4d11-b0b8-f653f8bb7caa admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:21:04.138585 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6f6a34a2-6d25-4d11-b0b8-f653f8bb7caa admin admin] RBAC: Authorization granted {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:21:04.168314 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 687/1356] 10.222.0.27 () {58 vars in 1122 bytes} [Thu Oct 13 04:21:03 2022] GET /identity/v3/roles => generated 1420 bytes in 253 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Oct 13 04:21:04.169294 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6f6a34a2-6d25-4d11-b0b8-f653f8bb7caa admin admin] Authenticating user token {{(pid=81409) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:21:04.202650 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6f6a34a2-6d25-4d11-b0b8-f653f8bb7caa admin admin] Validating token access rules against request {{(pid=81409) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:21:04.214183 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9d4a075e-722e-426c-8e4c-c18cc3d6c672 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8ba823f313144da3b683bacef8fd62e0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81409) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:21:04.215760 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d4a075e-722e-426c-8e4c-c18cc3d6c672 admin admin] REQUEST_METHOD: `PUT` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:21:04.217859 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d4a075e-722e-426c-8e4c-c18cc3d6c672 admin admin] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:21:04.218586 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d4a075e-722e-426c-8e4c-c18cc3d6c672 admin admin] PATH_INFO: `/v3/projects/49aa2543e5d44238a451f724b4219caa/users/6ab0dce872114681ac2b80714a6709e9/roles/ef55fbd94e1040be83695754075a7ed5` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:21:04.224589 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9d4a075e-722e-426c-8e4c-c18cc3d6c672 admin admin] RBAC: Authorizing `identity:create_grant(project_id=49aa2543e5d44238a451f724b4219caa, user_id=6ab0dce872114681ac2b80714a6709e9, role_id=ef55fbd94e1040be83695754075a7ed5)` {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:21:04.230563 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9d4a075e-722e-426c-8e4c-c18cc3d6c672 admin admin] RBAC: Authorization granted {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:21:04.260451 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5ab11db7-09b9-4494-9843-e6c9323e933a admin admin] Validating token access rules against request {{(pid=81408) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:21:04.282746 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 688/1357] 10.222.0.27 () {60 vars in 1483 bytes} [Thu Oct 13 04:21:04 2022] PUT /identity/v3/projects/49aa2543e5d44238a451f724b4219caa/users/6ab0dce872114681ac2b80714a6709e9/roles/ef55fbd94e1040be83695754075a7ed5 => generated 0 bytes in 100 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 13 04:21:04.284628 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4c58fde-7e78-45ee-9e1f-18579b9b3c62 None None] REQUEST_METHOD: `POST` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:21:04.284914 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4c58fde-7e78-45ee-9e1f-18579b9b3c62 None None] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:21:04.285186 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4c58fde-7e78-45ee-9e1f-18579b9b3c62 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:21:04.344871 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.auth.core [None req-d4c58fde-7e78-45ee-9e1f-18579b9b3c62 None None] MFA Rules not processed for user `6ab0dce872114681ac2b80714a6709e9`. Rule list: `[]` (Enabled: `True`). {{(pid=81409) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 13 04:21:04.345793 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.fernet_utils [None req-d4c58fde-7e78-45ee-9e1f-18579b9b3c62 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81409) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:21:04.423258 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-77f3b11f-a70c-4db8-a49d-2f419835b2a9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8ba823f313144da3b683bacef8fd62e0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81408) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:21:04.423907 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77f3b11f-a70c-4db8-a49d-2f419835b2a9 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:21:04.424177 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77f3b11f-a70c-4db8-a49d-2f419835b2a9 admin admin] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:21:04.424455 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77f3b11f-a70c-4db8-a49d-2f419835b2a9 admin admin] PATH_INFO: `/v3/users/450b79e9ba6e455aa07bcdd013b52d1f` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:21:04.430973 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-77f3b11f-a70c-4db8-a49d-2f419835b2a9 admin admin] RBAC: Authorizing `identity:delete_user(user_id=450b79e9ba6e455aa07bcdd013b52d1f)` {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:21:04.434334 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-77f3b11f-a70c-4db8-a49d-2f419835b2a9 admin admin] RBAC: Authorization granted {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:21:04.472478 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 689/1358] 10.222.0.27 () {58 vars in 958 bytes} [Thu Oct 13 04:21:04 2022] POST /identity/v3/auth/tokens => generated 3337 bytes in 189 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 13 04:21:04.479918 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d4c58fde-7e78-45ee-9e1f-18579b9b3c62 None None] Authenticating user token {{(pid=81409) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:21:04.482081 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.fernet_utils [None req-d4c58fde-7e78-45ee-9e1f-18579b9b3c62 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81409) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:21:04.575352 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.notifications [None req-77f3b11f-a70c-4db8-a49d-2f419835b2a9 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '450b79e9ba6e455aa07bcdd013b52d1f'} {{(pid=81408) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 13 04:21:04.597225 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.notifications [None req-77f3b11f-a70c-4db8-a49d-2f419835b2a9 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '450b79e9ba6e455aa07bcdd013b52d1f'} {{(pid=81408) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 13 04:21:04.598102 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.notifications [None req-77f3b11f-a70c-4db8-a49d-2f419835b2a9 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '450b79e9ba6e455aa07bcdd013b52d1f'} {{(pid=81408) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 13 04:21:04.598835 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.notifications [None req-77f3b11f-a70c-4db8-a49d-2f419835b2a9 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '450b79e9ba6e455aa07bcdd013b52d1f'} {{(pid=81408) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 13 04:21:04.612477 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d4c58fde-7e78-45ee-9e1f-18579b9b3c62 None None] Validating token access rules against request {{(pid=81409) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:21:04.618393 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b83834d4-e091-49f7-bc69-7d3f40555c82 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6141b8a6c3d54e179d6879bb019f4d12', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dcec0431fb6849499d6aec35a234daa6', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81409) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:21:04.619005 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.notifications [None req-77f3b11f-a70c-4db8-a49d-2f419835b2a9 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '450b79e9ba6e455aa07bcdd013b52d1f'} {{(pid=81408) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 13 04:21:04.619472 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b83834d4-e091-49f7-bc69-7d3f40555c82 service cinder] REQUEST_METHOD: `GET` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:21:04.620437 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b83834d4-e091-49f7-bc69-7d3f40555c82 service cinder] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:21:04.621230 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b83834d4-e091-49f7-bc69-7d3f40555c82 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:21:04.624044 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.fernet_utils [None req-b83834d4-e091-49f7-bc69-7d3f40555c82 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81409) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:21:04.644463 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b83834d4-e091-49f7-bc69-7d3f40555c82 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:21:04.646449 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b83834d4-e091-49f7-bc69-7d3f40555c82 service cinder] RBAC: Authorization granted {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:21:04.650755 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 670/1359] 10.222.0.27 () {58 vars in 1224 bytes} [Thu Oct 13 04:21:04 2022] DELETE /identity/v3/users/450b79e9ba6e455aa07bcdd013b52d1f => generated 0 bytes in 567 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 13 04:21:04.656150 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5cbbac41-01e1-490b-923a-dfc729740827 None None] REQUEST_METHOD: `POST` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:21:04.656469 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5cbbac41-01e1-490b-923a-dfc729740827 None None] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:21:04.656747 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5cbbac41-01e1-490b-923a-dfc729740827 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:21:04.668386 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 690/1360] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Oct 13 04:21:04 2022] GET /identity/v3/auth/tokens => generated 3337 bytes in 189 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Oct 13 04:21:04.695698 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.auth.core [None req-5cbbac41-01e1-490b-923a-dfc729740827 None None] MFA Rules not processed for user `0e2d979563d6421995451a0e86bf3022`. Rule list: `[]` (Enabled: `True`). {{(pid=81408) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 13 04:21:04.699000 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.fernet_utils [None req-5cbbac41-01e1-490b-923a-dfc729740827 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81408) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:21:04.740953 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG dbcounter [-] [81408] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81408) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:21:04.746513 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b83834d4-e091-49f7-bc69-7d3f40555c82 service cinder] Authenticating user token {{(pid=81409) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:21:04.748085 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.fernet_utils [None req-b83834d4-e091-49f7-bc69-7d3f40555c82 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81409) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:21:04.771787 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b83834d4-e091-49f7-bc69-7d3f40555c82 service cinder] Validating token access rules against request {{(pid=81409) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:21:04.856676 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 671/1361] 10.222.0.27 () {58 vars in 958 bytes} [Thu Oct 13 04:21:04 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 202 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 13 04:21:04.874242 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5cbbac41-01e1-490b-923a-dfc729740827 None None] Authenticating user token {{(pid=81408) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:21:04.876130 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.fernet_utils [None req-5cbbac41-01e1-490b-923a-dfc729740827 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81408) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:21:04.919520 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-33e2d864-89b9-4036-8fac-974ec8d040c5 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6141b8a6c3d54e179d6879bb019f4d12', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dcec0431fb6849499d6aec35a234daa6', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81409) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:21:04.922839 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33e2d864-89b9-4036-8fac-974ec8d040c5 service cinder] REQUEST_METHOD: `GET` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:21:04.925090 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33e2d864-89b9-4036-8fac-974ec8d040c5 service cinder] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:21:04.926595 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33e2d864-89b9-4036-8fac-974ec8d040c5 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:21:04.932150 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.fernet_utils [None req-33e2d864-89b9-4036-8fac-974ec8d040c5 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81409) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:21:05.008787 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG dbcounter [-] [81409] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81409) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:21:05.031414 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5cbbac41-01e1-490b-923a-dfc729740827 None None] Validating token access rules against request {{(pid=81408) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:21:05.046278 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-03a0cdd3-53aa-49a8-9f2b-0b6491468848 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd8812c3974f84871bb2d761cf0016257', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dcec0431fb6849499d6aec35a234daa6', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81408) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:21:05.050977 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03a0cdd3-53aa-49a8-9f2b-0b6491468848 service neutron] REQUEST_METHOD: `GET` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:21:05.051265 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03a0cdd3-53aa-49a8-9f2b-0b6491468848 service neutron] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:21:05.053072 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03a0cdd3-53aa-49a8-9f2b-0b6491468848 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:21:05.078060 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-03a0cdd3-53aa-49a8-9f2b-0b6491468848 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:21:05.079910 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-03a0cdd3-53aa-49a8-9f2b-0b6491468848 service neutron] RBAC: Authorization granted {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:21:05.096208 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 672/1362] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Oct 13 04:21:04 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 222 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 13 04:21:05.099706 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-33e2d864-89b9-4036-8fac-974ec8d040c5 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:21:05.101194 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-33e2d864-89b9-4036-8fac-974ec8d040c5 service cinder] RBAC: Authorization granted {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:21:05.125434 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 691/1363] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Oct 13 04:21:04 2022] GET /identity/v3/auth/tokens => generated 3268 bytes in 379 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 13 04:21:05.473115 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-03a0cdd3-53aa-49a8-9f2b-0b6491468848 service neutron] Authenticating user token {{(pid=81408) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:21:05.474899 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.fernet_utils [None req-03a0cdd3-53aa-49a8-9f2b-0b6491468848 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81408) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:21:05.507537 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-03a0cdd3-53aa-49a8-9f2b-0b6491468848 service neutron] Validating token access rules against request {{(pid=81408) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:21:05.527539 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8a61f2a2-75fb-4771-9644-cd25979afe6b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8ba823f313144da3b683bacef8fd62e0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81408) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:21:05.528704 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a61f2a2-75fb-4771-9644-cd25979afe6b admin admin] REQUEST_METHOD: `DELETE` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:21:05.530247 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a61f2a2-75fb-4771-9644-cd25979afe6b admin admin] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:21:05.530515 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a61f2a2-75fb-4771-9644-cd25979afe6b admin admin] PATH_INFO: `/v3/projects/f9a0de5ed456439092b335e5239a2df8` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:21:05.531784 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8a61f2a2-75fb-4771-9644-cd25979afe6b admin admin] RBAC: Authorizing `identity:delete_project(project_id=f9a0de5ed456439092b335e5239a2df8)` {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:21:05.534053 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8a61f2a2-75fb-4771-9644-cd25979afe6b admin admin] RBAC: Authorization granted {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:21:05.692760 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.notifications [None req-8a61f2a2-75fb-4771-9644-cd25979afe6b admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'f9a0de5ed456439092b335e5239a2df8', 'request_id': 'req-8a61f2a2-75fb-4771-9644-cd25979afe6b'} {{(pid=81408) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 13 04:21:05.714229 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.notifications [None req-8a61f2a2-75fb-4771-9644-cd25979afe6b admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'f9a0de5ed456439092b335e5239a2df8', 'request_id': 'req-8a61f2a2-75fb-4771-9644-cd25979afe6b'} {{(pid=81408) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 13 04:21:05.761317 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.notifications [None req-8a61f2a2-75fb-4771-9644-cd25979afe6b admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'f9a0de5ed456439092b335e5239a2df8', 'request_id': 'req-8a61f2a2-75fb-4771-9644-cd25979afe6b'} {{(pid=81408) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 13 04:21:05.789006 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.notifications [None req-8a61f2a2-75fb-4771-9644-cd25979afe6b admin admin] The token cache is being invalidate because project f9a0de5ed456439092b335e5239a2df8 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81408) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 13 04:21:05.789328 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.notifications [None req-8a61f2a2-75fb-4771-9644-cd25979afe6b admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81408) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 13 04:21:05.790469 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.notifications [None req-8a61f2a2-75fb-4771-9644-cd25979afe6b admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81408) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 13 04:21:05.797256 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 673/1364] 10.222.0.27 () {58 vars in 1233 bytes} [Thu Oct 13 04:21:05 2022] DELETE /identity/v3/projects/f9a0de5ed456439092b335e5239a2df8 => generated 0 bytes in 321 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 13 04:21:05.798886 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6eac91e-ffd8-4a72-936f-8600d2402900 None None] REQUEST_METHOD: `POST` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:21:05.799159 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6eac91e-ffd8-4a72-936f-8600d2402900 None None] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:21:05.799435 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6eac91e-ffd8-4a72-936f-8600d2402900 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:21:05.847534 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.auth.core [None req-b6eac91e-ffd8-4a72-936f-8600d2402900 None None] MFA Rules not processed for user `0e2d979563d6421995451a0e86bf3022`. Rule list: `[]` (Enabled: `True`). {{(pid=81409) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 13 04:21:05.848748 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.fernet_utils [None req-b6eac91e-ffd8-4a72-936f-8600d2402900 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81409) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:21:06.001989 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 692/1365] 10.222.0.27 () {58 vars in 958 bytes} [Thu Oct 13 04:21:05 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 204 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 13 04:21:06.008584 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39133f55-3d22-40eb-b9fd-fffe315ddb65 None None] REQUEST_METHOD: `POST` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:21:06.009234 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39133f55-3d22-40eb-b9fd-fffe315ddb65 None None] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:21:06.009670 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39133f55-3d22-40eb-b9fd-fffe315ddb65 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:21:06.047816 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.auth.core [None req-39133f55-3d22-40eb-b9fd-fffe315ddb65 None None] MFA Rules not processed for user `0e2d979563d6421995451a0e86bf3022`. Rule list: `[]` (Enabled: `True`). {{(pid=81408) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 13 04:21:06.051516 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.fernet_utils [None req-39133f55-3d22-40eb-b9fd-fffe315ddb65 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81408) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:21:06.070847 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 674/1366] 10.222.0.27 () {58 vars in 958 bytes} [Thu Oct 13 04:21:06 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 63 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 13 04:21:06.075755 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b6eac91e-ffd8-4a72-936f-8600d2402900 None None] Authenticating user token {{(pid=81409) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:21:06.086114 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b6eac91e-ffd8-4a72-936f-8600d2402900 None None] Validating token access rules against request {{(pid=81409) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:21:06.107027 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dc2d9930-93ad-4e97-997c-8b95a6ab34a1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8ba823f313144da3b683bacef8fd62e0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81409) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:21:06.108921 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc2d9930-93ad-4e97-997c-8b95a6ab34a1 admin admin] REQUEST_METHOD: `GET` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:21:06.110066 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc2d9930-93ad-4e97-997c-8b95a6ab34a1 admin admin] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:21:06.110066 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc2d9930-93ad-4e97-997c-8b95a6ab34a1 admin admin] PATH_INFO: `/v3/domains` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:21:06.112695 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dc2d9930-93ad-4e97-997c-8b95a6ab34a1 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81409) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 13 04:21:06.113078 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dc2d9930-93ad-4e97-997c-8b95a6ab34a1 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:21:06.114831 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dc2d9930-93ad-4e97-997c-8b95a6ab34a1 admin admin] RBAC: Authorization granted {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:21:06.151663 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 693/1367] 10.222.0.27 () {58 vars in 1153 bytes} [Thu Oct 13 04:21:06 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 76 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 13 04:21:06.156592 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-39133f55-3d22-40eb-b9fd-fffe315ddb65 None None] Authenticating user token {{(pid=81408) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:21:06.168925 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-39133f55-3d22-40eb-b9fd-fffe315ddb65 None None] Validating token access rules against request {{(pid=81408) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:21:06.182199 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e5b6734-3f13-46b9-bc06-1199d5ee8847 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8ba823f313144da3b683bacef8fd62e0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81408) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:21:06.183081 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e5b6734-3f13-46b9-bc06-1199d5ee8847 admin admin] REQUEST_METHOD: `POST` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:21:06.183451 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e5b6734-3f13-46b9-bc06-1199d5ee8847 admin admin] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:21:06.184295 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e5b6734-3f13-46b9-bc06-1199d5ee8847 admin admin] PATH_INFO: `/v3/projects` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:21:06.185050 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1e5b6734-3f13-46b9-bc06-1199d5ee8847 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:21:06.190170 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1e5b6734-3f13-46b9-bc06-1199d5ee8847 admin admin] RBAC: Authorization granted {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:21:06.259236 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 675/1368] 10.222.0.27 () {60 vars in 1153 bytes} [Thu Oct 13 04:21:06 2022] POST /identity/v3/projects => generated 367 bytes in 103 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 13 04:21:06.263615 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dc2d9930-93ad-4e97-997c-8b95a6ab34a1 admin admin] Authenticating user token {{(pid=81409) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:21:06.282160 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dc2d9930-93ad-4e97-997c-8b95a6ab34a1 admin admin] Validating token access rules against request {{(pid=81409) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:21:06.419384 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-25cc25dc-c3ec-4ed6-8d59-a94b6287b13c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8ba823f313144da3b683bacef8fd62e0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81409) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:21:06.420158 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25cc25dc-c3ec-4ed6-8d59-a94b6287b13c admin admin] REQUEST_METHOD: `POST` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:21:06.420674 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25cc25dc-c3ec-4ed6-8d59-a94b6287b13c admin admin] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:21:06.421179 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25cc25dc-c3ec-4ed6-8d59-a94b6287b13c admin admin] PATH_INFO: `/v3/users` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:21:06.421935 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-25cc25dc-c3ec-4ed6-8d59-a94b6287b13c admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:21:06.424002 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-25cc25dc-c3ec-4ed6-8d59-a94b6287b13c admin admin] RBAC: Authorization granted {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:21:06.472114 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 694/1369] 10.222.0.27 () {60 vars in 1144 bytes} [Thu Oct 13 04:21:06 2022] POST /identity/v3/users => generated 284 bytes in 206 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 13 04:21:06.475050 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e5b6734-3f13-46b9-bc06-1199d5ee8847 admin admin] Authenticating user token {{(pid=81408) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:21:06.500621 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e5b6734-3f13-46b9-bc06-1199d5ee8847 admin admin] Validating token access rules against request {{(pid=81408) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:21:06.506131 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-327164fc-5f39-4b62-a929-ad51304218fb admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8ba823f313144da3b683bacef8fd62e0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81408) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:21:06.507231 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-327164fc-5f39-4b62-a929-ad51304218fb admin admin] REQUEST_METHOD: `GET` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:21:06.507690 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-327164fc-5f39-4b62-a929-ad51304218fb admin admin] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:21:06.508261 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-327164fc-5f39-4b62-a929-ad51304218fb admin admin] PATH_INFO: `/v3/roles` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:21:06.509361 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-327164fc-5f39-4b62-a929-ad51304218fb admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:21:06.512268 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-327164fc-5f39-4b62-a929-ad51304218fb admin admin] RBAC: Authorization granted {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:21:06.539566 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 676/1370] 10.222.0.27 () {58 vars in 1122 bytes} [Thu Oct 13 04:21:06 2022] GET /identity/v3/roles => generated 1420 bytes in 63 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 13 04:21:06.541122 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-25cc25dc-c3ec-4ed6-8d59-a94b6287b13c admin admin] Authenticating user token {{(pid=81409) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:21:06.562723 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-25cc25dc-c3ec-4ed6-8d59-a94b6287b13c admin admin] Validating token access rules against request {{(pid=81409) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:21:06.571548 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bdeb4a1c-bb84-492b-84d9-ca1f7282db3c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8ba823f313144da3b683bacef8fd62e0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81409) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:21:06.572604 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bdeb4a1c-bb84-492b-84d9-ca1f7282db3c admin admin] REQUEST_METHOD: `GET` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:21:06.573109 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bdeb4a1c-bb84-492b-84d9-ca1f7282db3c admin admin] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:21:06.573592 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bdeb4a1c-bb84-492b-84d9-ca1f7282db3c admin admin] PATH_INFO: `/v3/roles` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:21:06.574475 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bdeb4a1c-bb84-492b-84d9-ca1f7282db3c admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:21:06.576343 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bdeb4a1c-bb84-492b-84d9-ca1f7282db3c admin admin] RBAC: Authorization granted {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:21:06.609587 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 695/1371] 10.222.0.27 () {58 vars in 1122 bytes} [Thu Oct 13 04:21:06 2022] GET /identity/v3/roles => generated 1420 bytes in 69 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Oct 13 04:21:06.614500 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-327164fc-5f39-4b62-a929-ad51304218fb admin admin] Authenticating user token {{(pid=81408) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:21:06.628100 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-327164fc-5f39-4b62-a929-ad51304218fb admin admin] Validating token access rules against request {{(pid=81408) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:21:06.635442 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-17db822b-96a5-4953-8bc5-75d0206d2b5f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8ba823f313144da3b683bacef8fd62e0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81408) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:21:06.636431 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17db822b-96a5-4953-8bc5-75d0206d2b5f admin admin] REQUEST_METHOD: `PUT` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:21:06.636851 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17db822b-96a5-4953-8bc5-75d0206d2b5f admin admin] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:21:06.637325 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17db822b-96a5-4953-8bc5-75d0206d2b5f admin admin] PATH_INFO: `/v3/projects/6592493745484100afa26c249568c269/users/51b364bcf10d45ec8e1871fa09f8de1f/roles/cc56c4ae13a7424ea771c15da4b153c1` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:21:06.648510 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG dbcounter [-] [81408] Writing DB stats keystone:SELECT=91,keystone:DELETE=7,keystone:INSERT=2 {{(pid=81408) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:21:06.658720 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-17db822b-96a5-4953-8bc5-75d0206d2b5f admin admin] RBAC: Authorizing `identity:create_grant(project_id=6592493745484100afa26c249568c269, user_id=51b364bcf10d45ec8e1871fa09f8de1f, role_id=cc56c4ae13a7424ea771c15da4b153c1)` {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:21:06.665713 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-17db822b-96a5-4953-8bc5-75d0206d2b5f admin admin] RBAC: Authorization granted {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:21:06.687013 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 677/1372] 10.222.0.27 () {60 vars in 1483 bytes} [Thu Oct 13 04:21:06 2022] PUT /identity/v3/projects/6592493745484100afa26c249568c269/users/51b364bcf10d45ec8e1871fa09f8de1f/roles/cc56c4ae13a7424ea771c15da4b153c1 => generated 0 bytes in 73 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 13 04:21:06.704969 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-926b932a-64be-406d-800a-ac65beecf681 None None] REQUEST_METHOD: `POST` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:21:06.705324 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-926b932a-64be-406d-800a-ac65beecf681 None None] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:21:06.705680 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-926b932a-64be-406d-800a-ac65beecf681 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:21:06.783111 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.auth.core [None req-926b932a-64be-406d-800a-ac65beecf681 None None] MFA Rules not processed for user `51b364bcf10d45ec8e1871fa09f8de1f`. Rule list: `[]` (Enabled: `True`). {{(pid=81409) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 13 04:21:06.785295 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.fernet_utils [None req-926b932a-64be-406d-800a-ac65beecf681 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81409) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:21:06.813046 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG dbcounter [-] [81409] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81409) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:21:06.918641 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 696/1373] 10.222.0.27 () {58 vars in 958 bytes} [Thu Oct 13 04:21:06 2022] POST /identity/v3/auth/tokens => generated 3244 bytes in 215 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 13 04:21:06.922266 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-17db822b-96a5-4953-8bc5-75d0206d2b5f admin admin] Authenticating user token {{(pid=81408) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:21:06.924628 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.fernet_utils [None req-17db822b-96a5-4953-8bc5-75d0206d2b5f admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81408) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:21:07.065677 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-17db822b-96a5-4953-8bc5-75d0206d2b5f admin admin] Validating token access rules against request {{(pid=81408) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:21:07.079251 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-94840160-4fb4-4085-b6c5-41df27683798 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6141b8a6c3d54e179d6879bb019f4d12', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dcec0431fb6849499d6aec35a234daa6', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81408) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:21:07.079890 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94840160-4fb4-4085-b6c5-41df27683798 service cinder] REQUEST_METHOD: `GET` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:21:07.080176 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94840160-4fb4-4085-b6c5-41df27683798 service cinder] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:21:07.080453 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94840160-4fb4-4085-b6c5-41df27683798 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:21:07.110695 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-94840160-4fb4-4085-b6c5-41df27683798 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:21:07.115405 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-94840160-4fb4-4085-b6c5-41df27683798 service cinder] RBAC: Authorization granted {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:21:07.148955 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 678/1374] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Oct 13 04:21:06 2022] GET /identity/v3/auth/tokens => generated 3244 bytes in 227 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 13 04:21:07.788489 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bca5eb3e-3c90-4432-a433-c0e3a257ac05 None None] REQUEST_METHOD: `POST` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:21:07.788988 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bca5eb3e-3c90-4432-a433-c0e3a257ac05 None None] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:21:07.789204 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bca5eb3e-3c90-4432-a433-c0e3a257ac05 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:21:07.845690 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.auth.core [None req-bca5eb3e-3c90-4432-a433-c0e3a257ac05 None None] MFA Rules not processed for user `999c89e1bc7a426886fc802e28d38596`. Rule list: `[]` (Enabled: `True`). {{(pid=81409) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 13 04:21:07.848178 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.fernet_utils [None req-bca5eb3e-3c90-4432-a433-c0e3a257ac05 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81409) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:21:08.038606 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 697/1375] 10.222.0.27 () {58 vars in 958 bytes} [Thu Oct 13 04:21:07 2022] POST /identity/v3/auth/tokens => generated 3260 bytes in 252 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 13 04:21:08.277294 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf802a97-9544-45d9-af8f-befb4a83978e None None] REQUEST_METHOD: `POST` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:21:08.277820 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf802a97-9544-45d9-af8f-befb4a83978e None None] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:21:08.277820 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf802a97-9544-45d9-af8f-befb4a83978e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:21:08.372173 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.auth.core [None req-bf802a97-9544-45d9-af8f-befb4a83978e None None] MFA Rules not processed for user `70ff2e3f5a0f4e0da74af8d0ea6b1506`. Rule list: `[]` (Enabled: `True`). {{(pid=81408) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 13 04:21:08.377050 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.fernet_utils [None req-bf802a97-9544-45d9-af8f-befb4a83978e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81408) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:21:08.378324 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bca5eb3e-3c90-4432-a433-c0e3a257ac05 None None] Authenticating user token {{(pid=81409) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:21:08.379822 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.fernet_utils [None req-bca5eb3e-3c90-4432-a433-c0e3a257ac05 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81409) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:21:08.546857 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 679/1376] 10.222.0.27 () {58 vars in 958 bytes} [Thu Oct 13 04:21:08 2022] POST /identity/v3/auth/tokens => generated 3256 bytes in 270 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 13 04:21:08.564836 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43c2f2ca-7b69-4290-957b-6ec3b95edaf2 None None] REQUEST_METHOD: `POST` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:21:08.565233 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43c2f2ca-7b69-4290-957b-6ec3b95edaf2 None None] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:21:08.565514 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43c2f2ca-7b69-4290-957b-6ec3b95edaf2 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:21:08.567699 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bca5eb3e-3c90-4432-a433-c0e3a257ac05 None None] Validating token access rules against request {{(pid=81409) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:21:08.572709 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0a4a561b-184e-445b-b4bb-0452c0d2a2ba admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8ba823f313144da3b683bacef8fd62e0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81409) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:21:08.573380 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a4a561b-184e-445b-b4bb-0452c0d2a2ba admin admin] REQUEST_METHOD: `DELETE` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:21:08.573620 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a4a561b-184e-445b-b4bb-0452c0d2a2ba admin admin] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:21:08.573892 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a4a561b-184e-445b-b4bb-0452c0d2a2ba admin admin] PATH_INFO: `/v3/projects/f63ab8de0cd04a4e9e46b4d9292faae6` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:21:08.578168 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0a4a561b-184e-445b-b4bb-0452c0d2a2ba admin admin] RBAC: Authorizing `identity:delete_project(project_id=f63ab8de0cd04a4e9e46b4d9292faae6)` {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:21:08.580123 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0a4a561b-184e-445b-b4bb-0452c0d2a2ba admin admin] RBAC: Authorization granted {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:21:08.601754 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.auth.core [None req-43c2f2ca-7b69-4290-957b-6ec3b95edaf2 None None] MFA Rules not processed for user `70ff2e3f5a0f4e0da74af8d0ea6b1506`. Rule list: `[]` (Enabled: `True`). {{(pid=81408) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 13 04:21:08.603206 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.fernet_utils [None req-43c2f2ca-7b69-4290-957b-6ec3b95edaf2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81408) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:21:08.613118 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 680/1377] 10.222.0.27 () {58 vars in 958 bytes} [Thu Oct 13 04:21:08 2022] POST /identity/v3/auth/tokens => generated 3256 bytes in 54 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 13 04:21:08.631804 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-43c2f2ca-7b69-4290-957b-6ec3b95edaf2 None None] Authenticating user token {{(pid=81408) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:21:08.634519 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.fernet_utils [None req-43c2f2ca-7b69-4290-957b-6ec3b95edaf2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81408) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:21:08.707842 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.notifications [None req-0a4a561b-184e-445b-b4bb-0452c0d2a2ba admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'f63ab8de0cd04a4e9e46b4d9292faae6', 'request_id': 'req-0a4a561b-184e-445b-b4bb-0452c0d2a2ba'} {{(pid=81409) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 13 04:21:08.732192 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.notifications [None req-0a4a561b-184e-445b-b4bb-0452c0d2a2ba admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'f63ab8de0cd04a4e9e46b4d9292faae6', 'request_id': 'req-0a4a561b-184e-445b-b4bb-0452c0d2a2ba'} {{(pid=81409) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 13 04:21:08.740826 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.notifications [None req-0a4a561b-184e-445b-b4bb-0452c0d2a2ba admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'f63ab8de0cd04a4e9e46b4d9292faae6', 'request_id': 'req-0a4a561b-184e-445b-b4bb-0452c0d2a2ba'} {{(pid=81409) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 13 04:21:08.750702 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG dbcounter [-] [81409] Writing DB stats keystone:SELECT=93,keystone:DELETE=6,keystone:INSERT=1 {{(pid=81409) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:21:08.767639 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.notifications [None req-0a4a561b-184e-445b-b4bb-0452c0d2a2ba admin admin] The token cache is being invalidate because project f63ab8de0cd04a4e9e46b4d9292faae6 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81409) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 13 04:21:08.769716 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.notifications [None req-0a4a561b-184e-445b-b4bb-0452c0d2a2ba admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81409) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 13 04:21:08.770419 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.notifications [None req-0a4a561b-184e-445b-b4bb-0452c0d2a2ba admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81409) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 13 04:21:08.780758 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 698/1378] 10.222.0.27 () {58 vars in 1233 bytes} [Thu Oct 13 04:21:08 2022] DELETE /identity/v3/projects/f63ab8de0cd04a4e9e46b4d9292faae6 => generated 0 bytes in 399 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 13 04:21:08.782827 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4781dcca-7da8-4606-bf32-8fb47e7388e0 None None] REQUEST_METHOD: `POST` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:21:08.783115 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4781dcca-7da8-4606-bf32-8fb47e7388e0 None None] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:21:08.783390 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4781dcca-7da8-4606-bf32-8fb47e7388e0 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:21:08.800383 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-43c2f2ca-7b69-4290-957b-6ec3b95edaf2 None None] Validating token access rules against request {{(pid=81408) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:21:08.816052 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG dbcounter [-] [81408] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81408) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:21:08.831248 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.auth.core [None req-4781dcca-7da8-4606-bf32-8fb47e7388e0 None None] MFA Rules not processed for user `0e2d979563d6421995451a0e86bf3022`. Rule list: `[]` (Enabled: `True`). {{(pid=81409) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 13 04:21:08.832482 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.fernet_utils [None req-4781dcca-7da8-4606-bf32-8fb47e7388e0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81409) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:21:08.952707 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 699/1379] 10.222.0.27 () {58 vars in 958 bytes} [Thu Oct 13 04:21:08 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 171 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 13 04:21:08.955384 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ddf344c-5a0e-4e51-b0b9-233ca09d9ea6 None None] REQUEST_METHOD: `POST` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:21:08.959782 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ddf344c-5a0e-4e51-b0b9-233ca09d9ea6 None None] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:21:08.960274 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ddf344c-5a0e-4e51-b0b9-233ca09d9ea6 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:21:08.978250 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d613efe4-c058-49b4-8d81-8d95d218e05e service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cb740ed7f7684b079e45f887be83f795', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dcec0431fb6849499d6aec35a234daa6', 'project_domain_id': 'default', 'roles': ['ResellerAdmin', 'admin', 'reader', 'service', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81408) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:21:08.978942 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d613efe4-c058-49b4-8d81-8d95d218e05e service nova] REQUEST_METHOD: `GET` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:21:08.979333 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d613efe4-c058-49b4-8d81-8d95d218e05e service nova] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:21:08.979744 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d613efe4-c058-49b4-8d81-8d95d218e05e service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:21:08.982819 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.fernet_utils [None req-d613efe4-c058-49b4-8d81-8d95d218e05e service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81408) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:21:09.026258 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.auth.core [None req-6ddf344c-5a0e-4e51-b0b9-233ca09d9ea6 None None] MFA Rules not processed for user `da9c75191a3b4ef49438aae9e35fcf13`. Rule list: `[]` (Enabled: `True`). {{(pid=81409) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 13 04:21:09.028318 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.fernet_utils [None req-6ddf344c-5a0e-4e51-b0b9-233ca09d9ea6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81409) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:21:09.096324 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d613efe4-c058-49b4-8d81-8d95d218e05e service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:21:09.099113 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d613efe4-c058-49b4-8d81-8d95d218e05e service nova] RBAC: Authorization granted {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:21:09.132374 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 681/1380] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Oct 13 04:21:08 2022] GET /identity/v3/auth/tokens => generated 3256 bytes in 507 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 13 04:21:09.139338 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7062adc-6b50-4ede-94d7-d053cd6ccc5d None None] REQUEST_METHOD: `POST` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:21:09.140627 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7062adc-6b50-4ede-94d7-d053cd6ccc5d None None] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:21:09.140902 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7062adc-6b50-4ede-94d7-d053cd6ccc5d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:21:09.195850 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 700/1381] 10.222.0.27 () {58 vars in 958 bytes} [Thu Oct 13 04:21:08 2022] POST /identity/v3/auth/tokens => generated 3270 bytes in 241 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 13 04:21:09.197325 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6541e8f5-a153-419f-9821-83579f859ad8 None None] REQUEST_METHOD: `POST` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:21:09.197620 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6541e8f5-a153-419f-9821-83579f859ad8 None None] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:21:09.197893 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6541e8f5-a153-419f-9821-83579f859ad8 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:21:09.221532 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.auth.core [None req-6541e8f5-a153-419f-9821-83579f859ad8 None None] MFA Rules not processed for user `0e2d979563d6421995451a0e86bf3022`. Rule list: `[]` (Enabled: `True`). {{(pid=81409) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 13 04:21:09.222741 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.fernet_utils [None req-6541e8f5-a153-419f-9821-83579f859ad8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81409) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:21:09.237820 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 701/1382] 10.222.0.27 () {58 vars in 958 bytes} [Thu Oct 13 04:21:09 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 41 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 13 04:21:09.241921 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bfb00f49-5394-475f-bf82-82ebe56c3a37 None None] REQUEST_METHOD: `POST` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:21:09.242466 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bfb00f49-5394-475f-bf82-82ebe56c3a37 None None] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:21:09.243157 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bfb00f49-5394-475f-bf82-82ebe56c3a37 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:21:09.243932 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.auth.core [None req-e7062adc-6b50-4ede-94d7-d053cd6ccc5d None None] MFA Rules not processed for user `35a058c849104a9699ecfcd0c81d398e`. Rule list: `[]` (Enabled: `True`). {{(pid=81408) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 13 04:21:09.245244 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.fernet_utils [None req-e7062adc-6b50-4ede-94d7-d053cd6ccc5d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81408) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:21:09.291722 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.auth.core [None req-bfb00f49-5394-475f-bf82-82ebe56c3a37 None None] MFA Rules not processed for user `da9c75191a3b4ef49438aae9e35fcf13`. Rule list: `[]` (Enabled: `True`). {{(pid=81409) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 13 04:21:09.300422 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.fernet_utils [None req-bfb00f49-5394-475f-bf82-82ebe56c3a37 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81409) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:21:09.330126 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 702/1383] 10.222.0.27 () {58 vars in 958 bytes} [Thu Oct 13 04:21:09 2022] POST /identity/v3/auth/tokens => generated 3270 bytes in 91 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 13 04:21:09.331407 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bfb00f49-5394-475f-bf82-82ebe56c3a37 None None] Authenticating user token {{(pid=81409) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:21:09.361056 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bfb00f49-5394-475f-bf82-82ebe56c3a37 None None] Validating token access rules against request {{(pid=81409) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:21:09.374786 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d4c10789-6599-4205-b10d-e1439dd7d21f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8ba823f313144da3b683bacef8fd62e0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81409) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:21:09.375553 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4c10789-6599-4205-b10d-e1439dd7d21f admin admin] REQUEST_METHOD: `GET` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:21:09.379528 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4c10789-6599-4205-b10d-e1439dd7d21f admin admin] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:21:09.379819 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4c10789-6599-4205-b10d-e1439dd7d21f admin admin] PATH_INFO: `/v3/domains` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:21:09.380803 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d4c10789-6599-4205-b10d-e1439dd7d21f admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81409) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 13 04:21:09.381397 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d4c10789-6599-4205-b10d-e1439dd7d21f admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:21:09.383325 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d4c10789-6599-4205-b10d-e1439dd7d21f admin admin] RBAC: Authorization granted {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:21:09.421194 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 703/1384] 10.222.0.27 () {58 vars in 1153 bytes} [Thu Oct 13 04:21:09 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 90 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 13 04:21:09.423184 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d4c10789-6599-4205-b10d-e1439dd7d21f admin admin] Authenticating user token {{(pid=81409) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:21:09.439625 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 682/1385] 10.222.0.27 () {58 vars in 958 bytes} [Thu Oct 13 04:21:09 2022] POST /identity/v3/auth/tokens => generated 3266 bytes in 307 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 13 04:21:09.444549 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d4c10789-6599-4205-b10d-e1439dd7d21f admin admin] Validating token access rules against request {{(pid=81409) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:21:09.458724 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8d6a4070-00b2-492f-916b-4b2020130258 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8ba823f313144da3b683bacef8fd62e0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81409) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:21:09.459368 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d6a4070-00b2-492f-916b-4b2020130258 admin admin] REQUEST_METHOD: `POST` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:21:09.459650 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d6a4070-00b2-492f-916b-4b2020130258 admin admin] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:21:09.459925 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d6a4070-00b2-492f-916b-4b2020130258 admin admin] PATH_INFO: `/v3/projects` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:21:09.460574 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8d6a4070-00b2-492f-916b-4b2020130258 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:21:09.464492 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8d6a4070-00b2-492f-916b-4b2020130258 admin admin] RBAC: Authorization granted {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:21:09.531038 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 704/1386] 10.222.0.27 () {60 vars in 1153 bytes} [Thu Oct 13 04:21:09 2022] POST /identity/v3/projects => generated 405 bytes in 108 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 13 04:21:09.532320 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e7062adc-6b50-4ede-94d7-d053cd6ccc5d None None] Authenticating user token {{(pid=81408) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:21:09.561417 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e7062adc-6b50-4ede-94d7-d053cd6ccc5d None None] Validating token access rules against request {{(pid=81408) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:21:09.703959 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG dbcounter [-] [81408] Writing DB stats keystone:SELECT=100 {{(pid=81408) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:21:09.769853 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-af24871c-13ea-4f30-b998-cc1b5d3559a9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8ba823f313144da3b683bacef8fd62e0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81408) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:21:09.770874 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af24871c-13ea-4f30-b998-cc1b5d3559a9 admin admin] REQUEST_METHOD: `POST` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:21:09.771302 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af24871c-13ea-4f30-b998-cc1b5d3559a9 admin admin] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:21:09.771720 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af24871c-13ea-4f30-b998-cc1b5d3559a9 admin admin] PATH_INFO: `/v3/users` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:21:09.772491 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-af24871c-13ea-4f30-b998-cc1b5d3559a9 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:21:09.774612 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-af24871c-13ea-4f30-b998-cc1b5d3559a9 admin admin] RBAC: Authorization granted {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:21:09.818545 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 683/1387] 10.222.0.27 () {60 vars in 1144 bytes} [Thu Oct 13 04:21:09 2022] POST /identity/v3/users => generated 303 bytes in 287 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 13 04:21:09.822967 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8d6a4070-00b2-492f-916b-4b2020130258 admin admin] Authenticating user token {{(pid=81409) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:21:09.854977 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8d6a4070-00b2-492f-916b-4b2020130258 admin admin] Validating token access rules against request {{(pid=81409) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:21:09.879903 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7e75665c-98cb-4e7b-9950-6ccf92d2a8f7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8ba823f313144da3b683bacef8fd62e0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81409) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:21:09.880540 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e75665c-98cb-4e7b-9950-6ccf92d2a8f7 admin admin] REQUEST_METHOD: `GET` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:21:09.880815 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e75665c-98cb-4e7b-9950-6ccf92d2a8f7 admin admin] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:21:09.881080 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e75665c-98cb-4e7b-9950-6ccf92d2a8f7 admin admin] PATH_INFO: `/v3/roles` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:21:09.881725 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7e75665c-98cb-4e7b-9950-6ccf92d2a8f7 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:21:09.887786 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7e75665c-98cb-4e7b-9950-6ccf92d2a8f7 admin admin] RBAC: Authorization granted {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:21:09.913652 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc12fd3f-4ed4-416c-9038-e1906006cc38 None None] REQUEST_METHOD: `GET` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:21:09.913908 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc12fd3f-4ed4-416c-9038-e1906006cc38 None None] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:21:09.914184 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc12fd3f-4ed4-416c-9038-e1906006cc38 None None] PATH_INFO: `/` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:21:09.915719 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 684/1388] 10.222.0.27 () {54 vars in 912 bytes} [Thu Oct 13 04:21:09 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 13 04:21:09.919677 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bc12fd3f-4ed4-416c-9038-e1906006cc38 None None] Authenticating user token {{(pid=81408) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:21:09.921782 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.fernet_utils [None req-bc12fd3f-4ed4-416c-9038-e1906006cc38 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81408) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:21:09.922734 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 705/1389] 10.222.0.27 () {58 vars in 1122 bytes} [Thu Oct 13 04:21:09 2022] GET /identity/v3/roles => generated 1420 bytes in 100 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 13 04:21:09.925356 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3118f1cf-e628-4ebd-b11f-20674141e4b5 None None] REQUEST_METHOD: `POST` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:21:09.925643 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3118f1cf-e628-4ebd-b11f-20674141e4b5 None None] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:21:09.925915 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3118f1cf-e628-4ebd-b11f-20674141e4b5 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:21:09.954000 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bc12fd3f-4ed4-416c-9038-e1906006cc38 None None] Validating token access rules against request {{(pid=81408) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:21:09.963104 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fce36469-5fdb-49e0-b8b0-9f8bc71f9487 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8ba823f313144da3b683bacef8fd62e0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81408) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:21:09.963656 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fce36469-5fdb-49e0-b8b0-9f8bc71f9487 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:21:09.963932 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fce36469-5fdb-49e0-b8b0-9f8bc71f9487 admin admin] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:21:09.964203 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fce36469-5fdb-49e0-b8b0-9f8bc71f9487 admin admin] PATH_INFO: `/v3/users/d15c3f1e7b4f4f188466ccf02a1d33b4` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:21:09.968899 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fce36469-5fdb-49e0-b8b0-9f8bc71f9487 admin admin] RBAC: Authorizing `identity:delete_user(user_id=d15c3f1e7b4f4f188466ccf02a1d33b4)` {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:21:09.978142 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fce36469-5fdb-49e0-b8b0-9f8bc71f9487 admin admin] RBAC: Authorization granted {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:21:10.024122 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.auth.core [None req-3118f1cf-e628-4ebd-b11f-20674141e4b5 None None] MFA Rules not processed for user `d8812c3974f84871bb2d761cf0016257`. Rule list: `[]` (Enabled: `True`). {{(pid=81409) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 13 04:21:10.025353 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.fernet_utils [None req-3118f1cf-e628-4ebd-b11f-20674141e4b5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81409) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:21:10.049836 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG dbcounter [-] [81409] Writing DB stats keystone:DELETE=1,keystone:SELECT=98,keystone:INSERT=1 {{(pid=81409) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:21:10.208934 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.notifications [None req-fce36469-5fdb-49e0-b8b0-9f8bc71f9487 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'd15c3f1e7b4f4f188466ccf02a1d33b4'} {{(pid=81408) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 13 04:21:10.223955 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 706/1390] 10.222.0.27 () {58 vars in 1009 bytes} [Thu Oct 13 04:21:09 2022] POST /identity/v3/auth/tokens => generated 3121 bytes in 299 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 13 04:21:10.225286 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3118f1cf-e628-4ebd-b11f-20674141e4b5 None None] Authenticating user token {{(pid=81409) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:21:10.246815 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3118f1cf-e628-4ebd-b11f-20674141e4b5 None None] Validating token access rules against request {{(pid=81409) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:21:10.267192 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ce8d7156-0efc-4766-93e7-9794fb645a75 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8ba823f313144da3b683bacef8fd62e0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81409) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:21:10.267998 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce8d7156-0efc-4766-93e7-9794fb645a75 admin admin] REQUEST_METHOD: `GET` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:21:10.268412 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce8d7156-0efc-4766-93e7-9794fb645a75 admin admin] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:21:10.268829 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce8d7156-0efc-4766-93e7-9794fb645a75 admin admin] PATH_INFO: `/v3/roles` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:21:10.269651 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ce8d7156-0efc-4766-93e7-9794fb645a75 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:21:10.271466 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ce8d7156-0efc-4766-93e7-9794fb645a75 admin admin] RBAC: Authorization granted {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:21:10.278892 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.notifications [None req-fce36469-5fdb-49e0-b8b0-9f8bc71f9487 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'd15c3f1e7b4f4f188466ccf02a1d33b4'} {{(pid=81408) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 13 04:21:10.281004 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.notifications [None req-fce36469-5fdb-49e0-b8b0-9f8bc71f9487 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'd15c3f1e7b4f4f188466ccf02a1d33b4'} {{(pid=81408) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 13 04:21:10.282555 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.notifications [None req-fce36469-5fdb-49e0-b8b0-9f8bc71f9487 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'd15c3f1e7b4f4f188466ccf02a1d33b4'} {{(pid=81408) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 13 04:21:10.305002 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.notifications [None req-fce36469-5fdb-49e0-b8b0-9f8bc71f9487 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'd15c3f1e7b4f4f188466ccf02a1d33b4'} {{(pid=81408) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 13 04:21:10.308890 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 707/1391] 10.222.0.27 () {58 vars in 1122 bytes} [Thu Oct 13 04:21:10 2022] GET /identity/v3/roles => generated 1420 bytes in 84 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 13 04:21:10.310372 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef6b1675-1de4-4f33-9190-b91e4d9cd143 None None] REQUEST_METHOD: `GET` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:21:10.317607 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef6b1675-1de4-4f33-9190-b91e4d9cd143 None None] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:21:10.317967 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef6b1675-1de4-4f33-9190-b91e4d9cd143 None None] PATH_INFO: `/` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:21:10.320900 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 708/1392] 10.222.0.27 () {54 vars in 912 bytes} [Thu Oct 13 04:21:10 2022] GET /identity => generated 270 bytes in 11 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 13 04:21:10.322166 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ef6b1675-1de4-4f33-9190-b91e4d9cd143 None None] Authenticating user token {{(pid=81409) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:21:10.332204 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.fernet_utils [None req-ef6b1675-1de4-4f33-9190-b91e4d9cd143 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81409) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:21:10.362869 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ef6b1675-1de4-4f33-9190-b91e4d9cd143 None None] Validating token access rules against request {{(pid=81409) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:21:10.377380 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 685/1393] 10.222.0.27 () {58 vars in 1224 bytes} [Thu Oct 13 04:21:09 2022] DELETE /identity/v3/users/d15c3f1e7b4f4f188466ccf02a1d33b4 => generated 0 bytes in 458 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 13 04:21:10.378822 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7c717f2-8956-4dda-a916-a6f58f743a2f None None] REQUEST_METHOD: `POST` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:21:10.379103 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7c717f2-8956-4dda-a916-a6f58f743a2f None None] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:21:10.379377 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7c717f2-8956-4dda-a916-a6f58f743a2f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:21:10.382087 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0a593717-4eb5-4f2b-9a7e-06f6d66833d6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8ba823f313144da3b683bacef8fd62e0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81409) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:21:10.382837 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a593717-4eb5-4f2b-9a7e-06f6d66833d6 admin admin] REQUEST_METHOD: `PUT` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:21:10.383092 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a593717-4eb5-4f2b-9a7e-06f6d66833d6 admin admin] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:21:10.383362 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a593717-4eb5-4f2b-9a7e-06f6d66833d6 admin admin] PATH_INFO: `/v3/projects/5d861bc994b44e8a9ab9eaf7f1d048b1/users/a07e8e6b4e0d48fa9018fd1b8a7c5f92/roles/cc56c4ae13a7424ea771c15da4b153c1` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:21:10.423576 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0a593717-4eb5-4f2b-9a7e-06f6d66833d6 admin admin] RBAC: Authorizing `identity:create_grant(project_id=5d861bc994b44e8a9ab9eaf7f1d048b1, user_id=a07e8e6b4e0d48fa9018fd1b8a7c5f92, role_id=cc56c4ae13a7424ea771c15da4b153c1)` {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:21:10.425617 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.auth.core [None req-a7c717f2-8956-4dda-a916-a6f58f743a2f None None] MFA Rules not processed for user `d8812c3974f84871bb2d761cf0016257`. Rule list: `[]` (Enabled: `True`). {{(pid=81408) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 13 04:21:10.426870 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.fernet_utils [None req-a7c717f2-8956-4dda-a916-a6f58f743a2f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81408) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:21:10.428328 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0a593717-4eb5-4f2b-9a7e-06f6d66833d6 admin admin] RBAC: Authorization granted {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:21:10.460752 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 709/1394] 10.222.0.27 () {60 vars in 1483 bytes} [Thu Oct 13 04:21:10 2022] PUT /identity/v3/projects/5d861bc994b44e8a9ab9eaf7f1d048b1/users/a07e8e6b4e0d48fa9018fd1b8a7c5f92/roles/cc56c4ae13a7424ea771c15da4b153c1 => generated 0 bytes in 139 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 13 04:21:10.462164 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6476b56-7d92-4404-8d7f-841310835541 None None] REQUEST_METHOD: `POST` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:21:10.462424 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6476b56-7d92-4404-8d7f-841310835541 None None] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:21:10.462694 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6476b56-7d92-4404-8d7f-841310835541 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:21:10.532868 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.auth.core [None req-f6476b56-7d92-4404-8d7f-841310835541 None None] MFA Rules not processed for user `0e2d979563d6421995451a0e86bf3022`. Rule list: `[]` (Enabled: `True`). {{(pid=81409) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 13 04:21:10.534172 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.fernet_utils [None req-f6476b56-7d92-4404-8d7f-841310835541 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81409) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:21:10.583177 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 686/1395] 10.222.0.27 () {58 vars in 1009 bytes} [Thu Oct 13 04:21:10 2022] POST /identity/v3/auth/tokens => generated 3121 bytes in 205 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 13 04:21:10.584997 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a7c717f2-8956-4dda-a916-a6f58f743a2f None None] Authenticating user token {{(pid=81408) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:21:10.586620 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.fernet_utils [None req-a7c717f2-8956-4dda-a916-a6f58f743a2f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81408) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:21:10.608631 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a7c717f2-8956-4dda-a916-a6f58f743a2f None None] Validating token access rules against request {{(pid=81408) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:21:10.708823 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 710/1396] 10.222.0.27 () {58 vars in 958 bytes} [Thu Oct 13 04:21:10 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 248 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 13 04:21:10.711888 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ce80ad5-3103-4326-ab33-539995a57fb5 None None] REQUEST_METHOD: `POST` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:21:10.711888 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ce80ad5-3103-4326-ab33-539995a57fb5 None None] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:21:10.711888 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ce80ad5-3103-4326-ab33-539995a57fb5 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:21:10.726384 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG dbcounter [-] [81408] Writing DB stats keystone:SELECT=86,keystone:INSERT=4,keystone:DELETE=10 {{(pid=81408) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:21:10.743841 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d1c046a3-3ed5-4a6d-999d-d21f055912fa service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd8812c3974f84871bb2d761cf0016257', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dcec0431fb6849499d6aec35a234daa6', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81408) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:21:10.745740 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1c046a3-3ed5-4a6d-999d-d21f055912fa service neutron] REQUEST_METHOD: `GET` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:21:10.746791 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1c046a3-3ed5-4a6d-999d-d21f055912fa service neutron] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:21:10.747284 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1c046a3-3ed5-4a6d-999d-d21f055912fa service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:21:10.750118 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.auth.core [None req-9ce80ad5-3103-4326-ab33-539995a57fb5 None None] MFA Rules not processed for user `1f6cf725908041ccaceb1ffa694ad161`. Rule list: `[]` (Enabled: `True`). {{(pid=81409) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 13 04:21:10.750649 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.fernet_utils [None req-d1c046a3-3ed5-4a6d-999d-d21f055912fa service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81408) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:21:10.751353 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.fernet_utils [None req-9ce80ad5-3103-4326-ab33-539995a57fb5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81409) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:21:10.766617 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d1c046a3-3ed5-4a6d-999d-d21f055912fa service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:21:10.768484 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d1c046a3-3ed5-4a6d-999d-d21f055912fa service neutron] RBAC: Authorization granted {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:21:10.795344 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 687/1397] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Oct 13 04:21:10 2022] GET /identity/v3/auth/tokens => generated 3121 bytes in 211 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 13 04:21:10.799108 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0078588-5160-40eb-835b-19e25996e897 None None] REQUEST_METHOD: `POST` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:21:10.799665 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0078588-5160-40eb-835b-19e25996e897 None None] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:21:10.800161 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0078588-5160-40eb-835b-19e25996e897 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:21:10.852446 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.auth.core [None req-e0078588-5160-40eb-835b-19e25996e897 None None] MFA Rules not processed for user `0e2d979563d6421995451a0e86bf3022`. Rule list: `[]` (Enabled: `True`). {{(pid=81408) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 13 04:21:10.856890 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.fernet_utils [None req-e0078588-5160-40eb-835b-19e25996e897 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81408) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:21:10.879735 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 688/1398] 10.222.0.27 () {58 vars in 958 bytes} [Thu Oct 13 04:21:10 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 84 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 13 04:21:10.880842 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e0078588-5160-40eb-835b-19e25996e897 None None] Authenticating user token {{(pid=81408) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:21:10.884486 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.fernet_utils [None req-e0078588-5160-40eb-835b-19e25996e897 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81408) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:21:10.914150 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e0078588-5160-40eb-835b-19e25996e897 None None] Validating token access rules against request {{(pid=81408) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:21:10.925477 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c1bdc100-d2bf-4bb8-8c3d-0b5861423a15 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd8812c3974f84871bb2d761cf0016257', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dcec0431fb6849499d6aec35a234daa6', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81408) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:21:10.926924 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1bdc100-d2bf-4bb8-8c3d-0b5861423a15 service neutron] REQUEST_METHOD: `GET` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:21:10.926924 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1bdc100-d2bf-4bb8-8c3d-0b5861423a15 service neutron] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:21:10.927396 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1bdc100-d2bf-4bb8-8c3d-0b5861423a15 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:21:10.945421 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 711/1399] 10.222.0.27 () {58 vars in 958 bytes} [Thu Oct 13 04:21:10 2022] POST /identity/v3/auth/tokens => generated 3254 bytes in 236 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 13 04:21:10.946418 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9ce80ad5-3103-4326-ab33-539995a57fb5 None None] Authenticating user token {{(pid=81409) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:21:10.951290 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c1bdc100-d2bf-4bb8-8c3d-0b5861423a15 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:21:10.953132 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c1bdc100-d2bf-4bb8-8c3d-0b5861423a15 service neutron] RBAC: Authorization granted {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:21:10.969622 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9ce80ad5-3103-4326-ab33-539995a57fb5 None None] Validating token access rules against request {{(pid=81409) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:21:10.974969 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8a3d5e28-cb39-47ac-9f4f-e56890bfb2f2 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd8812c3974f84871bb2d761cf0016257', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dcec0431fb6849499d6aec35a234daa6', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81409) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:21:10.975978 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a3d5e28-cb39-47ac-9f4f-e56890bfb2f2 service neutron] REQUEST_METHOD: `GET` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:21:10.976264 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a3d5e28-cb39-47ac-9f4f-e56890bfb2f2 service neutron] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:21:10.976533 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a3d5e28-cb39-47ac-9f4f-e56890bfb2f2 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:21:10.977793 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 689/1400] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Oct 13 04:21:10 2022] GET /identity/v3/auth/tokens => generated 3121 bytes in 97 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 13 04:21:10.980212 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c1bdc100-d2bf-4bb8-8c3d-0b5861423a15 service neutron] Authenticating user token {{(pid=81408) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:21:11.003842 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c1bdc100-d2bf-4bb8-8c3d-0b5861423a15 service neutron] Validating token access rules against request {{(pid=81408) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:21:11.027758 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7e65af1d-241e-45ec-8fca-896129630601 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd8812c3974f84871bb2d761cf0016257', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dcec0431fb6849499d6aec35a234daa6', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81408) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:21:11.030856 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e65af1d-241e-45ec-8fca-896129630601 service neutron] REQUEST_METHOD: `GET` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:21:11.031411 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e65af1d-241e-45ec-8fca-896129630601 service neutron] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:21:11.032913 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e65af1d-241e-45ec-8fca-896129630601 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:21:11.043254 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8a3d5e28-cb39-47ac-9f4f-e56890bfb2f2 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:21:11.050850 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8a3d5e28-cb39-47ac-9f4f-e56890bfb2f2 service neutron] RBAC: Authorization granted {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:21:11.057675 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7e65af1d-241e-45ec-8fca-896129630601 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:21:11.059336 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7e65af1d-241e-45ec-8fca-896129630601 service neutron] RBAC: Authorization granted {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:21:11.084560 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 712/1401] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Oct 13 04:21:10 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 138 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 13 04:21:11.115459 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 690/1402] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Oct 13 04:21:10 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 137 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 13 04:21:11.908277 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8a3d5e28-cb39-47ac-9f4f-e56890bfb2f2 service neutron] Authenticating user token {{(pid=81409) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:21:11.909790 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.fernet_utils [None req-8a3d5e28-cb39-47ac-9f4f-e56890bfb2f2 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81409) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:21:11.916437 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG dbcounter [-] [81409] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81409) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:21:11.958621 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7e65af1d-241e-45ec-8fca-896129630601 service neutron] Authenticating user token {{(pid=81408) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:21:11.960098 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.fernet_utils [None req-7e65af1d-241e-45ec-8fca-896129630601 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81408) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:21:12.224287 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8a3d5e28-cb39-47ac-9f4f-e56890bfb2f2 service neutron] Validating token access rules against request {{(pid=81409) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:21:12.225416 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7e65af1d-241e-45ec-8fca-896129630601 service neutron] Validating token access rules against request {{(pid=81408) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:21:12.238239 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e625ea79-954d-4a92-b6f5-384538804da5 service placement] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ba1b6e5868a549a99760ebe6068fd6b9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dcec0431fb6849499d6aec35a234daa6', 'project_domain_id': 'default', 'roles': ['service', 'member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81408) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:21:12.238872 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e625ea79-954d-4a92-b6f5-384538804da5 service placement] REQUEST_METHOD: `GET` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:21:12.239146 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e625ea79-954d-4a92-b6f5-384538804da5 service placement] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:21:12.239417 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e625ea79-954d-4a92-b6f5-384538804da5 service placement] PATH_INFO: `/v3/auth/tokens` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:21:12.240399 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c371c7cc-4649-4b61-8b5f-3dda0da8490c service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cb740ed7f7684b079e45f887be83f795', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dcec0431fb6849499d6aec35a234daa6', 'project_domain_id': 'default', 'roles': ['ResellerAdmin', 'admin', 'reader', 'service', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81409) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:21:12.241057 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c371c7cc-4649-4b61-8b5f-3dda0da8490c service nova] REQUEST_METHOD: `GET` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:21:12.241289 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c371c7cc-4649-4b61-8b5f-3dda0da8490c service nova] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:21:12.241551 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c371c7cc-4649-4b61-8b5f-3dda0da8490c service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:21:12.243756 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.fernet_utils [None req-c371c7cc-4649-4b61-8b5f-3dda0da8490c service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81409) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:21:12.245511 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.fernet_utils [None req-e625ea79-954d-4a92-b6f5-384538804da5 service placement] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81408) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:21:12.276255 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e625ea79-954d-4a92-b6f5-384538804da5 service placement] RBAC: Authorizing `identity:validate_token()` {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:21:12.280562 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e625ea79-954d-4a92-b6f5-384538804da5 service placement] RBAC: Authorization granted {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:21:12.315877 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 691/1403] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Oct 13 04:21:11 2022] GET /identity/v3/auth/tokens => generated 3308 bytes in 358 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 13 04:21:12.318118 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e625ea79-954d-4a92-b6f5-384538804da5 service placement] Authenticating user token {{(pid=81408) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:21:12.321322 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.fernet_utils [None req-e625ea79-954d-4a92-b6f5-384538804da5 service placement] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81408) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:21:12.364535 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e625ea79-954d-4a92-b6f5-384538804da5 service placement] Validating token access rules against request {{(pid=81408) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:21:12.382568 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-66b48024-b4f0-41b9-bf19-e8ec014b4d6a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8ba823f313144da3b683bacef8fd62e0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81408) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:21:12.383215 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66b48024-b4f0-41b9-bf19-e8ec014b4d6a admin admin] REQUEST_METHOD: `DELETE` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:21:12.383491 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66b48024-b4f0-41b9-bf19-e8ec014b4d6a admin admin] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:21:12.383762 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66b48024-b4f0-41b9-bf19-e8ec014b4d6a admin admin] PATH_INFO: `/v3/projects/862fb54ea0634004989eb055ce0f7300` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:21:12.403934 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-66b48024-b4f0-41b9-bf19-e8ec014b4d6a admin admin] RBAC: Authorizing `identity:delete_project(project_id=862fb54ea0634004989eb055ce0f7300)` {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:21:12.417488 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-66b48024-b4f0-41b9-bf19-e8ec014b4d6a admin admin] RBAC: Authorization granted {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:21:12.456206 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c371c7cc-4649-4b61-8b5f-3dda0da8490c service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:21:12.462439 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c371c7cc-4649-4b61-8b5f-3dda0da8490c service nova] RBAC: Authorization granted {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:21:12.493639 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 713/1404] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Oct 13 04:21:11 2022] GET /identity/v3/auth/tokens => generated 3270 bytes in 586 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 13 04:21:12.495382 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c371c7cc-4649-4b61-8b5f-3dda0da8490c service nova] Authenticating user token {{(pid=81409) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:21:12.499509 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.fernet_utils [None req-c371c7cc-4649-4b61-8b5f-3dda0da8490c service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81409) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:21:12.543874 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c371c7cc-4649-4b61-8b5f-3dda0da8490c service nova] Validating token access rules against request {{(pid=81409) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:21:12.558102 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-11a34bcb-afdb-4a74-9eb0-3d4466a16460 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cb740ed7f7684b079e45f887be83f795', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dcec0431fb6849499d6aec35a234daa6', 'project_domain_id': 'default', 'roles': ['ResellerAdmin', 'admin', 'reader', 'service', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81409) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:21:12.559527 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11a34bcb-afdb-4a74-9eb0-3d4466a16460 service nova] REQUEST_METHOD: `GET` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:21:12.559953 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11a34bcb-afdb-4a74-9eb0-3d4466a16460 service nova] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:21:12.560368 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11a34bcb-afdb-4a74-9eb0-3d4466a16460 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:21:12.565268 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.fernet_utils [None req-11a34bcb-afdb-4a74-9eb0-3d4466a16460 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81409) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:21:12.644881 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.notifications [None req-66b48024-b4f0-41b9-bf19-e8ec014b4d6a admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '862fb54ea0634004989eb055ce0f7300', 'request_id': 'req-66b48024-b4f0-41b9-bf19-e8ec014b4d6a'} {{(pid=81408) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 13 04:21:12.693096 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.notifications [None req-66b48024-b4f0-41b9-bf19-e8ec014b4d6a admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '862fb54ea0634004989eb055ce0f7300', 'request_id': 'req-66b48024-b4f0-41b9-bf19-e8ec014b4d6a'} {{(pid=81408) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 13 04:21:12.728211 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.notifications [None req-66b48024-b4f0-41b9-bf19-e8ec014b4d6a admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '862fb54ea0634004989eb055ce0f7300', 'request_id': 'req-66b48024-b4f0-41b9-bf19-e8ec014b4d6a'} {{(pid=81408) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 13 04:21:12.771618 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.notifications [None req-66b48024-b4f0-41b9-bf19-e8ec014b4d6a admin admin] The token cache is being invalidate because project 862fb54ea0634004989eb055ce0f7300 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81408) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 13 04:21:12.771911 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.notifications [None req-66b48024-b4f0-41b9-bf19-e8ec014b4d6a admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81408) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 13 04:21:12.774239 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.notifications [None req-66b48024-b4f0-41b9-bf19-e8ec014b4d6a admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81408) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 13 04:21:12.781681 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 692/1405] 10.222.0.27 () {58 vars in 1233 bytes} [Thu Oct 13 04:21:12 2022] DELETE /identity/v3/projects/862fb54ea0634004989eb055ce0f7300 => generated 0 bytes in 462 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 13 04:21:12.784292 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-66b48024-b4f0-41b9-bf19-e8ec014b4d6a admin admin] Authenticating user token {{(pid=81408) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:21:12.785255 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-11a34bcb-afdb-4a74-9eb0-3d4466a16460 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:21:12.790724 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.fernet_utils [None req-66b48024-b4f0-41b9-bf19-e8ec014b4d6a admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81408) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:21:12.791954 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-11a34bcb-afdb-4a74-9eb0-3d4466a16460 service nova] RBAC: Authorization granted {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:21:12.823719 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 714/1406] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Oct 13 04:21:12 2022] GET /identity/v3/auth/tokens => generated 3266 bytes in 329 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 13 04:21:12.932499 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG dbcounter [-] [81408] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=81408) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:21:13.002174 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-11a34bcb-afdb-4a74-9eb0-3d4466a16460 service nova] Authenticating user token {{(pid=81409) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:21:13.024971 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-11a34bcb-afdb-4a74-9eb0-3d4466a16460 service nova] Validating token access rules against request {{(pid=81409) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:21:13.090895 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-66b48024-b4f0-41b9-bf19-e8ec014b4d6a admin admin] Validating token access rules against request {{(pid=81408) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:21:13.103250 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-796a4a83-ebbf-451d-a1a9-c6d651eecbc7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8ba823f313144da3b683bacef8fd62e0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81408) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:21:13.111336 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-796a4a83-ebbf-451d-a1a9-c6d651eecbc7 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:21:13.111826 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-796a4a83-ebbf-451d-a1a9-c6d651eecbc7 admin admin] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:21:13.112346 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-796a4a83-ebbf-451d-a1a9-c6d651eecbc7 admin admin] PATH_INFO: `/v3/users/6ab0dce872114681ac2b80714a6709e9` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:21:13.118775 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-796a4a83-ebbf-451d-a1a9-c6d651eecbc7 admin admin] RBAC: Authorizing `identity:delete_user(user_id=6ab0dce872114681ac2b80714a6709e9)` {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:21:13.129850 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-796a4a83-ebbf-451d-a1a9-c6d651eecbc7 admin admin] RBAC: Authorization granted {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:21:13.191220 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG dbcounter [-] [81409] Writing DB stats keystone:SELECT=100 {{(pid=81409) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:21:13.303779 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.notifications [None req-796a4a83-ebbf-451d-a1a9-c6d651eecbc7 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '6ab0dce872114681ac2b80714a6709e9'} {{(pid=81408) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 13 04:21:13.306888 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1704f493-c861-479b-9afa-171affeb4c08 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cb740ed7f7684b079e45f887be83f795', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dcec0431fb6849499d6aec35a234daa6', 'project_domain_id': 'default', 'roles': ['ResellerAdmin', 'admin', 'reader', 'service', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81409) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:21:13.311830 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1704f493-c861-479b-9afa-171affeb4c08 service nova] REQUEST_METHOD: `GET` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:21:13.312455 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1704f493-c861-479b-9afa-171affeb4c08 service nova] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:21:13.313130 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1704f493-c861-479b-9afa-171affeb4c08 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:21:13.315856 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.fernet_utils [None req-1704f493-c861-479b-9afa-171affeb4c08 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81409) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:21:13.337897 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.notifications [None req-796a4a83-ebbf-451d-a1a9-c6d651eecbc7 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '6ab0dce872114681ac2b80714a6709e9'} {{(pid=81408) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 13 04:21:13.341155 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.notifications [None req-796a4a83-ebbf-451d-a1a9-c6d651eecbc7 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '6ab0dce872114681ac2b80714a6709e9'} {{(pid=81408) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 13 04:21:13.343851 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.notifications [None req-796a4a83-ebbf-451d-a1a9-c6d651eecbc7 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '6ab0dce872114681ac2b80714a6709e9'} {{(pid=81408) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 13 04:21:13.384295 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.notifications [None req-796a4a83-ebbf-451d-a1a9-c6d651eecbc7 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '6ab0dce872114681ac2b80714a6709e9'} {{(pid=81408) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 13 04:21:13.460622 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 693/1407] 10.222.0.27 () {58 vars in 1224 bytes} [Thu Oct 13 04:21:12 2022] DELETE /identity/v3/users/6ab0dce872114681ac2b80714a6709e9 => generated 0 bytes in 678 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 13 04:21:13.519254 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1704f493-c861-479b-9afa-171affeb4c08 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:21:13.521951 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1704f493-c861-479b-9afa-171affeb4c08 service nova] RBAC: Authorization granted {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:21:13.552674 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 715/1408] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Oct 13 04:21:13 2022] GET /identity/v3/auth/tokens => generated 3254 bytes in 551 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Oct 13 04:21:14.148062 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c92f4294-99b8-4a76-9531-40e007b54654 None None] REQUEST_METHOD: `POST` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:21:14.149061 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c92f4294-99b8-4a76-9531-40e007b54654 None None] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:21:14.149691 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c92f4294-99b8-4a76-9531-40e007b54654 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:21:14.242690 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.auth.core [None req-c92f4294-99b8-4a76-9531-40e007b54654 None None] MFA Rules not processed for user `a07e8e6b4e0d48fa9018fd1b8a7c5f92`. Rule list: `[]` (Enabled: `True`). {{(pid=81408) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 13 04:21:14.247699 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.fernet_utils [None req-c92f4294-99b8-4a76-9531-40e007b54654 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81408) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:21:14.581478 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1704f493-c861-479b-9afa-171affeb4c08 service nova] Authenticating user token {{(pid=81409) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:21:14.583694 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.fernet_utils [None req-1704f493-c861-479b-9afa-171affeb4c08 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81409) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:21:14.585284 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 694/1409] 10.222.0.27 () {58 vars in 958 bytes} [Thu Oct 13 04:21:14 2022] POST /identity/v3/auth/tokens => generated 3282 bytes in 438 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 13 04:21:14.631193 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1704f493-c861-479b-9afa-171affeb4c08 service nova] Validating token access rules against request {{(pid=81409) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:21:14.695891 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4d59ed3-7476-41a1-8862-51e91ff0aabd None None] REQUEST_METHOD: `POST` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:21:14.696246 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4d59ed3-7476-41a1-8862-51e91ff0aabd None None] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:21:14.696522 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4d59ed3-7476-41a1-8862-51e91ff0aabd None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:21:14.753479 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.auth.core [None req-c4d59ed3-7476-41a1-8862-51e91ff0aabd None None] MFA Rules not processed for user `da9c75191a3b4ef49438aae9e35fcf13`. Rule list: `[]` (Enabled: `True`). {{(pid=81408) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 13 04:21:14.755618 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.fernet_utils [None req-c4d59ed3-7476-41a1-8862-51e91ff0aabd None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81408) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:21:14.842729 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-555e56cb-9862-4f73-9784-f212db689c41 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8ba823f313144da3b683bacef8fd62e0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81409) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:21:14.843295 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-555e56cb-9862-4f73-9784-f212db689c41 admin admin] REQUEST_METHOD: `POST` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:21:14.843577 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-555e56cb-9862-4f73-9784-f212db689c41 admin admin] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:21:14.843851 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-555e56cb-9862-4f73-9784-f212db689c41 admin admin] PATH_INFO: `/v3/projects` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:21:14.844514 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-555e56cb-9862-4f73-9784-f212db689c41 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:21:14.848153 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-555e56cb-9862-4f73-9784-f212db689c41 admin admin] RBAC: Authorization granted {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:21:14.872111 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG dbcounter [-] [81408] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81408) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:21:14.954682 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 716/1410] 10.222.0.27 () {60 vars in 1153 bytes} [Thu Oct 13 04:21:14 2022] POST /identity/v3/projects => generated 403 bytes in 374 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 13 04:21:14.955695 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-555e56cb-9862-4f73-9784-f212db689c41 admin admin] Authenticating user token {{(pid=81409) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:21:14.961628 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.fernet_utils [None req-555e56cb-9862-4f73-9784-f212db689c41 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81409) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:21:14.993780 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 695/1411] 10.222.0.27 () {58 vars in 958 bytes} [Thu Oct 13 04:21:14 2022] POST /identity/v3/auth/tokens => generated 3270 bytes in 299 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 13 04:21:14.995487 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c4d59ed3-7476-41a1-8862-51e91ff0aabd None None] Authenticating user token {{(pid=81408) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:21:15.027682 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.fernet_utils [None req-c4d59ed3-7476-41a1-8862-51e91ff0aabd None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81408) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:21:15.155552 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c4d59ed3-7476-41a1-8862-51e91ff0aabd None None] Validating token access rules against request {{(pid=81408) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:21:15.285598 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-555e56cb-9862-4f73-9784-f212db689c41 admin admin] Validating token access rules against request {{(pid=81409) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:21:15.321239 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ee0368ad-6827-4809-9124-9199fd86187b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8ba823f313144da3b683bacef8fd62e0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81409) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:21:15.322298 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee0368ad-6827-4809-9124-9199fd86187b admin admin] REQUEST_METHOD: `DELETE` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:21:15.322770 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee0368ad-6827-4809-9124-9199fd86187b admin admin] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:21:15.323233 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee0368ad-6827-4809-9124-9199fd86187b admin admin] PATH_INFO: `/v3/projects/49aa2543e5d44238a451f724b4219caa` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:21:15.324610 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ee0368ad-6827-4809-9124-9199fd86187b admin admin] RBAC: Authorizing `identity:delete_project(project_id=49aa2543e5d44238a451f724b4219caa)` {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:21:15.329707 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ee0368ad-6827-4809-9124-9199fd86187b admin admin] RBAC: Authorization granted {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:21:15.334041 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG dbcounter [-] [81409] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81409) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:21:15.449109 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.notifications [None req-ee0368ad-6827-4809-9124-9199fd86187b admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '49aa2543e5d44238a451f724b4219caa', 'request_id': 'req-ee0368ad-6827-4809-9124-9199fd86187b'} {{(pid=81409) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 13 04:21:15.461581 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.notifications [None req-ee0368ad-6827-4809-9124-9199fd86187b admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '49aa2543e5d44238a451f724b4219caa', 'request_id': 'req-ee0368ad-6827-4809-9124-9199fd86187b'} {{(pid=81409) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 13 04:21:15.492462 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.notifications [None req-ee0368ad-6827-4809-9124-9199fd86187b admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '49aa2543e5d44238a451f724b4219caa', 'request_id': 'req-ee0368ad-6827-4809-9124-9199fd86187b'} {{(pid=81409) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 13 04:21:15.497705 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-278e1681-3a78-4c58-ac33-98f88401dd03 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8ba823f313144da3b683bacef8fd62e0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81408) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:21:15.498585 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-278e1681-3a78-4c58-ac33-98f88401dd03 admin admin] REQUEST_METHOD: `POST` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:21:15.499100 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-278e1681-3a78-4c58-ac33-98f88401dd03 admin admin] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:21:15.499519 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-278e1681-3a78-4c58-ac33-98f88401dd03 admin admin] PATH_INFO: `/v3/users` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:21:15.500308 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-278e1681-3a78-4c58-ac33-98f88401dd03 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:21:15.503019 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-278e1681-3a78-4c58-ac33-98f88401dd03 admin admin] RBAC: Authorization granted {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:21:15.529934 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.notifications [None req-ee0368ad-6827-4809-9124-9199fd86187b admin admin] The token cache is being invalidate because project 49aa2543e5d44238a451f724b4219caa was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81409) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 13 04:21:15.532664 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.notifications [None req-ee0368ad-6827-4809-9124-9199fd86187b admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81409) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 13 04:21:15.536307 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.notifications [None req-ee0368ad-6827-4809-9124-9199fd86187b admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81409) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 13 04:21:15.544627 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 717/1412] 10.222.0.27 () {58 vars in 1233 bytes} [Thu Oct 13 04:21:14 2022] DELETE /identity/v3/projects/49aa2543e5d44238a451f724b4219caa => generated 0 bytes in 589 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 13 04:21:15.549588 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 696/1413] 10.222.0.27 () {60 vars in 1144 bytes} [Thu Oct 13 04:21:14 2022] POST /identity/v3/users => generated 308 bytes in 555 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 13 04:21:15.550300 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f1dbe31-2b54-459b-a898-e6e2b0b1f475 None None] REQUEST_METHOD: `POST` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:21:15.550480 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f1dbe31-2b54-459b-a898-e6e2b0b1f475 None None] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:21:15.551914 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f1dbe31-2b54-459b-a898-e6e2b0b1f475 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:21:15.552344 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-278e1681-3a78-4c58-ac33-98f88401dd03 admin admin] Authenticating user token {{(pid=81408) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:21:15.554825 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.fernet_utils [None req-278e1681-3a78-4c58-ac33-98f88401dd03 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81408) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:21:15.571344 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-278e1681-3a78-4c58-ac33-98f88401dd03 admin admin] Validating token access rules against request {{(pid=81408) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:21:15.660182 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.auth.core [None req-6f1dbe31-2b54-459b-a898-e6e2b0b1f475 None None] MFA Rules not processed for user `0e2d979563d6421995451a0e86bf3022`. Rule list: `[]` (Enabled: `True`). {{(pid=81409) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 13 04:21:15.666906 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.fernet_utils [None req-6f1dbe31-2b54-459b-a898-e6e2b0b1f475 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81409) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:21:15.719329 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-082324e9-2d01-412b-91f5-f7cbce35089f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8ba823f313144da3b683bacef8fd62e0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81408) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:21:15.719967 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-082324e9-2d01-412b-91f5-f7cbce35089f admin admin] REQUEST_METHOD: `GET` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:21:15.720247 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-082324e9-2d01-412b-91f5-f7cbce35089f admin admin] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:21:15.730517 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-082324e9-2d01-412b-91f5-f7cbce35089f admin admin] PATH_INFO: `/v3/roles` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:21:15.731618 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-082324e9-2d01-412b-91f5-f7cbce35089f admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:21:15.733357 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-082324e9-2d01-412b-91f5-f7cbce35089f admin admin] RBAC: Authorization granted {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:21:15.753168 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 697/1414] 10.222.0.27 () {58 vars in 1122 bytes} [Thu Oct 13 04:21:15 2022] GET /identity/v3/roles => generated 1420 bytes in 203 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 13 04:21:15.768449 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-082324e9-2d01-412b-91f5-f7cbce35089f admin admin] Authenticating user token {{(pid=81408) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:21:15.795077 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-082324e9-2d01-412b-91f5-f7cbce35089f admin admin] Validating token access rules against request {{(pid=81408) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:21:15.809972 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e9c777e6-cd2c-469b-b984-2459e3fef487 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8ba823f313144da3b683bacef8fd62e0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81408) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:21:15.810904 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9c777e6-cd2c-469b-b984-2459e3fef487 admin admin] REQUEST_METHOD: `PUT` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:21:15.811097 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9c777e6-cd2c-469b-b984-2459e3fef487 admin admin] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:21:15.812047 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9c777e6-cd2c-469b-b984-2459e3fef487 admin admin] PATH_INFO: `/v3/projects/9e18bc36229e4e118f04206e4b5765a7/users/7e311300d40945edb61546b619d13f31/roles/ef55fbd94e1040be83695754075a7ed5` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:21:15.851257 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e9c777e6-cd2c-469b-b984-2459e3fef487 admin admin] RBAC: Authorizing `identity:create_grant(project_id=9e18bc36229e4e118f04206e4b5765a7, user_id=7e311300d40945edb61546b619d13f31, role_id=ef55fbd94e1040be83695754075a7ed5)` {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:21:15.854142 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e9c777e6-cd2c-469b-b984-2459e3fef487 admin admin] RBAC: Authorization granted {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:21:15.893426 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 698/1415] 10.222.0.27 () {60 vars in 1483 bytes} [Thu Oct 13 04:21:15 2022] PUT /identity/v3/projects/9e18bc36229e4e118f04206e4b5765a7/users/7e311300d40945edb61546b619d13f31/roles/ef55fbd94e1040be83695754075a7ed5 => generated 0 bytes in 133 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 13 04:21:15.894691 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e9c777e6-cd2c-469b-b984-2459e3fef487 admin admin] Authenticating user token {{(pid=81408) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:21:15.954987 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 718/1416] 10.222.0.27 () {58 vars in 958 bytes} [Thu Oct 13 04:21:15 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 406 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 13 04:21:15.962703 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c679fd28-4911-4c4f-a0d7-b469d318ee6b None None] REQUEST_METHOD: `POST` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:21:15.963237 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c679fd28-4911-4c4f-a0d7-b469d318ee6b None None] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:21:15.963662 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c679fd28-4911-4c4f-a0d7-b469d318ee6b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:21:16.024016 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.auth.core [None req-c679fd28-4911-4c4f-a0d7-b469d318ee6b None None] MFA Rules not processed for user `0e2d979563d6421995451a0e86bf3022`. Rule list: `[]` (Enabled: `True`). {{(pid=81409) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 13 04:21:16.024267 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG dbcounter [-] [81408] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=81408) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:21:16.025282 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.fernet_utils [None req-c679fd28-4911-4c4f-a0d7-b469d318ee6b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81409) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:21:16.061439 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e9c777e6-cd2c-469b-b984-2459e3fef487 admin admin] Validating token access rules against request {{(pid=81408) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:21:16.077054 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-67de97d7-a2b7-48ab-9533-25ed34deed33 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8ba823f313144da3b683bacef8fd62e0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81408) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:21:16.077743 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67de97d7-a2b7-48ab-9533-25ed34deed33 admin admin] REQUEST_METHOD: `GET` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:21:16.078071 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67de97d7-a2b7-48ab-9533-25ed34deed33 admin admin] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:21:16.079044 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67de97d7-a2b7-48ab-9533-25ed34deed33 admin admin] PATH_INFO: `/v3/roles` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:21:16.079859 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-67de97d7-a2b7-48ab-9533-25ed34deed33 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:21:16.099048 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-67de97d7-a2b7-48ab-9533-25ed34deed33 admin admin] RBAC: Authorization granted {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:21:16.123516 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 699/1417] 10.222.0.27 () {58 vars in 1122 bytes} [Thu Oct 13 04:21:15 2022] GET /identity/v3/roles => generated 1420 bytes in 229 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Oct 13 04:21:16.128691 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-67de97d7-a2b7-48ab-9533-25ed34deed33 admin admin] Authenticating user token {{(pid=81408) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:21:16.147813 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-67de97d7-a2b7-48ab-9533-25ed34deed33 admin admin] Validating token access rules against request {{(pid=81408) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:21:16.161600 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f7c793f5-8706-4666-83fb-aae6a8d423ca admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8ba823f313144da3b683bacef8fd62e0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81408) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:21:16.163152 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 719/1418] 10.222.0.27 () {58 vars in 958 bytes} [Thu Oct 13 04:21:15 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 201 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 13 04:21:16.165629 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7c793f5-8706-4666-83fb-aae6a8d423ca admin admin] REQUEST_METHOD: `PUT` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:21:16.165629 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7c793f5-8706-4666-83fb-aae6a8d423ca admin admin] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:21:16.165629 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7c793f5-8706-4666-83fb-aae6a8d423ca admin admin] PATH_INFO: `/v3/projects/9e18bc36229e4e118f04206e4b5765a7/users/7e311300d40945edb61546b619d13f31/roles/ef55fbd94e1040be83695754075a7ed5` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:21:16.168291 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c679fd28-4911-4c4f-a0d7-b469d318ee6b None None] Authenticating user token {{(pid=81409) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:21:16.169079 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f7c793f5-8706-4666-83fb-aae6a8d423ca admin admin] RBAC: Authorizing `identity:create_grant(project_id=9e18bc36229e4e118f04206e4b5765a7, user_id=7e311300d40945edb61546b619d13f31, role_id=ef55fbd94e1040be83695754075a7ed5)` {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:21:16.173384 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f7c793f5-8706-4666-83fb-aae6a8d423ca admin admin] RBAC: Authorization granted {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:21:16.192387 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 700/1419] 10.222.0.27 () {60 vars in 1483 bytes} [Thu Oct 13 04:21:16 2022] PUT /identity/v3/projects/9e18bc36229e4e118f04206e4b5765a7/users/7e311300d40945edb61546b619d13f31/roles/ef55fbd94e1040be83695754075a7ed5 => generated 0 bytes in 64 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 13 04:21:16.210657 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c679fd28-4911-4c4f-a0d7-b469d318ee6b None None] Validating token access rules against request {{(pid=81409) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:21:16.305938 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG dbcounter [-] [81409] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=81409) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:21:16.412752 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be94d353-2bea-4681-8518-41acb2b9e33a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8ba823f313144da3b683bacef8fd62e0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81409) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:21:16.416027 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be94d353-2bea-4681-8518-41acb2b9e33a admin admin] REQUEST_METHOD: `GET` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:21:16.417653 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be94d353-2bea-4681-8518-41acb2b9e33a admin admin] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:21:16.418658 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be94d353-2bea-4681-8518-41acb2b9e33a admin admin] PATH_INFO: `/v3/domains` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:21:16.419571 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-be94d353-2bea-4681-8518-41acb2b9e33a admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81409) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 13 04:21:16.421012 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-be94d353-2bea-4681-8518-41acb2b9e33a admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:21:16.422955 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-be94d353-2bea-4681-8518-41acb2b9e33a admin admin] RBAC: Authorization granted {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:21:16.453489 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 720/1420] 10.222.0.27 () {58 vars in 1153 bytes} [Thu Oct 13 04:21:16 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 287 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 13 04:21:16.456942 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f7c793f5-8706-4666-83fb-aae6a8d423ca admin admin] Authenticating user token {{(pid=81408) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:21:16.477797 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f7c793f5-8706-4666-83fb-aae6a8d423ca admin admin] Validating token access rules against request {{(pid=81408) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:21:16.493998 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e87353d7-2a87-432a-be2e-5418ee6f182a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8ba823f313144da3b683bacef8fd62e0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81408) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:21:16.494657 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e87353d7-2a87-432a-be2e-5418ee6f182a admin admin] REQUEST_METHOD: `POST` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:21:16.494933 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e87353d7-2a87-432a-be2e-5418ee6f182a admin admin] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:21:16.495200 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e87353d7-2a87-432a-be2e-5418ee6f182a admin admin] PATH_INFO: `/v3/projects` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:21:16.498934 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e87353d7-2a87-432a-be2e-5418ee6f182a admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:21:16.507763 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e87353d7-2a87-432a-be2e-5418ee6f182a admin admin] RBAC: Authorization granted {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:21:16.581911 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 701/1421] 10.222.0.27 () {60 vars in 1153 bytes} [Thu Oct 13 04:21:16 2022] POST /identity/v3/projects => generated 401 bytes in 126 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 13 04:21:16.586075 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be94d353-2bea-4681-8518-41acb2b9e33a admin admin] Authenticating user token {{(pid=81409) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:21:16.648562 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be94d353-2bea-4681-8518-41acb2b9e33a admin admin] Validating token access rules against request {{(pid=81409) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:21:16.840271 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7546eda2-6263-4ad1-b434-5db6a80cbc7e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8ba823f313144da3b683bacef8fd62e0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81409) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:21:16.842092 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7546eda2-6263-4ad1-b434-5db6a80cbc7e admin admin] REQUEST_METHOD: `POST` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:21:16.842685 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7546eda2-6263-4ad1-b434-5db6a80cbc7e admin admin] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:21:16.843208 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7546eda2-6263-4ad1-b434-5db6a80cbc7e admin admin] PATH_INFO: `/v3/users` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:21:16.845329 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7546eda2-6263-4ad1-b434-5db6a80cbc7e admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:21:16.853702 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7546eda2-6263-4ad1-b434-5db6a80cbc7e admin admin] RBAC: Authorization granted {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:21:16.934632 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 721/1422] 10.222.0.27 () {60 vars in 1144 bytes} [Thu Oct 13 04:21:16 2022] POST /identity/v3/users => generated 301 bytes in 346 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 13 04:21:16.936702 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e87353d7-2a87-432a-be2e-5418ee6f182a admin admin] Authenticating user token {{(pid=81408) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:21:16.949631 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e87353d7-2a87-432a-be2e-5418ee6f182a admin admin] Validating token access rules against request {{(pid=81408) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:21:16.979604 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-10844cec-2a2c-419d-ae5e-b3ed9647017a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8ba823f313144da3b683bacef8fd62e0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81408) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:21:16.980249 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10844cec-2a2c-419d-ae5e-b3ed9647017a admin admin] REQUEST_METHOD: `GET` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:21:16.980522 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10844cec-2a2c-419d-ae5e-b3ed9647017a admin admin] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:21:16.980789 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10844cec-2a2c-419d-ae5e-b3ed9647017a admin admin] PATH_INFO: `/v3/roles` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:21:16.981710 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-10844cec-2a2c-419d-ae5e-b3ed9647017a admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:21:16.994303 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-10844cec-2a2c-419d-ae5e-b3ed9647017a admin admin] RBAC: Authorization granted {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:21:17.026439 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 702/1423] 10.222.0.27 () {58 vars in 1122 bytes} [Thu Oct 13 04:21:16 2022] GET /identity/v3/roles => generated 1420 bytes in 88 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 13 04:21:17.028371 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7546eda2-6263-4ad1-b434-5db6a80cbc7e admin admin] Authenticating user token {{(pid=81409) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:21:17.042889 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7546eda2-6263-4ad1-b434-5db6a80cbc7e admin admin] Validating token access rules against request {{(pid=81409) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:21:17.053218 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-56c6bac6-c2b8-4b1e-b90b-4fda28d9fd1e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8ba823f313144da3b683bacef8fd62e0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81409) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:21:17.054147 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56c6bac6-c2b8-4b1e-b90b-4fda28d9fd1e admin admin] REQUEST_METHOD: `GET` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:21:17.054849 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56c6bac6-c2b8-4b1e-b90b-4fda28d9fd1e admin admin] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:21:17.055359 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56c6bac6-c2b8-4b1e-b90b-4fda28d9fd1e admin admin] PATH_INFO: `/v3/roles` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:21:17.056455 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-56c6bac6-c2b8-4b1e-b90b-4fda28d9fd1e admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:21:17.058458 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-56c6bac6-c2b8-4b1e-b90b-4fda28d9fd1e admin admin] RBAC: Authorization granted {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:21:17.081947 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-10844cec-2a2c-419d-ae5e-b3ed9647017a admin admin] Authenticating user token {{(pid=81408) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:21:17.094151 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 722/1424] 10.222.0.27 () {58 vars in 1122 bytes} [Thu Oct 13 04:21:17 2022] GET /identity/v3/roles => generated 1420 bytes in 66 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 13 04:21:17.100274 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-10844cec-2a2c-419d-ae5e-b3ed9647017a admin admin] Validating token access rules against request {{(pid=81408) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:21:17.117148 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4aeb7b76-ff3b-49af-876d-9cebc108bc5c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8ba823f313144da3b683bacef8fd62e0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81408) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:21:17.117894 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4aeb7b76-ff3b-49af-876d-9cebc108bc5c admin admin] REQUEST_METHOD: `PUT` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:21:17.118174 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4aeb7b76-ff3b-49af-876d-9cebc108bc5c admin admin] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:21:17.118480 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4aeb7b76-ff3b-49af-876d-9cebc108bc5c admin admin] PATH_INFO: `/v3/projects/085767f5d45b40eeb7c213298abe6545/users/6372398a35be42acba1b1822905e7a50/roles/cc56c4ae13a7424ea771c15da4b153c1` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:21:17.156191 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4aeb7b76-ff3b-49af-876d-9cebc108bc5c admin admin] RBAC: Authorizing `identity:create_grant(project_id=085767f5d45b40eeb7c213298abe6545, user_id=6372398a35be42acba1b1822905e7a50, role_id=cc56c4ae13a7424ea771c15da4b153c1)` {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:21:17.160705 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4aeb7b76-ff3b-49af-876d-9cebc108bc5c admin admin] RBAC: Authorization granted {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:21:17.236045 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 703/1425] 10.222.0.27 () {60 vars in 1483 bytes} [Thu Oct 13 04:21:17 2022] PUT /identity/v3/projects/085767f5d45b40eeb7c213298abe6545/users/6372398a35be42acba1b1822905e7a50/roles/cc56c4ae13a7424ea771c15da4b153c1 => generated 0 bytes in 145 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 13 04:21:17.242565 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-569a9fbf-6312-4a7f-9580-8b744fc50f0b None None] REQUEST_METHOD: `POST` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:21:17.245041 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-569a9fbf-6312-4a7f-9580-8b744fc50f0b None None] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:21:17.245692 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-569a9fbf-6312-4a7f-9580-8b744fc50f0b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:21:17.351883 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.auth.core [None req-569a9fbf-6312-4a7f-9580-8b744fc50f0b None None] MFA Rules not processed for user `6372398a35be42acba1b1822905e7a50`. Rule list: `[]` (Enabled: `True`). {{(pid=81409) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 13 04:21:17.353490 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.fernet_utils [None req-569a9fbf-6312-4a7f-9580-8b744fc50f0b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81409) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:21:17.496860 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9c51cb9-e0ee-488b-9a1e-fe610a0abb28 None None] REQUEST_METHOD: `POST` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:21:17.497139 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9c51cb9-e0ee-488b-9a1e-fe610a0abb28 None None] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:21:17.497593 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9c51cb9-e0ee-488b-9a1e-fe610a0abb28 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:21:17.641809 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 723/1426] 10.222.0.27 () {58 vars in 958 bytes} [Thu Oct 13 04:21:17 2022] POST /identity/v3/auth/tokens => generated 3278 bytes in 404 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 13 04:21:17.647610 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-569a9fbf-6312-4a7f-9580-8b744fc50f0b None None] Authenticating user token {{(pid=81409) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:21:17.708436 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.auth.core [None req-a9c51cb9-e0ee-488b-9a1e-fe610a0abb28 None None] MFA Rules not processed for user `487b1605fd4143cfaa422a3a7a0344c7`. Rule list: `[]` (Enabled: `True`). {{(pid=81408) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 13 04:21:17.710696 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.fernet_utils [None req-a9c51cb9-e0ee-488b-9a1e-fe610a0abb28 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81408) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:21:17.791625 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG dbcounter [-] [81409] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81409) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:21:17.857530 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-569a9fbf-6312-4a7f-9580-8b744fc50f0b None None] Validating token access rules against request {{(pid=81409) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:21:17.864193 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2af84ce7-d93d-4837-a0f7-72251a44624d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8ba823f313144da3b683bacef8fd62e0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81409) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:21:17.865098 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2af84ce7-d93d-4837-a0f7-72251a44624d admin admin] REQUEST_METHOD: `POST` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:21:17.866036 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2af84ce7-d93d-4837-a0f7-72251a44624d admin admin] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:21:17.866841 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2af84ce7-d93d-4837-a0f7-72251a44624d admin admin] PATH_INFO: `/v3/projects` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:21:17.867678 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2af84ce7-d93d-4837-a0f7-72251a44624d admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:21:17.871043 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2af84ce7-d93d-4837-a0f7-72251a44624d admin admin] RBAC: Authorization granted {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:21:17.919528 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 724/1427] 10.222.0.27 () {60 vars in 1153 bytes} [Thu Oct 13 04:21:17 2022] POST /identity/v3/projects => generated 401 bytes in 272 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 13 04:21:17.924163 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2af84ce7-d93d-4837-a0f7-72251a44624d admin admin] Authenticating user token {{(pid=81409) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:21:17.954046 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 704/1428] 10.222.0.27 () {58 vars in 1009 bytes} [Thu Oct 13 04:21:17 2022] POST /identity/v3/auth/tokens => generated 3195 bytes in 459 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 13 04:21:18.064887 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2af84ce7-d93d-4837-a0f7-72251a44624d admin admin] Validating token access rules against request {{(pid=81409) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:21:18.078998 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ffd457ec-2a2f-4169-ad8d-017564a70abd admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8ba823f313144da3b683bacef8fd62e0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81409) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:21:18.079657 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ffd457ec-2a2f-4169-ad8d-017564a70abd admin admin] REQUEST_METHOD: `POST` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:21:18.081672 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ffd457ec-2a2f-4169-ad8d-017564a70abd admin admin] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:21:18.081672 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ffd457ec-2a2f-4169-ad8d-017564a70abd admin admin] PATH_INFO: `/v3/users` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:21:18.082526 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ffd457ec-2a2f-4169-ad8d-017564a70abd admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:21:18.087021 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ffd457ec-2a2f-4169-ad8d-017564a70abd admin admin] RBAC: Authorization granted {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:21:18.160818 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 725/1429] 10.222.0.27 () {60 vars in 1144 bytes} [Thu Oct 13 04:21:17 2022] POST /identity/v3/users => generated 307 bytes in 237 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 13 04:21:18.165182 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a9c51cb9-e0ee-488b-9a1e-fe610a0abb28 None None] Authenticating user token {{(pid=81408) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:21:18.197530 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a9c51cb9-e0ee-488b-9a1e-fe610a0abb28 None None] Validating token access rules against request {{(pid=81408) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:21:18.212836 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6b97352e-9a14-407c-90d3-9b2dcfec6991 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8ba823f313144da3b683bacef8fd62e0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81408) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:21:18.213687 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b97352e-9a14-407c-90d3-9b2dcfec6991 admin admin] REQUEST_METHOD: `GET` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:21:18.214165 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ffd457ec-2a2f-4169-ad8d-017564a70abd admin admin] Authenticating user token {{(pid=81409) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:21:18.214352 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b97352e-9a14-407c-90d3-9b2dcfec6991 admin admin] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:21:18.214925 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b97352e-9a14-407c-90d3-9b2dcfec6991 admin admin] PATH_INFO: `/v3/roles` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:21:18.215745 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6b97352e-9a14-407c-90d3-9b2dcfec6991 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:21:18.216135 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.fernet_utils [None req-ffd457ec-2a2f-4169-ad8d-017564a70abd admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81409) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:21:18.220687 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6b97352e-9a14-407c-90d3-9b2dcfec6991 admin admin] RBAC: Authorization granted {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:21:18.252907 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 705/1430] 10.222.0.27 () {58 vars in 1122 bytes} [Thu Oct 13 04:21:18 2022] GET /identity/v3/roles => generated 1420 bytes in 88 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 13 04:21:18.256098 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6b97352e-9a14-407c-90d3-9b2dcfec6991 admin admin] Authenticating user token {{(pid=81408) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:21:18.271891 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6b97352e-9a14-407c-90d3-9b2dcfec6991 admin admin] Validating token access rules against request {{(pid=81408) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:21:18.297340 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5278aa87-2f33-43e1-861b-722d57f0b9f2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8ba823f313144da3b683bacef8fd62e0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81408) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:21:18.298696 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5278aa87-2f33-43e1-861b-722d57f0b9f2 admin admin] REQUEST_METHOD: `PUT` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:21:18.299022 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5278aa87-2f33-43e1-861b-722d57f0b9f2 admin admin] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:21:18.299299 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5278aa87-2f33-43e1-861b-722d57f0b9f2 admin admin] PATH_INFO: `/v3/projects/8e50e80a13214d0497c3d0f3c8b06d5f/users/26c38224278541b3a64848042505ff26/roles/ef55fbd94e1040be83695754075a7ed5` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:21:18.335214 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5278aa87-2f33-43e1-861b-722d57f0b9f2 admin admin] RBAC: Authorizing `identity:create_grant(project_id=8e50e80a13214d0497c3d0f3c8b06d5f, user_id=26c38224278541b3a64848042505ff26, role_id=ef55fbd94e1040be83695754075a7ed5)` {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:21:18.338739 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5278aa87-2f33-43e1-861b-722d57f0b9f2 admin admin] RBAC: Authorization granted {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:21:18.365534 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 706/1431] 10.222.0.27 () {60 vars in 1483 bytes} [Thu Oct 13 04:21:18 2022] PUT /identity/v3/projects/8e50e80a13214d0497c3d0f3c8b06d5f/users/26c38224278541b3a64848042505ff26/roles/ef55fbd94e1040be83695754075a7ed5 => generated 0 bytes in 110 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 13 04:21:18.374505 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5278aa87-2f33-43e1-861b-722d57f0b9f2 admin admin] Authenticating user token {{(pid=81408) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:21:18.395995 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG dbcounter [-] [81408] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=81408) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:21:18.522681 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5278aa87-2f33-43e1-861b-722d57f0b9f2 admin admin] Validating token access rules against request {{(pid=81408) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:21:18.539553 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-add7cb78-7f1b-4d65-95cc-cbd91ddf0036 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8ba823f313144da3b683bacef8fd62e0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81408) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:21:18.542425 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-add7cb78-7f1b-4d65-95cc-cbd91ddf0036 admin admin] REQUEST_METHOD: `GET` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:21:18.543463 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-add7cb78-7f1b-4d65-95cc-cbd91ddf0036 admin admin] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:21:18.544533 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-add7cb78-7f1b-4d65-95cc-cbd91ddf0036 admin admin] PATH_INFO: `/v3/roles` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:21:18.546183 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-add7cb78-7f1b-4d65-95cc-cbd91ddf0036 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:21:18.551287 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-add7cb78-7f1b-4d65-95cc-cbd91ddf0036 admin admin] RBAC: Authorization granted {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:21:18.578260 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 707/1432] 10.222.0.27 () {58 vars in 1122 bytes} [Thu Oct 13 04:21:18 2022] GET /identity/v3/roles => generated 1420 bytes in 207 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 13 04:21:18.579220 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-add7cb78-7f1b-4d65-95cc-cbd91ddf0036 admin admin] Authenticating user token {{(pid=81408) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:21:18.605036 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-add7cb78-7f1b-4d65-95cc-cbd91ddf0036 admin admin] Validating token access rules against request {{(pid=81408) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:21:18.606766 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ffd457ec-2a2f-4169-ad8d-017564a70abd admin admin] Validating token access rules against request {{(pid=81409) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:21:18.618788 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-067e0573-65f2-4adb-8225-5784fb0d7faf admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8ba823f313144da3b683bacef8fd62e0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81408) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:21:18.619917 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-067e0573-65f2-4adb-8225-5784fb0d7faf admin admin] REQUEST_METHOD: `PUT` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:21:18.622071 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-067e0573-65f2-4adb-8225-5784fb0d7faf admin admin] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:21:18.622536 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-067e0573-65f2-4adb-8225-5784fb0d7faf admin admin] PATH_INFO: `/v3/projects/8e50e80a13214d0497c3d0f3c8b06d5f/users/26c38224278541b3a64848042505ff26/roles/ef55fbd94e1040be83695754075a7ed5` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:21:18.626988 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-067e0573-65f2-4adb-8225-5784fb0d7faf admin admin] RBAC: Authorizing `identity:create_grant(project_id=8e50e80a13214d0497c3d0f3c8b06d5f, user_id=26c38224278541b3a64848042505ff26, role_id=ef55fbd94e1040be83695754075a7ed5)` {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:21:18.633954 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-067e0573-65f2-4adb-8225-5784fb0d7faf admin admin] RBAC: Authorization granted {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:21:18.674341 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 708/1433] 10.222.0.27 () {60 vars in 1483 bytes} [Thu Oct 13 04:21:18 2022] PUT /identity/v3/projects/8e50e80a13214d0497c3d0f3c8b06d5f/users/26c38224278541b3a64848042505ff26/roles/ef55fbd94e1040be83695754075a7ed5 => generated 0 bytes in 86 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 13 04:21:18.678365 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9b3a23d-1e88-45a9-91cd-88bc35e916f8 None None] REQUEST_METHOD: `POST` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:21:18.678904 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9b3a23d-1e88-45a9-91cd-88bc35e916f8 None None] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:21:18.679349 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9b3a23d-1e88-45a9-91cd-88bc35e916f8 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:21:18.709613 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG dbcounter [-] [81409] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=81409) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:21:18.744137 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.auth.core [None req-b9b3a23d-1e88-45a9-91cd-88bc35e916f8 None None] MFA Rules not processed for user `26c38224278541b3a64848042505ff26`. Rule list: `[]` (Enabled: `True`). {{(pid=81408) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 13 04:21:18.745344 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.fernet_utils [None req-b9b3a23d-1e88-45a9-91cd-88bc35e916f8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81408) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:21:18.796970 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d3b35bb6-6fc6-4a79-ba6d-4e3c597c3374 service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90a0f466b664458898cf9dc4112ae651', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dcec0431fb6849499d6aec35a234daa6', 'project_domain_id': 'default', 'roles': ['service', 'member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81409) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:21:18.797782 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3b35bb6-6fc6-4a79-ba6d-4e3c597c3374 service swift] REQUEST_METHOD: `GET` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:21:18.798062 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3b35bb6-6fc6-4a79-ba6d-4e3c597c3374 service swift] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:21:18.798364 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3b35bb6-6fc6-4a79-ba6d-4e3c597c3374 service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:21:18.930146 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 709/1434] 10.222.0.27 () {58 vars in 958 bytes} [Thu Oct 13 04:21:18 2022] POST /identity/v3/auth/tokens => generated 3345 bytes in 253 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 13 04:21:18.945547 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b9b3a23d-1e88-45a9-91cd-88bc35e916f8 None None] Authenticating user token {{(pid=81408) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:21:18.950138 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.fernet_utils [None req-b9b3a23d-1e88-45a9-91cd-88bc35e916f8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81408) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:21:18.955683 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d3b35bb6-6fc6-4a79-ba6d-4e3c597c3374 service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:21:18.957295 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d3b35bb6-6fc6-4a79-ba6d-4e3c597c3374 service swift] RBAC: Authorization granted {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:21:18.981168 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 726/1435] 10.222.0.27 () {60 vars in 1384 bytes} [Thu Oct 13 04:21:18 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 767 msecs (HTTP/1.1 200) 6 headers in 379 bytes (1 switches on core 0) Oct 13 04:21:19.023298 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b154dbe2-a4a9-41f6-8e75-462325236e59 None None] REQUEST_METHOD: `POST` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:21:19.023590 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b154dbe2-a4a9-41f6-8e75-462325236e59 None None] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:21:19.023863 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b154dbe2-a4a9-41f6-8e75-462325236e59 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:21:19.128020 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.auth.core [None req-b154dbe2-a4a9-41f6-8e75-462325236e59 None None] MFA Rules not processed for user `7e311300d40945edb61546b619d13f31`. Rule list: `[]` (Enabled: `True`). {{(pid=81409) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 13 04:21:19.129731 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.fernet_utils [None req-b154dbe2-a4a9-41f6-8e75-462325236e59 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81409) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:21:19.166184 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b9b3a23d-1e88-45a9-91cd-88bc35e916f8 None None] Validating token access rules against request {{(pid=81408) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:21:19.191015 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fecec67e-ea41-4f53-9817-807b8b09a676 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6141b8a6c3d54e179d6879bb019f4d12', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dcec0431fb6849499d6aec35a234daa6', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81408) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:21:19.191850 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fecec67e-ea41-4f53-9817-807b8b09a676 service cinder] REQUEST_METHOD: `GET` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:21:19.192275 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fecec67e-ea41-4f53-9817-807b8b09a676 service cinder] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:21:19.192689 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fecec67e-ea41-4f53-9817-807b8b09a676 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:21:19.245134 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fecec67e-ea41-4f53-9817-807b8b09a676 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:21:19.252035 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fecec67e-ea41-4f53-9817-807b8b09a676 service cinder] RBAC: Authorization granted {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:21:19.293185 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 710/1436] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Oct 13 04:21:18 2022] GET /identity/v3/auth/tokens => generated 3345 bytes in 348 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 13 04:21:19.409403 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 727/1437] 10.222.0.27 () {58 vars in 958 bytes} [Thu Oct 13 04:21:19 2022] POST /identity/v3/auth/tokens => generated 3347 bytes in 387 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 13 04:21:19.541996 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fecec67e-ea41-4f53-9817-807b8b09a676 service cinder] Authenticating user token {{(pid=81408) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:21:19.552397 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fecec67e-ea41-4f53-9817-807b8b09a676 service cinder] Validating token access rules against request {{(pid=81408) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:21:19.569842 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8bb0e1b7-752c-4624-ab16-cabc7e25f74c service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6141b8a6c3d54e179d6879bb019f4d12', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dcec0431fb6849499d6aec35a234daa6', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81408) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:21:19.570887 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8bb0e1b7-752c-4624-ab16-cabc7e25f74c service cinder] REQUEST_METHOD: `GET` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:21:19.571166 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8bb0e1b7-752c-4624-ab16-cabc7e25f74c service cinder] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:21:19.572694 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8bb0e1b7-752c-4624-ab16-cabc7e25f74c service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:21:19.643414 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG dbcounter [-] [81408] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81408) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:21:19.711595 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e2db52e-a0a7-441c-8eed-1f0eafb3b111 None None] REQUEST_METHOD: `POST` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:21:19.711773 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e2db52e-a0a7-441c-8eed-1f0eafb3b111 None None] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:21:19.711947 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e2db52e-a0a7-441c-8eed-1f0eafb3b111 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:21:19.786308 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.auth.core [None req-0e2db52e-a0a7-441c-8eed-1f0eafb3b111 None None] MFA Rules not processed for user `35a058c849104a9699ecfcd0c81d398e`. Rule list: `[]` (Enabled: `True`). {{(pid=81409) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 13 04:21:19.786308 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.fernet_utils [None req-0e2db52e-a0a7-441c-8eed-1f0eafb3b111 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81409) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:21:19.845427 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8bb0e1b7-752c-4624-ab16-cabc7e25f74c service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:21:19.884682 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8bb0e1b7-752c-4624-ab16-cabc7e25f74c service cinder] RBAC: Authorization granted {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:21:19.971253 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 711/1438] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Oct 13 04:21:19 2022] GET /identity/v3/auth/tokens => generated 3278 bytes in 429 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 13 04:21:19.984371 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 728/1439] 10.222.0.27 () {58 vars in 958 bytes} [Thu Oct 13 04:21:19 2022] POST /identity/v3/auth/tokens => generated 3266 bytes in 289 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 13 04:21:21.124437 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11e36a43-e86e-440c-a107-fa78425bf0ef None None] REQUEST_METHOD: `POST` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:21:21.125009 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11e36a43-e86e-440c-a107-fa78425bf0ef None None] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:21:21.125009 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11e36a43-e86e-440c-a107-fa78425bf0ef None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:21:21.170303 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.auth.core [None req-11e36a43-e86e-440c-a107-fa78425bf0ef None None] MFA Rules not processed for user `1f6cf725908041ccaceb1ffa694ad161`. Rule list: `[]` (Enabled: `True`). {{(pid=81408) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 13 04:21:21.176897 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.fernet_utils [None req-11e36a43-e86e-440c-a107-fa78425bf0ef None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81408) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:21:21.360738 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 712/1440] 10.222.0.27 () {58 vars in 958 bytes} [Thu Oct 13 04:21:21 2022] POST /identity/v3/auth/tokens => generated 3254 bytes in 237 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 13 04:21:22.474815 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-377e9f37-7f41-41e1-967c-c96f53f9e6fb None None] REQUEST_METHOD: `POST` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:21:22.475838 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-377e9f37-7f41-41e1-967c-c96f53f9e6fb None None] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:21:22.476412 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-377e9f37-7f41-41e1-967c-c96f53f9e6fb None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:21:22.489722 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG dbcounter [-] [81409] Writing DB stats keystone:SELECT=100 {{(pid=81409) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:21:22.557780 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.auth.core [None req-377e9f37-7f41-41e1-967c-c96f53f9e6fb None None] MFA Rules not processed for user `d8812c3974f84871bb2d761cf0016257`. Rule list: `[]` (Enabled: `True`). {{(pid=81409) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 13 04:21:22.559025 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.fernet_utils [None req-377e9f37-7f41-41e1-967c-c96f53f9e6fb None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81409) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:21:22.750947 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 729/1441] 10.222.0.36 () {58 vars in 1015 bytes} [Thu Oct 13 04:21:22 2022] POST /identity/v3/auth/tokens => generated 3121 bytes in 277 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 13 04:21:22.833895 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-11e36a43-e86e-440c-a107-fa78425bf0ef None None] Authenticating user token {{(pid=81408) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:21:22.836559 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.fernet_utils [None req-11e36a43-e86e-440c-a107-fa78425bf0ef None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81408) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:21:22.860379 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-11e36a43-e86e-440c-a107-fa78425bf0ef None None] Validating token access rules against request {{(pid=81408) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:21:22.865170 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-723d1191-2d1d-478a-ae92-12a27c17db60 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd8812c3974f84871bb2d761cf0016257', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dcec0431fb6849499d6aec35a234daa6', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81408) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:21:22.866162 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-723d1191-2d1d-478a-ae92-12a27c17db60 service neutron] REQUEST_METHOD: `GET` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:21:22.866881 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-723d1191-2d1d-478a-ae92-12a27c17db60 service neutron] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:21:22.867445 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-723d1191-2d1d-478a-ae92-12a27c17db60 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:21:22.880447 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-723d1191-2d1d-478a-ae92-12a27c17db60 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:21:22.883635 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-723d1191-2d1d-478a-ae92-12a27c17db60 service neutron] RBAC: Authorization granted {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:21:22.899252 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 713/1442] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Oct 13 04:21:22 2022] GET /identity/v3/auth/tokens => generated 3121 bytes in 66 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 13 04:21:23.409262 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3fe7428-17f4-4a8c-98d7-6ecb31556a8e None None] REQUEST_METHOD: `GET` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:21:23.409682 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3fe7428-17f4-4a8c-98d7-6ecb31556a8e None None] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:21:23.410005 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3fe7428-17f4-4a8c-98d7-6ecb31556a8e None None] PATH_INFO: `/` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:21:23.412123 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 730/1443] 10.222.0.27 () {54 vars in 921 bytes} [Thu Oct 13 04:21:23 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 13 04:21:23.462194 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c46a3ee4-b890-4072-8bd7-e6c768d54ee0 None None] REQUEST_METHOD: `POST` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:21:23.462515 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c46a3ee4-b890-4072-8bd7-e6c768d54ee0 None None] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:21:23.463345 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c46a3ee4-b890-4072-8bd7-e6c768d54ee0 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:21:23.514412 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.auth.core [None req-c46a3ee4-b890-4072-8bd7-e6c768d54ee0 None None] MFA Rules not processed for user `ba1b6e5868a549a99760ebe6068fd6b9`. Rule list: `[]` (Enabled: `True`). {{(pid=81408) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 13 04:21:23.516556 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.fernet_utils [None req-c46a3ee4-b890-4072-8bd7-e6c768d54ee0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81408) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:21:23.693224 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d3fe7428-17f4-4a8c-98d7-6ecb31556a8e None None] Authenticating user token {{(pid=81409) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:21:23.695392 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.fernet_utils [None req-d3fe7428-17f4-4a8c-98d7-6ecb31556a8e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81409) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:21:23.809088 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 714/1444] 10.222.0.27 () {58 vars in 1018 bytes} [Thu Oct 13 04:21:23 2022] POST /identity/v3/auth/tokens => generated 3308 bytes in 345 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 13 04:21:23.968810 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d3fe7428-17f4-4a8c-98d7-6ecb31556a8e None None] Validating token access rules against request {{(pid=81409) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:21:23.978927 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d8e16574-0ef2-43e5-8df2-29f523467d6f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8ba823f313144da3b683bacef8fd62e0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81409) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:21:23.979593 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8e16574-0ef2-43e5-8df2-29f523467d6f admin admin] REQUEST_METHOD: `DELETE` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:21:23.979866 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8e16574-0ef2-43e5-8df2-29f523467d6f admin admin] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:21:23.980136 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8e16574-0ef2-43e5-8df2-29f523467d6f admin admin] PATH_INFO: `/v3/users/51b364bcf10d45ec8e1871fa09f8de1f` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:21:23.984614 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d8e16574-0ef2-43e5-8df2-29f523467d6f admin admin] RBAC: Authorizing `identity:delete_user(user_id=51b364bcf10d45ec8e1871fa09f8de1f)` {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:21:23.988229 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d8e16574-0ef2-43e5-8df2-29f523467d6f admin admin] RBAC: Authorization granted {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:21:24.312684 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c46a3ee4-b890-4072-8bd7-e6c768d54ee0 None None] Authenticating user token {{(pid=81408) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:21:24.314239 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.fernet_utils [None req-c46a3ee4-b890-4072-8bd7-e6c768d54ee0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81408) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:21:24.354498 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c46a3ee4-b890-4072-8bd7-e6c768d54ee0 None None] Validating token access rules against request {{(pid=81408) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:21:24.389668 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f0105014-7b3f-406c-9107-671777b1f171 service placement] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ba1b6e5868a549a99760ebe6068fd6b9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dcec0431fb6849499d6aec35a234daa6', 'project_domain_id': 'default', 'roles': ['service', 'member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81408) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:21:24.393041 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0105014-7b3f-406c-9107-671777b1f171 service placement] REQUEST_METHOD: `GET` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:21:24.393321 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0105014-7b3f-406c-9107-671777b1f171 service placement] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:21:24.393594 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0105014-7b3f-406c-9107-671777b1f171 service placement] PATH_INFO: `/v3/auth/tokens` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:21:24.452637 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f0105014-7b3f-406c-9107-671777b1f171 service placement] RBAC: Authorizing `identity:validate_token()` {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:21:24.475905 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f0105014-7b3f-406c-9107-671777b1f171 service placement] RBAC: Authorization granted {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:21:24.552656 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 715/1445] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Oct 13 04:21:24 2022] GET /identity/v3/auth/tokens => generated 3308 bytes in 240 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Oct 13 04:21:24.557999 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.notifications [None req-d8e16574-0ef2-43e5-8df2-29f523467d6f admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '51b364bcf10d45ec8e1871fa09f8de1f'} {{(pid=81409) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 13 04:21:24.558735 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.notifications [None req-d8e16574-0ef2-43e5-8df2-29f523467d6f admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '51b364bcf10d45ec8e1871fa09f8de1f'} {{(pid=81409) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 13 04:21:24.584280 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.notifications [None req-d8e16574-0ef2-43e5-8df2-29f523467d6f admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '51b364bcf10d45ec8e1871fa09f8de1f'} {{(pid=81409) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 13 04:21:24.585028 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.notifications [None req-d8e16574-0ef2-43e5-8df2-29f523467d6f admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '51b364bcf10d45ec8e1871fa09f8de1f'} {{(pid=81409) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 13 04:21:24.608629 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.notifications [None req-d8e16574-0ef2-43e5-8df2-29f523467d6f admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '51b364bcf10d45ec8e1871fa09f8de1f'} {{(pid=81409) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 13 04:21:24.661743 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 731/1446] 10.222.0.27 () {58 vars in 1224 bytes} [Thu Oct 13 04:21:23 2022] DELETE /identity/v3/users/51b364bcf10d45ec8e1871fa09f8de1f => generated 0 bytes in 970 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 13 04:21:24.667651 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f4c5b1b-96a3-4037-babf-ef5573dbe29b None None] REQUEST_METHOD: `POST` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:21:24.667934 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f4c5b1b-96a3-4037-babf-ef5573dbe29b None None] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:21:24.668208 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f4c5b1b-96a3-4037-babf-ef5573dbe29b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:21:24.730770 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.auth.core [None req-0f4c5b1b-96a3-4037-babf-ef5573dbe29b None None] MFA Rules not processed for user `0e2d979563d6421995451a0e86bf3022`. Rule list: `[]` (Enabled: `True`). {{(pid=81408) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 13 04:21:24.734757 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.fernet_utils [None req-0f4c5b1b-96a3-4037-babf-ef5573dbe29b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81408) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:21:24.824493 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG dbcounter [-] [81408] Writing DB stats keystone:SELECT=100 {{(pid=81408) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:21:24.965896 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 716/1447] 10.222.0.27 () {58 vars in 958 bytes} [Thu Oct 13 04:21:24 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 300 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 13 04:21:25.157954 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d8e16574-0ef2-43e5-8df2-29f523467d6f admin admin] Authenticating user token {{(pid=81409) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:21:25.161685 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.fernet_utils [None req-d8e16574-0ef2-43e5-8df2-29f523467d6f admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81409) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:21:25.323377 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG dbcounter [-] [81409] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81409) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:21:25.443531 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d8e16574-0ef2-43e5-8df2-29f523467d6f admin admin] Validating token access rules against request {{(pid=81409) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:21:25.456009 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7669b64b-7fac-48ce-8919-c516232daae1 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd8812c3974f84871bb2d761cf0016257', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dcec0431fb6849499d6aec35a234daa6', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81409) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:21:25.460014 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7669b64b-7fac-48ce-8919-c516232daae1 service neutron] REQUEST_METHOD: `GET` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:21:25.461624 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7669b64b-7fac-48ce-8919-c516232daae1 service neutron] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:21:25.462034 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7669b64b-7fac-48ce-8919-c516232daae1 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:21:25.500506 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7669b64b-7fac-48ce-8919-c516232daae1 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:21:25.504940 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7669b64b-7fac-48ce-8919-c516232daae1 service neutron] RBAC: Authorization granted {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:21:25.567477 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 732/1448] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Oct 13 04:21:25 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 413 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 13 04:21:26.151471 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0f4c5b1b-96a3-4037-babf-ef5573dbe29b None None] Authenticating user token {{(pid=81408) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:21:26.154687 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.fernet_utils [None req-0f4c5b1b-96a3-4037-babf-ef5573dbe29b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81408) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:21:26.191861 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0f4c5b1b-96a3-4037-babf-ef5573dbe29b None None] Validating token access rules against request {{(pid=81408) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:21:26.208498 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-47044051-8b4d-4c0e-9934-4a187e9d09c2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8ba823f313144da3b683bacef8fd62e0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81408) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:21:26.209776 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47044051-8b4d-4c0e-9934-4a187e9d09c2 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:21:26.210321 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47044051-8b4d-4c0e-9934-4a187e9d09c2 admin admin] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:21:26.210783 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47044051-8b4d-4c0e-9934-4a187e9d09c2 admin admin] PATH_INFO: `/v3/users/6372398a35be42acba1b1822905e7a50` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:21:26.212706 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-47044051-8b4d-4c0e-9934-4a187e9d09c2 admin admin] RBAC: Authorizing `identity:delete_user(user_id=6372398a35be42acba1b1822905e7a50)` {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:21:26.215409 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-47044051-8b4d-4c0e-9934-4a187e9d09c2 admin admin] RBAC: Authorization granted {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:21:26.356428 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.notifications [None req-47044051-8b4d-4c0e-9934-4a187e9d09c2 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '6372398a35be42acba1b1822905e7a50'} {{(pid=81408) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 13 04:21:26.377511 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.notifications [None req-47044051-8b4d-4c0e-9934-4a187e9d09c2 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '6372398a35be42acba1b1822905e7a50'} {{(pid=81408) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 13 04:21:26.380526 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.notifications [None req-47044051-8b4d-4c0e-9934-4a187e9d09c2 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '6372398a35be42acba1b1822905e7a50'} {{(pid=81408) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 13 04:21:26.382065 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.notifications [None req-47044051-8b4d-4c0e-9934-4a187e9d09c2 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '6372398a35be42acba1b1822905e7a50'} {{(pid=81408) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 13 04:21:26.402669 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.notifications [None req-47044051-8b4d-4c0e-9934-4a187e9d09c2 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '6372398a35be42acba1b1822905e7a50'} {{(pid=81408) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 13 04:21:26.477648 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 717/1449] 10.222.0.27 () {58 vars in 1224 bytes} [Thu Oct 13 04:21:26 2022] DELETE /identity/v3/users/6372398a35be42acba1b1822905e7a50 => generated 0 bytes in 324 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 13 04:21:26.480708 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f94b631-9d56-4a6e-8141-08f2a27a0964 None None] REQUEST_METHOD: `POST` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:21:26.481491 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f94b631-9d56-4a6e-8141-08f2a27a0964 None None] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:21:26.481748 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f94b631-9d56-4a6e-8141-08f2a27a0964 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:21:26.531019 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.auth.core [None req-3f94b631-9d56-4a6e-8141-08f2a27a0964 None None] MFA Rules not processed for user `0e2d979563d6421995451a0e86bf3022`. Rule list: `[]` (Enabled: `True`). {{(pid=81409) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 13 04:21:26.535584 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.fernet_utils [None req-3f94b631-9d56-4a6e-8141-08f2a27a0964 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81409) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:21:26.711579 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 733/1450] 10.222.0.27 () {58 vars in 958 bytes} [Thu Oct 13 04:21:26 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 232 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 13 04:21:26.827132 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-47044051-8b4d-4c0e-9934-4a187e9d09c2 admin admin] Authenticating user token {{(pid=81408) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:21:26.829525 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.fernet_utils [None req-47044051-8b4d-4c0e-9934-4a187e9d09c2 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81408) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:21:26.987246 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-47044051-8b4d-4c0e-9934-4a187e9d09c2 admin admin] Validating token access rules against request {{(pid=81408) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:21:27.000975 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6d30e604-f943-40e8-912c-c89ccd87a4d9 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd8812c3974f84871bb2d761cf0016257', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dcec0431fb6849499d6aec35a234daa6', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81408) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:21:27.001880 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d30e604-f943-40e8-912c-c89ccd87a4d9 service neutron] REQUEST_METHOD: `GET` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:21:27.007436 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d30e604-f943-40e8-912c-c89ccd87a4d9 service neutron] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:21:27.007949 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d30e604-f943-40e8-912c-c89ccd87a4d9 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:21:27.027801 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6d30e604-f943-40e8-912c-c89ccd87a4d9 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:21:27.032372 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6d30e604-f943-40e8-912c-c89ccd87a4d9 service neutron] RBAC: Authorization granted {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:21:27.059248 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 718/1451] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Oct 13 04:21:26 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 233 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 13 04:21:27.943154 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6fb07fb2-4eb6-4f90-973e-e03fe29d0876 None None] REQUEST_METHOD: `GET` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:21:27.944104 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6fb07fb2-4eb6-4f90-973e-e03fe29d0876 None None] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:21:27.944520 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6fb07fb2-4eb6-4f90-973e-e03fe29d0876 None None] PATH_INFO: `/` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:21:27.946750 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 734/1452] 10.222.0.27 () {54 vars in 921 bytes} [Thu Oct 13 04:21:27 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 13 04:21:28.030406 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-799b90e3-4ae6-40d0-b3fc-ca4cccda469c None None] REQUEST_METHOD: `POST` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:21:28.030997 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-799b90e3-4ae6-40d0-b3fc-ca4cccda469c None None] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:21:28.031997 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-799b90e3-4ae6-40d0-b3fc-ca4cccda469c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:21:28.094498 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.auth.core [None req-799b90e3-4ae6-40d0-b3fc-ca4cccda469c None None] MFA Rules not processed for user `cb740ed7f7684b079e45f887be83f795`. Rule list: `[]` (Enabled: `True`). {{(pid=81408) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 13 04:21:28.096264 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.fernet_utils [None req-799b90e3-4ae6-40d0-b3fc-ca4cccda469c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81408) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:21:28.228680 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG dbcounter [-] [81408] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81408) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:21:28.284168 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 719/1453] 10.222.0.27 () {58 vars in 1018 bytes} [Thu Oct 13 04:21:28 2022] POST /identity/v3/auth/tokens => generated 3372 bytes in 255 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 13 04:21:29.224004 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-061228d0-2657-4568-9b89-f00366b27370 None None] REQUEST_METHOD: `GET` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:21:29.225110 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-061228d0-2657-4568-9b89-f00366b27370 None None] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:21:29.225560 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-061228d0-2657-4568-9b89-f00366b27370 None None] PATH_INFO: `/` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:21:29.227257 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 735/1454] 10.222.0.27 () {54 vars in 921 bytes} [Thu Oct 13 04:21:29 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 13 04:21:29.277746 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7961fe00-f650-4b85-bdd0-20f1efb88196 None None] REQUEST_METHOD: `POST` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:21:29.278436 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7961fe00-f650-4b85-bdd0-20f1efb88196 None None] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:21:29.278436 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7961fe00-f650-4b85-bdd0-20f1efb88196 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:21:29.352920 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.auth.core [None req-7961fe00-f650-4b85-bdd0-20f1efb88196 None None] MFA Rules not processed for user `ba1b6e5868a549a99760ebe6068fd6b9`. Rule list: `[]` (Enabled: `True`). {{(pid=81408) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 13 04:21:29.360500 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.fernet_utils [None req-7961fe00-f650-4b85-bdd0-20f1efb88196 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81408) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:21:29.557646 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-061228d0-2657-4568-9b89-f00366b27370 None None] Authenticating user token {{(pid=81409) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:21:29.559738 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.fernet_utils [None req-061228d0-2657-4568-9b89-f00366b27370 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81409) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:21:29.606721 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 720/1455] 10.222.0.27 () {58 vars in 1018 bytes} [Thu Oct 13 04:21:29 2022] POST /identity/v3/auth/tokens => generated 3308 bytes in 330 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 13 04:21:29.612640 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-061228d0-2657-4568-9b89-f00366b27370 None None] Validating token access rules against request {{(pid=81409) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:21:29.619268 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-022a7474-3705-43d2-9199-2065f65f41fa admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8ba823f313144da3b683bacef8fd62e0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81409) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:21:29.621171 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-022a7474-3705-43d2-9199-2065f65f41fa admin admin] REQUEST_METHOD: `DELETE` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:21:29.622014 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-022a7474-3705-43d2-9199-2065f65f41fa admin admin] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:21:29.623305 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-022a7474-3705-43d2-9199-2065f65f41fa admin admin] PATH_INFO: `/v3/users/a07e8e6b4e0d48fa9018fd1b8a7c5f92` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:21:29.625148 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-022a7474-3705-43d2-9199-2065f65f41fa admin admin] RBAC: Authorizing `identity:delete_user(user_id=a07e8e6b4e0d48fa9018fd1b8a7c5f92)` {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:21:29.627019 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7961fe00-f650-4b85-bdd0-20f1efb88196 None None] Authenticating user token {{(pid=81408) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:21:29.628497 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-022a7474-3705-43d2-9199-2065f65f41fa admin admin] RBAC: Authorization granted {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:21:29.629207 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.fernet_utils [None req-7961fe00-f650-4b85-bdd0-20f1efb88196 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81408) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:21:29.656685 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7961fe00-f650-4b85-bdd0-20f1efb88196 None None] Validating token access rules against request {{(pid=81408) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:21:29.672177 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6b081928-4d56-48a4-b331-9774407b5bdb admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8ba823f313144da3b683bacef8fd62e0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81408) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:21:29.672905 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b081928-4d56-48a4-b331-9774407b5bdb admin admin] REQUEST_METHOD: `DELETE` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:21:29.673214 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b081928-4d56-48a4-b331-9774407b5bdb admin admin] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:21:29.673784 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b081928-4d56-48a4-b331-9774407b5bdb admin admin] PATH_INFO: `/v3/projects/6592493745484100afa26c249568c269` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:21:29.675115 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6b081928-4d56-48a4-b331-9774407b5bdb admin admin] RBAC: Authorizing `identity:delete_project(project_id=6592493745484100afa26c249568c269)` {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:21:29.678934 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6b081928-4d56-48a4-b331-9774407b5bdb admin admin] RBAC: Authorization granted {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:21:29.719632 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.notifications [None req-022a7474-3705-43d2-9199-2065f65f41fa admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'a07e8e6b4e0d48fa9018fd1b8a7c5f92'} {{(pid=81409) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 13 04:21:29.729320 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.notifications [None req-022a7474-3705-43d2-9199-2065f65f41fa admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'a07e8e6b4e0d48fa9018fd1b8a7c5f92'} {{(pid=81409) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 13 04:21:29.746966 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.notifications [None req-022a7474-3705-43d2-9199-2065f65f41fa admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'a07e8e6b4e0d48fa9018fd1b8a7c5f92'} {{(pid=81409) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 13 04:21:29.749748 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.notifications [None req-022a7474-3705-43d2-9199-2065f65f41fa admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'a07e8e6b4e0d48fa9018fd1b8a7c5f92'} {{(pid=81409) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 13 04:21:29.766894 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.notifications [None req-022a7474-3705-43d2-9199-2065f65f41fa admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'a07e8e6b4e0d48fa9018fd1b8a7c5f92'} {{(pid=81409) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 13 04:21:29.822007 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 736/1456] 10.222.0.27 () {58 vars in 1224 bytes} [Thu Oct 13 04:21:29 2022] DELETE /identity/v3/users/a07e8e6b4e0d48fa9018fd1b8a7c5f92 => generated 0 bytes in 261 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 13 04:21:29.826304 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4aa6b19-154e-48e9-9112-6e518b98f171 None None] REQUEST_METHOD: `GET` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:21:29.827319 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4aa6b19-154e-48e9-9112-6e518b98f171 None None] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:21:29.828554 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4aa6b19-154e-48e9-9112-6e518b98f171 None None] PATH_INFO: `/` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:21:29.832449 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 737/1457] 10.222.0.27 () {54 vars in 961 bytes} [Thu Oct 13 04:21:29 2022] GET /identity => generated 270 bytes in 10 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 13 04:21:29.841026 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e58e1bd1-d4c1-4e51-a752-1e1d44146e2d None None] REQUEST_METHOD: `POST` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:21:29.842714 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e58e1bd1-d4c1-4e51-a752-1e1d44146e2d None None] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:21:29.844254 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e58e1bd1-d4c1-4e51-a752-1e1d44146e2d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:21:29.854471 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.notifications [None req-6b081928-4d56-48a4-b331-9774407b5bdb admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '6592493745484100afa26c249568c269', 'request_id': 'req-6b081928-4d56-48a4-b331-9774407b5bdb'} {{(pid=81408) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 13 04:21:29.894769 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.notifications [None req-6b081928-4d56-48a4-b331-9774407b5bdb admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '6592493745484100afa26c249568c269', 'request_id': 'req-6b081928-4d56-48a4-b331-9774407b5bdb'} {{(pid=81408) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 13 04:21:29.904878 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.auth.core [None req-e58e1bd1-d4c1-4e51-a752-1e1d44146e2d None None] MFA Rules not processed for user `ba1b6e5868a549a99760ebe6068fd6b9`. Rule list: `[]` (Enabled: `True`). {{(pid=81409) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 13 04:21:29.906314 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.notifications [None req-6b081928-4d56-48a4-b331-9774407b5bdb admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '6592493745484100afa26c249568c269', 'request_id': 'req-6b081928-4d56-48a4-b331-9774407b5bdb'} {{(pid=81408) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 13 04:21:29.908734 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.fernet_utils [None req-e58e1bd1-d4c1-4e51-a752-1e1d44146e2d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81409) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:21:29.936584 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.notifications [None req-6b081928-4d56-48a4-b331-9774407b5bdb admin admin] The token cache is being invalidate because project 6592493745484100afa26c249568c269 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81408) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 13 04:21:29.936794 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.notifications [None req-6b081928-4d56-48a4-b331-9774407b5bdb admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81408) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 13 04:21:29.946163 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.notifications [None req-6b081928-4d56-48a4-b331-9774407b5bdb admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81408) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 13 04:21:29.961849 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 721/1458] 10.222.0.27 () {58 vars in 1233 bytes} [Thu Oct 13 04:21:29 2022] DELETE /identity/v3/projects/6592493745484100afa26c249568c269 => generated 0 bytes in 330 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 13 04:21:29.963774 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8bd132ff-23ba-4ede-b818-a9d89c17cb18 None None] REQUEST_METHOD: `POST` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:21:29.964047 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8bd132ff-23ba-4ede-b818-a9d89c17cb18 None None] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:21:29.964326 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8bd132ff-23ba-4ede-b818-a9d89c17cb18 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:21:30.034288 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG dbcounter [-] [81409] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81409) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:21:30.043497 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.auth.core [None req-8bd132ff-23ba-4ede-b818-a9d89c17cb18 None None] MFA Rules not processed for user `0e2d979563d6421995451a0e86bf3022`. Rule list: `[]` (Enabled: `True`). {{(pid=81408) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 13 04:21:30.045992 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.fernet_utils [None req-8bd132ff-23ba-4ede-b818-a9d89c17cb18 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81408) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:21:30.114999 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 738/1459] 10.222.0.27 () {58 vars in 1058 bytes} [Thu Oct 13 04:21:29 2022] POST /identity/v3/auth/tokens => generated 3308 bytes in 276 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 13 04:21:30.116245 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e58e1bd1-d4c1-4e51-a752-1e1d44146e2d None None] Authenticating user token {{(pid=81409) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:21:30.117803 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.fernet_utils [None req-e58e1bd1-d4c1-4e51-a752-1e1d44146e2d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81409) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:21:30.251935 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG dbcounter [-] [81408] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=81408) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:21:30.290004 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 722/1460] 10.222.0.27 () {58 vars in 958 bytes} [Thu Oct 13 04:21:29 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 328 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 13 04:21:30.298407 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8bd132ff-23ba-4ede-b818-a9d89c17cb18 None None] Authenticating user token {{(pid=81408) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:21:30.302963 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e58e1bd1-d4c1-4e51-a752-1e1d44146e2d None None] Validating token access rules against request {{(pid=81409) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:21:30.321744 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-da79e37a-07ed-401c-8809-fec317b4d425 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cb740ed7f7684b079e45f887be83f795', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dcec0431fb6849499d6aec35a234daa6', 'project_domain_id': 'default', 'roles': ['ResellerAdmin', 'admin', 'reader', 'service', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81409) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:21:30.322729 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da79e37a-07ed-401c-8809-fec317b4d425 service nova] REQUEST_METHOD: `GET` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:21:30.325332 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da79e37a-07ed-401c-8809-fec317b4d425 service nova] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:21:30.325742 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da79e37a-07ed-401c-8809-fec317b4d425 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:21:30.330035 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.fernet_utils [None req-da79e37a-07ed-401c-8809-fec317b4d425 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81409) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:21:30.338814 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8bd132ff-23ba-4ede-b818-a9d89c17cb18 None None] Validating token access rules against request {{(pid=81408) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:21:30.357416 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-591c317c-2081-461e-a569-737e4d22760b service placement] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ba1b6e5868a549a99760ebe6068fd6b9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dcec0431fb6849499d6aec35a234daa6', 'project_domain_id': 'default', 'roles': ['service', 'member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81408) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:21:30.359765 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-591c317c-2081-461e-a569-737e4d22760b service placement] REQUEST_METHOD: `GET` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:21:30.362396 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-591c317c-2081-461e-a569-737e4d22760b service placement] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:21:30.363026 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-591c317c-2081-461e-a569-737e4d22760b service placement] PATH_INFO: `/v3/auth/tokens` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:21:30.366639 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-da79e37a-07ed-401c-8809-fec317b4d425 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:21:30.369244 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.fernet_utils [None req-591c317c-2081-461e-a569-737e4d22760b service placement] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81408) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:21:30.370687 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-da79e37a-07ed-401c-8809-fec317b4d425 service nova] RBAC: Authorization granted {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:21:30.405781 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-591c317c-2081-461e-a569-737e4d22760b service placement] RBAC: Authorizing `identity:validate_token()` {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:21:30.411418 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-591c317c-2081-461e-a569-737e4d22760b service placement] RBAC: Authorization granted {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:21:30.416969 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 739/1461] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Oct 13 04:21:30 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 301 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 13 04:21:30.423577 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d64e133b-997c-418d-af8c-5aafd1aa6379 None None] REQUEST_METHOD: `POST` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:21:30.426634 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d64e133b-997c-418d-af8c-5aafd1aa6379 None None] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:21:30.427075 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d64e133b-997c-418d-af8c-5aafd1aa6379 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:21:30.440927 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 723/1462] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Oct 13 04:21:30 2022] GET /identity/v3/auth/tokens => generated 3308 bytes in 143 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 13 04:21:30.505826 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.auth.core [None req-d64e133b-997c-418d-af8c-5aafd1aa6379 None None] MFA Rules not processed for user `0e2d979563d6421995451a0e86bf3022`. Rule list: `[]` (Enabled: `True`). {{(pid=81409) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 13 04:21:30.508206 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.fernet_utils [None req-d64e133b-997c-418d-af8c-5aafd1aa6379 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81409) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:21:30.540418 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 740/1463] 10.222.0.27 () {58 vars in 958 bytes} [Thu Oct 13 04:21:30 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 122 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 13 04:21:30.541189 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-591c317c-2081-461e-a569-737e4d22760b service placement] Authenticating user token {{(pid=81408) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:21:30.558733 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-591c317c-2081-461e-a569-737e4d22760b service placement] Validating token access rules against request {{(pid=81408) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:21:30.571287 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9966700c-dedb-4f83-8deb-e2c683539487 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8ba823f313144da3b683bacef8fd62e0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81408) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:21:30.572749 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9966700c-dedb-4f83-8deb-e2c683539487 admin admin] REQUEST_METHOD: `GET` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:21:30.574566 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9966700c-dedb-4f83-8deb-e2c683539487 admin admin] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:21:30.575017 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9966700c-dedb-4f83-8deb-e2c683539487 admin admin] PATH_INFO: `/v3/domains` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:21:30.575946 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9966700c-dedb-4f83-8deb-e2c683539487 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81408) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 13 04:21:30.578018 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9966700c-dedb-4f83-8deb-e2c683539487 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:21:30.584292 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9966700c-dedb-4f83-8deb-e2c683539487 admin admin] RBAC: Authorization granted {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:21:30.624105 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d64e133b-997c-418d-af8c-5aafd1aa6379 None None] Authenticating user token {{(pid=81409) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:21:30.632080 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 724/1464] 10.222.0.27 () {58 vars in 1153 bytes} [Thu Oct 13 04:21:30 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 91 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 13 04:21:30.637767 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d64e133b-997c-418d-af8c-5aafd1aa6379 None None] Validating token access rules against request {{(pid=81409) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:21:30.660329 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-87e9ab6a-b78f-44fb-ae15-5ee5759b7a28 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8ba823f313144da3b683bacef8fd62e0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81409) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:21:30.661254 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87e9ab6a-b78f-44fb-ae15-5ee5759b7a28 admin admin] REQUEST_METHOD: `POST` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:21:30.661678 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87e9ab6a-b78f-44fb-ae15-5ee5759b7a28 admin admin] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:21:30.662115 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87e9ab6a-b78f-44fb-ae15-5ee5759b7a28 admin admin] PATH_INFO: `/v3/projects` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:21:30.663067 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-87e9ab6a-b78f-44fb-ae15-5ee5759b7a28 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:21:30.665169 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-87e9ab6a-b78f-44fb-ae15-5ee5759b7a28 admin admin] RBAC: Authorization granted {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:21:30.757954 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9966700c-dedb-4f83-8deb-e2c683539487 admin admin] Authenticating user token {{(pid=81408) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:21:30.759132 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 741/1465] 10.222.0.27 () {60 vars in 1153 bytes} [Thu Oct 13 04:21:30 2022] POST /identity/v3/projects => generated 389 bytes in 135 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 13 04:21:30.781211 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9966700c-dedb-4f83-8deb-e2c683539487 admin admin] Validating token access rules against request {{(pid=81408) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:21:30.979410 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-490064b1-3329-43aa-9299-ab9f0d6478e7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8ba823f313144da3b683bacef8fd62e0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81408) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:21:30.987351 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-490064b1-3329-43aa-9299-ab9f0d6478e7 admin admin] REQUEST_METHOD: `POST` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:21:30.987795 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-490064b1-3329-43aa-9299-ab9f0d6478e7 admin admin] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:21:30.988215 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-490064b1-3329-43aa-9299-ab9f0d6478e7 admin admin] PATH_INFO: `/v3/users` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:21:30.989021 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-490064b1-3329-43aa-9299-ab9f0d6478e7 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:21:30.991157 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-490064b1-3329-43aa-9299-ab9f0d6478e7 admin admin] RBAC: Authorization granted {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:21:31.065343 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 725/1466] 10.222.0.27 () {60 vars in 1144 bytes} [Thu Oct 13 04:21:30 2022] POST /identity/v3/users => generated 295 bytes in 308 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 13 04:21:31.069924 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-87e9ab6a-b78f-44fb-ae15-5ee5759b7a28 admin admin] Authenticating user token {{(pid=81409) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:21:31.100565 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-87e9ab6a-b78f-44fb-ae15-5ee5759b7a28 admin admin] Validating token access rules against request {{(pid=81409) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:21:31.116692 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-490064b1-3329-43aa-9299-ab9f0d6478e7 admin admin] Authenticating user token {{(pid=81408) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:21:31.119411 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.fernet_utils [None req-490064b1-3329-43aa-9299-ab9f0d6478e7 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81408) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:21:31.124885 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0962187a-ca5a-45ae-911c-5fc9dd9d010d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8ba823f313144da3b683bacef8fd62e0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81409) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:21:31.127776 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0962187a-ca5a-45ae-911c-5fc9dd9d010d admin admin] REQUEST_METHOD: `GET` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:21:31.134893 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0962187a-ca5a-45ae-911c-5fc9dd9d010d admin admin] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:21:31.140212 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0962187a-ca5a-45ae-911c-5fc9dd9d010d admin admin] PATH_INFO: `/v3/roles` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:21:31.143370 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0962187a-ca5a-45ae-911c-5fc9dd9d010d admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:21:31.145765 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0962187a-ca5a-45ae-911c-5fc9dd9d010d admin admin] RBAC: Authorization granted {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:21:31.173098 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-490064b1-3329-43aa-9299-ab9f0d6478e7 admin admin] Validating token access rules against request {{(pid=81408) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:21:31.178388 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 742/1467] 10.222.0.27 () {58 vars in 1122 bytes} [Thu Oct 13 04:21:31 2022] GET /identity/v3/roles => generated 1420 bytes in 109 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 13 04:21:31.188328 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0962187a-ca5a-45ae-911c-5fc9dd9d010d admin admin] Authenticating user token {{(pid=81409) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:21:31.194270 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-970d8050-d4c9-41b3-87d1-52e96816c567 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8ba823f313144da3b683bacef8fd62e0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81408) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:21:31.195419 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-970d8050-d4c9-41b3-87d1-52e96816c567 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:21:31.195873 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-970d8050-d4c9-41b3-87d1-52e96816c567 admin admin] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:21:31.196332 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-970d8050-d4c9-41b3-87d1-52e96816c567 admin admin] PATH_INFO: `/v3/projects/085767f5d45b40eeb7c213298abe6545` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:21:31.201915 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-970d8050-d4c9-41b3-87d1-52e96816c567 admin admin] RBAC: Authorizing `identity:delete_project(project_id=085767f5d45b40eeb7c213298abe6545)` {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:21:31.205085 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-970d8050-d4c9-41b3-87d1-52e96816c567 admin admin] RBAC: Authorization granted {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:21:31.227045 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0962187a-ca5a-45ae-911c-5fc9dd9d010d admin admin] Validating token access rules against request {{(pid=81409) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:21:31.243586 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fa9b2c12-41ff-4652-ae3c-4dfaf55a54fb admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8ba823f313144da3b683bacef8fd62e0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81409) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:21:31.244485 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa9b2c12-41ff-4652-ae3c-4dfaf55a54fb admin admin] REQUEST_METHOD: `GET` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:21:31.245075 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa9b2c12-41ff-4652-ae3c-4dfaf55a54fb admin admin] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:21:31.247153 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa9b2c12-41ff-4652-ae3c-4dfaf55a54fb admin admin] PATH_INFO: `/v3/roles` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:21:31.248522 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fa9b2c12-41ff-4652-ae3c-4dfaf55a54fb admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:21:31.266248 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fa9b2c12-41ff-4652-ae3c-4dfaf55a54fb admin admin] RBAC: Authorization granted {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:21:31.299631 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 743/1468] 10.222.0.27 () {58 vars in 1122 bytes} [Thu Oct 13 04:21:31 2022] GET /identity/v3/roles => generated 1420 bytes in 113 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 13 04:21:31.300861 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fa9b2c12-41ff-4652-ae3c-4dfaf55a54fb admin admin] Authenticating user token {{(pid=81409) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:21:31.338328 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.notifications [None req-970d8050-d4c9-41b3-87d1-52e96816c567 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '085767f5d45b40eeb7c213298abe6545', 'request_id': 'req-970d8050-d4c9-41b3-87d1-52e96816c567'} {{(pid=81408) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 13 04:21:31.347858 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fa9b2c12-41ff-4652-ae3c-4dfaf55a54fb admin admin] Validating token access rules against request {{(pid=81409) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:21:31.351501 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.notifications [None req-970d8050-d4c9-41b3-87d1-52e96816c567 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '085767f5d45b40eeb7c213298abe6545', 'request_id': 'req-970d8050-d4c9-41b3-87d1-52e96816c567'} {{(pid=81408) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 13 04:21:31.374987 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.notifications [None req-970d8050-d4c9-41b3-87d1-52e96816c567 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '085767f5d45b40eeb7c213298abe6545', 'request_id': 'req-970d8050-d4c9-41b3-87d1-52e96816c567'} {{(pid=81408) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 13 04:21:31.403615 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.notifications [None req-970d8050-d4c9-41b3-87d1-52e96816c567 admin admin] The token cache is being invalidate because project 085767f5d45b40eeb7c213298abe6545 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81408) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 13 04:21:31.403913 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.notifications [None req-970d8050-d4c9-41b3-87d1-52e96816c567 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81408) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 13 04:21:31.412780 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.notifications [None req-970d8050-d4c9-41b3-87d1-52e96816c567 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81408) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 13 04:21:31.427129 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 726/1469] 10.222.0.27 () {58 vars in 1233 bytes} [Thu Oct 13 04:21:31 2022] DELETE /identity/v3/projects/085767f5d45b40eeb7c213298abe6545 => generated 0 bytes in 311 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 13 04:21:31.431092 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-970d8050-d4c9-41b3-87d1-52e96816c567 admin admin] Authenticating user token {{(pid=81408) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:21:31.443153 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.fernet_utils [None req-970d8050-d4c9-41b3-87d1-52e96816c567 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81408) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:21:31.526102 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eb836457-a10f-4c6f-9fbd-cc6a1b6227c7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8ba823f313144da3b683bacef8fd62e0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81409) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:21:31.527762 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb836457-a10f-4c6f-9fbd-cc6a1b6227c7 admin admin] REQUEST_METHOD: `PUT` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:21:31.528270 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb836457-a10f-4c6f-9fbd-cc6a1b6227c7 admin admin] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:21:31.529165 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb836457-a10f-4c6f-9fbd-cc6a1b6227c7 admin admin] PATH_INFO: `/v3/projects/fb96c1d494324ee9a7cbf76664094cee/users/63a5dd2f709d429d82238913f31c8570/roles/cc56c4ae13a7424ea771c15da4b153c1` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:21:31.565151 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eb836457-a10f-4c6f-9fbd-cc6a1b6227c7 admin admin] RBAC: Authorizing `identity:create_grant(project_id=fb96c1d494324ee9a7cbf76664094cee, user_id=63a5dd2f709d429d82238913f31c8570, role_id=cc56c4ae13a7424ea771c15da4b153c1)` {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:21:31.568243 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eb836457-a10f-4c6f-9fbd-cc6a1b6227c7 admin admin] RBAC: Authorization granted {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:21:31.613252 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG dbcounter [-] [81409] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=81409) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:21:31.622564 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 744/1470] 10.222.0.27 () {60 vars in 1483 bytes} [Thu Oct 13 04:21:31 2022] PUT /identity/v3/projects/fb96c1d494324ee9a7cbf76664094cee/users/63a5dd2f709d429d82238913f31c8570/roles/cc56c4ae13a7424ea771c15da4b153c1 => generated 0 bytes in 322 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 13 04:21:31.635071 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG dbcounter [-] [81408] Writing DB stats keystone:SELECT=89,keystone:INSERT=4,keystone:DELETE=7 {{(pid=81408) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:21:31.636236 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54ecb800-c11b-4b2a-b67c-66240e03e974 None None] REQUEST_METHOD: `POST` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:21:31.636602 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54ecb800-c11b-4b2a-b67c-66240e03e974 None None] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:21:31.636886 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54ecb800-c11b-4b2a-b67c-66240e03e974 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:21:31.667671 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-970d8050-d4c9-41b3-87d1-52e96816c567 admin admin] Validating token access rules against request {{(pid=81408) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:21:31.742425 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.auth.core [None req-54ecb800-c11b-4b2a-b67c-66240e03e974 None None] MFA Rules not processed for user `63a5dd2f709d429d82238913f31c8570`. Rule list: `[]` (Enabled: `True`). {{(pid=81409) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 13 04:21:31.766012 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.fernet_utils [None req-54ecb800-c11b-4b2a-b67c-66240e03e974 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81409) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:21:31.932895 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aac43c76-b211-4b17-9cbb-5217feb7726c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8ba823f313144da3b683bacef8fd62e0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81408) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:21:31.934193 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aac43c76-b211-4b17-9cbb-5217feb7726c admin admin] REQUEST_METHOD: `DELETE` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:21:31.934799 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aac43c76-b211-4b17-9cbb-5217feb7726c admin admin] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:21:31.935300 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aac43c76-b211-4b17-9cbb-5217feb7726c admin admin] PATH_INFO: `/v3/users/26c38224278541b3a64848042505ff26` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:21:31.952339 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aac43c76-b211-4b17-9cbb-5217feb7726c admin admin] RBAC: Authorizing `identity:delete_user(user_id=26c38224278541b3a64848042505ff26)` {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:21:31.959013 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aac43c76-b211-4b17-9cbb-5217feb7726c admin admin] RBAC: Authorization granted {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:21:32.066619 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 745/1471] 10.222.0.27 () {58 vars in 958 bytes} [Thu Oct 13 04:21:31 2022] POST /identity/v3/auth/tokens => generated 3266 bytes in 432 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 13 04:21:32.067911 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-54ecb800-c11b-4b2a-b67c-66240e03e974 None None] Authenticating user token {{(pid=81409) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:21:32.076049 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.fernet_utils [None req-54ecb800-c11b-4b2a-b67c-66240e03e974 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81409) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:21:32.130256 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-54ecb800-c11b-4b2a-b67c-66240e03e974 None None] Validating token access rules against request {{(pid=81409) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:21:32.139996 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.notifications [None req-aac43c76-b211-4b17-9cbb-5217feb7726c admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '26c38224278541b3a64848042505ff26'} {{(pid=81408) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 13 04:21:32.152688 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b099c04f-a622-4eeb-b24c-23226c860dcd admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8ba823f313144da3b683bacef8fd62e0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81409) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:21:32.153323 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b099c04f-a622-4eeb-b24c-23226c860dcd admin admin] REQUEST_METHOD: `POST` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:21:32.154165 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b099c04f-a622-4eeb-b24c-23226c860dcd admin admin] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:21:32.154658 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b099c04f-a622-4eeb-b24c-23226c860dcd admin admin] PATH_INFO: `/v3/projects` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:21:32.154979 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.notifications [None req-aac43c76-b211-4b17-9cbb-5217feb7726c admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '26c38224278541b3a64848042505ff26'} {{(pid=81408) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 13 04:21:32.155387 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b099c04f-a622-4eeb-b24c-23226c860dcd admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:21:32.156300 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.notifications [None req-aac43c76-b211-4b17-9cbb-5217feb7726c admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '26c38224278541b3a64848042505ff26'} {{(pid=81408) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 13 04:21:32.157190 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.notifications [None req-aac43c76-b211-4b17-9cbb-5217feb7726c admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '26c38224278541b3a64848042505ff26'} {{(pid=81408) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 13 04:21:32.157322 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b099c04f-a622-4eeb-b24c-23226c860dcd admin admin] RBAC: Authorization granted {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:21:32.172456 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.notifications [None req-aac43c76-b211-4b17-9cbb-5217feb7726c admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '26c38224278541b3a64848042505ff26'} {{(pid=81408) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 13 04:21:32.220629 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 746/1472] 10.222.0.27 () {60 vars in 1153 bytes} [Thu Oct 13 04:21:32 2022] POST /identity/v3/projects => generated 389 bytes in 153 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 13 04:21:32.226073 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b099c04f-a622-4eeb-b24c-23226c860dcd admin admin] Authenticating user token {{(pid=81409) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:21:32.241801 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.fernet_utils [None req-b099c04f-a622-4eeb-b24c-23226c860dcd admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81409) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:21:32.249273 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 727/1473] 10.222.0.27 () {58 vars in 1224 bytes} [Thu Oct 13 04:21:31 2022] DELETE /identity/v3/users/26c38224278541b3a64848042505ff26 => generated 0 bytes in 816 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 13 04:21:32.402513 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b099c04f-a622-4eeb-b24c-23226c860dcd admin admin] Validating token access rules against request {{(pid=81409) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:21:32.477927 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1de478f4-1e1f-4182-b129-b4dd11495770 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8ba823f313144da3b683bacef8fd62e0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81409) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:21:32.478590 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1de478f4-1e1f-4182-b129-b4dd11495770 admin admin] REQUEST_METHOD: `POST` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:21:32.478868 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1de478f4-1e1f-4182-b129-b4dd11495770 admin admin] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:21:32.479289 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1de478f4-1e1f-4182-b129-b4dd11495770 admin admin] PATH_INFO: `/v3/users` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:21:32.479941 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1de478f4-1e1f-4182-b129-b4dd11495770 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:21:32.482543 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1de478f4-1e1f-4182-b129-b4dd11495770 admin admin] RBAC: Authorization granted {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:21:32.516797 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 747/1474] 10.222.0.27 () {60 vars in 1144 bytes} [Thu Oct 13 04:21:32 2022] POST /identity/v3/users => generated 301 bytes in 289 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 13 04:21:32.518365 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aac43c76-b211-4b17-9cbb-5217feb7726c admin admin] Authenticating user token {{(pid=81408) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:21:32.528823 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aac43c76-b211-4b17-9cbb-5217feb7726c admin admin] Validating token access rules against request {{(pid=81408) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:21:32.534451 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-68a6bb54-cbea-4cd7-8a26-c7777316931b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8ba823f313144da3b683bacef8fd62e0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81408) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:21:32.535203 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68a6bb54-cbea-4cd7-8a26-c7777316931b admin admin] REQUEST_METHOD: `GET` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:21:32.535523 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68a6bb54-cbea-4cd7-8a26-c7777316931b admin admin] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:21:32.535902 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68a6bb54-cbea-4cd7-8a26-c7777316931b admin admin] PATH_INFO: `/v3/roles` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:21:32.536761 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-68a6bb54-cbea-4cd7-8a26-c7777316931b admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:21:32.539014 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-68a6bb54-cbea-4cd7-8a26-c7777316931b admin admin] RBAC: Authorization granted {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:21:32.551698 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 728/1475] 10.222.0.27 () {58 vars in 1122 bytes} [Thu Oct 13 04:21:32 2022] GET /identity/v3/roles => generated 1420 bytes in 34 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 13 04:21:32.558513 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1de478f4-1e1f-4182-b129-b4dd11495770 admin admin] Authenticating user token {{(pid=81409) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:21:32.565817 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG dbcounter [-] [81409] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=81409) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:21:32.570289 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1de478f4-1e1f-4182-b129-b4dd11495770 admin admin] Validating token access rules against request {{(pid=81409) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:21:32.579121 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e1ca3d91-8c58-409b-b86e-fb9f3a54f5a7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8ba823f313144da3b683bacef8fd62e0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81409) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:21:32.579554 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1ca3d91-8c58-409b-b86e-fb9f3a54f5a7 admin admin] REQUEST_METHOD: `PUT` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:21:32.579835 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1ca3d91-8c58-409b-b86e-fb9f3a54f5a7 admin admin] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:21:32.580111 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1ca3d91-8c58-409b-b86e-fb9f3a54f5a7 admin admin] PATH_INFO: `/v3/projects/36d9f444580d4391ae1aa2fe0bee603d/users/2100d1f49fb44bc99345bedd2f68ca2d/roles/ef55fbd94e1040be83695754075a7ed5` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:21:32.627393 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e1ca3d91-8c58-409b-b86e-fb9f3a54f5a7 admin admin] RBAC: Authorizing `identity:create_grant(project_id=36d9f444580d4391ae1aa2fe0bee603d, user_id=2100d1f49fb44bc99345bedd2f68ca2d, role_id=ef55fbd94e1040be83695754075a7ed5)` {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:21:32.636566 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e1ca3d91-8c58-409b-b86e-fb9f3a54f5a7 admin admin] RBAC: Authorization granted {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:21:32.673152 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 748/1476] 10.222.0.27 () {60 vars in 1483 bytes} [Thu Oct 13 04:21:32 2022] PUT /identity/v3/projects/36d9f444580d4391ae1aa2fe0bee603d/users/2100d1f49fb44bc99345bedd2f68ca2d/roles/ef55fbd94e1040be83695754075a7ed5 => generated 0 bytes in 115 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 13 04:21:32.676929 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-68a6bb54-cbea-4cd7-8a26-c7777316931b admin admin] Authenticating user token {{(pid=81408) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:21:32.690147 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-68a6bb54-cbea-4cd7-8a26-c7777316931b admin admin] Validating token access rules against request {{(pid=81408) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:21:32.805983 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-24e0cead-fe27-4730-aed1-ae57b9b2fcbb admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8ba823f313144da3b683bacef8fd62e0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81408) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:21:32.806646 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24e0cead-fe27-4730-aed1-ae57b9b2fcbb admin admin] REQUEST_METHOD: `GET` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:21:32.806926 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24e0cead-fe27-4730-aed1-ae57b9b2fcbb admin admin] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:21:32.807203 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24e0cead-fe27-4730-aed1-ae57b9b2fcbb admin admin] PATH_INFO: `/v3/roles` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:21:32.808209 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-24e0cead-fe27-4730-aed1-ae57b9b2fcbb admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:21:32.809488 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-24e0cead-fe27-4730-aed1-ae57b9b2fcbb admin admin] RBAC: Authorization granted {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:21:32.813298 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG dbcounter [-] [81408] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81408) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:21:32.834501 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 729/1477] 10.222.0.27 () {58 vars in 1122 bytes} [Thu Oct 13 04:21:32 2022] GET /identity/v3/roles => generated 1420 bytes in 158 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 13 04:21:32.839430 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e1ca3d91-8c58-409b-b86e-fb9f3a54f5a7 admin admin] Authenticating user token {{(pid=81409) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:21:32.855206 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e1ca3d91-8c58-409b-b86e-fb9f3a54f5a7 admin admin] Validating token access rules against request {{(pid=81409) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:21:32.861495 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-415fe935-fb68-495a-a946-dc0a088c9c81 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8ba823f313144da3b683bacef8fd62e0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81409) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:21:32.862155 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-415fe935-fb68-495a-a946-dc0a088c9c81 admin admin] REQUEST_METHOD: `PUT` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:21:32.862475 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-415fe935-fb68-495a-a946-dc0a088c9c81 admin admin] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:21:32.862797 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-415fe935-fb68-495a-a946-dc0a088c9c81 admin admin] PATH_INFO: `/v3/projects/36d9f444580d4391ae1aa2fe0bee603d/users/2100d1f49fb44bc99345bedd2f68ca2d/roles/ef55fbd94e1040be83695754075a7ed5` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:21:32.865799 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-415fe935-fb68-495a-a946-dc0a088c9c81 admin admin] RBAC: Authorizing `identity:create_grant(project_id=36d9f444580d4391ae1aa2fe0bee603d, user_id=2100d1f49fb44bc99345bedd2f68ca2d, role_id=ef55fbd94e1040be83695754075a7ed5)` {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:21:32.868494 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-415fe935-fb68-495a-a946-dc0a088c9c81 admin admin] RBAC: Authorization granted {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:21:32.892652 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 749/1478] 10.222.0.27 () {60 vars in 1483 bytes} [Thu Oct 13 04:21:32 2022] PUT /identity/v3/projects/36d9f444580d4391ae1aa2fe0bee603d/users/2100d1f49fb44bc99345bedd2f68ca2d/roles/ef55fbd94e1040be83695754075a7ed5 => generated 0 bytes in 45 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 13 04:21:32.894265 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1a2fad2-a423-46be-ab4b-cf2f7c8a016f None None] REQUEST_METHOD: `POST` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:21:32.894553 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1a2fad2-a423-46be-ab4b-cf2f7c8a016f None None] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:21:32.894836 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1a2fad2-a423-46be-ab4b-cf2f7c8a016f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:21:32.943428 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.auth.core [None req-a1a2fad2-a423-46be-ab4b-cf2f7c8a016f None None] MFA Rules not processed for user `2100d1f49fb44bc99345bedd2f68ca2d`. Rule list: `[]` (Enabled: `True`). {{(pid=81408) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 13 04:21:32.944792 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.fernet_utils [None req-a1a2fad2-a423-46be-ab4b-cf2f7c8a016f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81408) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:21:33.032120 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 730/1479] 10.222.0.27 () {58 vars in 958 bytes} [Thu Oct 13 04:21:32 2022] POST /identity/v3/auth/tokens => generated 3333 bytes in 139 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 13 04:21:33.043276 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-415fe935-fb68-495a-a946-dc0a088c9c81 admin admin] Authenticating user token {{(pid=81409) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:21:33.047385 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.fernet_utils [None req-415fe935-fb68-495a-a946-dc0a088c9c81 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81409) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:21:33.187851 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-415fe935-fb68-495a-a946-dc0a088c9c81 admin admin] Validating token access rules against request {{(pid=81409) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:21:33.201967 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-97407bae-193a-484c-ba70-e28f52b56342 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6141b8a6c3d54e179d6879bb019f4d12', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dcec0431fb6849499d6aec35a234daa6', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81409) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:21:33.202653 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97407bae-193a-484c-ba70-e28f52b56342 service cinder] REQUEST_METHOD: `GET` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:21:33.203144 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97407bae-193a-484c-ba70-e28f52b56342 service cinder] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:21:33.203777 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97407bae-193a-484c-ba70-e28f52b56342 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:21:33.207914 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a1a2fad2-a423-46be-ab4b-cf2f7c8a016f None None] Authenticating user token {{(pid=81408) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:21:33.211792 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.fernet_utils [None req-a1a2fad2-a423-46be-ab4b-cf2f7c8a016f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81408) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:21:33.228642 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-97407bae-193a-484c-ba70-e28f52b56342 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:21:33.230750 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-97407bae-193a-484c-ba70-e28f52b56342 service cinder] RBAC: Authorization granted {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:21:33.282870 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 750/1480] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Oct 13 04:21:33 2022] GET /identity/v3/auth/tokens => generated 3333 bytes in 240 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Oct 13 04:21:33.288508 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17c55186-5f1f-4400-a971-1d2c4cc566d5 None None] REQUEST_METHOD: `GET` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:21:33.290554 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17c55186-5f1f-4400-a971-1d2c4cc566d5 None None] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:21:33.292059 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17c55186-5f1f-4400-a971-1d2c4cc566d5 None None] PATH_INFO: `/` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:21:33.298136 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 751/1481] 10.222.0.27 () {54 vars in 921 bytes} [Thu Oct 13 04:21:33 2022] GET /identity => generated 270 bytes in 15 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 13 04:21:33.317221 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a43522a-1125-416e-b697-95c543c5e4a2 None None] REQUEST_METHOD: `POST` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:21:33.318153 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a43522a-1125-416e-b697-95c543c5e4a2 None None] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:21:33.318319 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a43522a-1125-416e-b697-95c543c5e4a2 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:21:33.380582 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a1a2fad2-a423-46be-ab4b-cf2f7c8a016f None None] Validating token access rules against request {{(pid=81408) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:21:33.389563 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-62f34f81-5812-47ac-a242-2484f47d1d1b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8ba823f313144da3b683bacef8fd62e0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81408) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:21:33.390711 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62f34f81-5812-47ac-a242-2484f47d1d1b admin admin] REQUEST_METHOD: `DELETE` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:21:33.391061 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62f34f81-5812-47ac-a242-2484f47d1d1b admin admin] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:21:33.391414 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62f34f81-5812-47ac-a242-2484f47d1d1b admin admin] PATH_INFO: `/v3/projects/5d861bc994b44e8a9ab9eaf7f1d048b1` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:21:33.394032 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-62f34f81-5812-47ac-a242-2484f47d1d1b admin admin] RBAC: Authorizing `identity:delete_project(project_id=5d861bc994b44e8a9ab9eaf7f1d048b1)` {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:21:33.394509 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.auth.core [None req-3a43522a-1125-416e-b697-95c543c5e4a2 None None] MFA Rules not processed for user `cb740ed7f7684b079e45f887be83f795`. Rule list: `[]` (Enabled: `True`). {{(pid=81409) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 13 04:21:33.395866 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.fernet_utils [None req-3a43522a-1125-416e-b697-95c543c5e4a2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81409) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:21:33.396359 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-62f34f81-5812-47ac-a242-2484f47d1d1b admin admin] RBAC: Authorization granted {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:21:33.493163 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.notifications [None req-62f34f81-5812-47ac-a242-2484f47d1d1b admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '5d861bc994b44e8a9ab9eaf7f1d048b1', 'request_id': 'req-62f34f81-5812-47ac-a242-2484f47d1d1b'} {{(pid=81408) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 13 04:21:33.512045 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.notifications [None req-62f34f81-5812-47ac-a242-2484f47d1d1b admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '5d861bc994b44e8a9ab9eaf7f1d048b1', 'request_id': 'req-62f34f81-5812-47ac-a242-2484f47d1d1b'} {{(pid=81408) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 13 04:21:33.522940 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.notifications [None req-62f34f81-5812-47ac-a242-2484f47d1d1b admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '5d861bc994b44e8a9ab9eaf7f1d048b1', 'request_id': 'req-62f34f81-5812-47ac-a242-2484f47d1d1b'} {{(pid=81408) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 13 04:21:33.541150 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 752/1482] 10.222.0.27 () {58 vars in 1018 bytes} [Thu Oct 13 04:21:33 2022] POST /identity/v3/auth/tokens => generated 3372 bytes in 225 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 13 04:21:33.542123 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3a43522a-1125-416e-b697-95c543c5e4a2 None None] Authenticating user token {{(pid=81409) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:21:33.543107 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.notifications [None req-62f34f81-5812-47ac-a242-2484f47d1d1b admin admin] The token cache is being invalidate because project 5d861bc994b44e8a9ab9eaf7f1d048b1 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81408) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 13 04:21:33.543554 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.notifications [None req-62f34f81-5812-47ac-a242-2484f47d1d1b admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81408) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 13 04:21:33.544972 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.notifications [None req-62f34f81-5812-47ac-a242-2484f47d1d1b admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81408) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 13 04:21:33.547167 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 731/1483] 10.222.0.27 () {58 vars in 1233 bytes} [Thu Oct 13 04:21:33 2022] DELETE /identity/v3/projects/5d861bc994b44e8a9ab9eaf7f1d048b1 => generated 0 bytes in 340 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 13 04:21:33.548320 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-62f34f81-5812-47ac-a242-2484f47d1d1b admin admin] Authenticating user token {{(pid=81408) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:21:33.549854 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.fernet_utils [None req-62f34f81-5812-47ac-a242-2484f47d1d1b admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81408) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:21:33.558937 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3a43522a-1125-416e-b697-95c543c5e4a2 None None] Validating token access rules against request {{(pid=81409) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:21:33.570096 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG dbcounter [-] [81408] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=81408) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:21:33.675238 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-62f34f81-5812-47ac-a242-2484f47d1d1b admin admin] Validating token access rules against request {{(pid=81408) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:21:33.683752 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7f1dd0f7-dc17-4e81-83f7-11be3cf1a862 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8ba823f313144da3b683bacef8fd62e0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81409) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:21:33.684205 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e3932181-8a49-4491-ad7f-894356c693c2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8ba823f313144da3b683bacef8fd62e0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81408) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:21:33.684658 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f1dd0f7-dc17-4e81-83f7-11be3cf1a862 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:21:33.685009 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3932181-8a49-4491-ad7f-894356c693c2 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:21:33.685379 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3932181-8a49-4491-ad7f-894356c693c2 admin admin] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:21:33.685756 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3932181-8a49-4491-ad7f-894356c693c2 admin admin] PATH_INFO: `/v3/projects/8e50e80a13214d0497c3d0f3c8b06d5f` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:21:33.685872 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f1dd0f7-dc17-4e81-83f7-11be3cf1a862 admin admin] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:21:33.686156 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f1dd0f7-dc17-4e81-83f7-11be3cf1a862 admin admin] PATH_INFO: `/v3/users/63a5dd2f709d429d82238913f31c8570` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:21:33.687281 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e3932181-8a49-4491-ad7f-894356c693c2 admin admin] RBAC: Authorizing `identity:delete_project(project_id=8e50e80a13214d0497c3d0f3c8b06d5f)` {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:21:33.687393 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7f1dd0f7-dc17-4e81-83f7-11be3cf1a862 admin admin] RBAC: Authorizing `identity:delete_user(user_id=63a5dd2f709d429d82238913f31c8570)` {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:21:33.689448 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e3932181-8a49-4491-ad7f-894356c693c2 admin admin] RBAC: Authorization granted {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:21:33.689618 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7f1dd0f7-dc17-4e81-83f7-11be3cf1a862 admin admin] RBAC: Authorization granted {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:21:33.692113 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG dbcounter [-] [81409] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=81409) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:21:33.747084 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.notifications [None req-e3932181-8a49-4491-ad7f-894356c693c2 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '8e50e80a13214d0497c3d0f3c8b06d5f', 'request_id': 'req-e3932181-8a49-4491-ad7f-894356c693c2'} {{(pid=81408) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 13 04:21:33.756622 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.notifications [None req-7f1dd0f7-dc17-4e81-83f7-11be3cf1a862 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '63a5dd2f709d429d82238913f31c8570'} {{(pid=81409) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 13 04:21:33.757386 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.notifications [None req-7f1dd0f7-dc17-4e81-83f7-11be3cf1a862 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '63a5dd2f709d429d82238913f31c8570'} {{(pid=81409) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 13 04:21:33.758161 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.notifications [None req-e3932181-8a49-4491-ad7f-894356c693c2 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '8e50e80a13214d0497c3d0f3c8b06d5f', 'request_id': 'req-e3932181-8a49-4491-ad7f-894356c693c2'} {{(pid=81408) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 13 04:21:33.767251 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.notifications [None req-e3932181-8a49-4491-ad7f-894356c693c2 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '8e50e80a13214d0497c3d0f3c8b06d5f', 'request_id': 'req-e3932181-8a49-4491-ad7f-894356c693c2'} {{(pid=81408) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 13 04:21:33.767557 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.notifications [None req-7f1dd0f7-dc17-4e81-83f7-11be3cf1a862 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '63a5dd2f709d429d82238913f31c8570'} {{(pid=81409) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 13 04:21:33.768313 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.notifications [None req-7f1dd0f7-dc17-4e81-83f7-11be3cf1a862 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '63a5dd2f709d429d82238913f31c8570'} {{(pid=81409) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 13 04:21:33.778272 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.notifications [None req-7f1dd0f7-dc17-4e81-83f7-11be3cf1a862 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '63a5dd2f709d429d82238913f31c8570'} {{(pid=81409) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 13 04:21:33.785810 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.notifications [None req-e3932181-8a49-4491-ad7f-894356c693c2 admin admin] The token cache is being invalidate because project 8e50e80a13214d0497c3d0f3c8b06d5f was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81408) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 13 04:21:33.786640 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.notifications [None req-e3932181-8a49-4491-ad7f-894356c693c2 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81408) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 13 04:21:33.787589 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.notifications [None req-e3932181-8a49-4491-ad7f-894356c693c2 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81408) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 13 04:21:33.794347 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 732/1484] 10.222.0.27 () {58 vars in 1233 bytes} [Thu Oct 13 04:21:33 2022] DELETE /identity/v3/projects/8e50e80a13214d0497c3d0f3c8b06d5f => generated 0 bytes in 243 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 13 04:21:33.795526 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e3932181-8a49-4491-ad7f-894356c693c2 admin admin] Authenticating user token {{(pid=81408) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:21:33.798402 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.fernet_utils [None req-e3932181-8a49-4491-ad7f-894356c693c2 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81408) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:21:33.809394 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 753/1485] 10.222.0.27 () {58 vars in 1224 bytes} [Thu Oct 13 04:21:33 2022] DELETE /identity/v3/users/63a5dd2f709d429d82238913f31c8570 => generated 0 bytes in 264 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 13 04:21:33.811137 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7f1dd0f7-dc17-4e81-83f7-11be3cf1a862 admin admin] Authenticating user token {{(pid=81409) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:21:33.813155 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.fernet_utils [None req-7f1dd0f7-dc17-4e81-83f7-11be3cf1a862 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81409) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:21:33.908312 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e3932181-8a49-4491-ad7f-894356c693c2 admin admin] Validating token access rules against request {{(pid=81408) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:21:33.917050 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7f1dd0f7-dc17-4e81-83f7-11be3cf1a862 admin admin] Validating token access rules against request {{(pid=81409) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:21:33.924704 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ababa9c4-934c-45f6-8f20-f78427d6b8b1 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cb740ed7f7684b079e45f887be83f795', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dcec0431fb6849499d6aec35a234daa6', 'project_domain_id': 'default', 'roles': ['ResellerAdmin', 'admin', 'reader', 'service', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81409) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:21:33.925331 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ababa9c4-934c-45f6-8f20-f78427d6b8b1 service nova] REQUEST_METHOD: `GET` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:21:33.925657 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ababa9c4-934c-45f6-8f20-f78427d6b8b1 service nova] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:21:33.925934 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ababa9c4-934c-45f6-8f20-f78427d6b8b1 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:21:33.928038 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.fernet_utils [None req-ababa9c4-934c-45f6-8f20-f78427d6b8b1 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81409) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:21:33.947152 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ababa9c4-934c-45f6-8f20-f78427d6b8b1 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:21:33.949584 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ababa9c4-934c-45f6-8f20-f78427d6b8b1 service nova] RBAC: Authorization granted {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:21:33.970676 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 754/1486] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Oct 13 04:21:33 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 160 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 13 04:21:33.972746 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-423897ee-2112-4cdc-bc52-a8ddc6f08e8f None None] REQUEST_METHOD: `POST` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:21:33.973412 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-423897ee-2112-4cdc-bc52-a8ddc6f08e8f None None] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:21:33.973982 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-423897ee-2112-4cdc-bc52-a8ddc6f08e8f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:21:34.008731 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG dbcounter [-] [81408] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=81408) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:21:34.015223 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.auth.core [None req-423897ee-2112-4cdc-bc52-a8ddc6f08e8f None None] MFA Rules not processed for user `0e2d979563d6421995451a0e86bf3022`. Rule list: `[]` (Enabled: `True`). {{(pid=81409) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 13 04:21:34.017390 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.fernet_utils [None req-423897ee-2112-4cdc-bc52-a8ddc6f08e8f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81409) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:21:34.027732 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-14050da0-3691-4049-b8ba-92e4777083b5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8ba823f313144da3b683bacef8fd62e0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81408) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:21:34.028393 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14050da0-3691-4049-b8ba-92e4777083b5 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:21:34.029236 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14050da0-3691-4049-b8ba-92e4777083b5 admin admin] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:21:34.029510 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14050da0-3691-4049-b8ba-92e4777083b5 admin admin] PATH_INFO: `/v3/users/7e311300d40945edb61546b619d13f31` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:21:34.030843 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-14050da0-3691-4049-b8ba-92e4777083b5 admin admin] RBAC: Authorizing `identity:delete_user(user_id=7e311300d40945edb61546b619d13f31)` {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:21:34.033397 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-14050da0-3691-4049-b8ba-92e4777083b5 admin admin] RBAC: Authorization granted {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:21:34.052151 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 755/1487] 10.222.0.27 () {58 vars in 958 bytes} [Thu Oct 13 04:21:33 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 80 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 13 04:21:34.053393 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27971e5f-4514-4add-add4-8caef8f9a731 None None] REQUEST_METHOD: `POST` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:21:34.053674 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27971e5f-4514-4add-add4-8caef8f9a731 None None] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:21:34.053939 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27971e5f-4514-4add-add4-8caef8f9a731 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:21:34.102695 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.auth.core [None req-27971e5f-4514-4add-add4-8caef8f9a731 None None] MFA Rules not processed for user `0e2d979563d6421995451a0e86bf3022`. Rule list: `[]` (Enabled: `True`). {{(pid=81409) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 13 04:21:34.111249 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.fernet_utils [None req-27971e5f-4514-4add-add4-8caef8f9a731 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81409) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:21:34.132793 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 756/1488] 10.222.0.27 () {58 vars in 958 bytes} [Thu Oct 13 04:21:34 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 80 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 13 04:21:34.134570 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.notifications [None req-14050da0-3691-4049-b8ba-92e4777083b5 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '7e311300d40945edb61546b619d13f31'} {{(pid=81408) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 13 04:21:34.134927 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a07ddcc-a8e9-4766-adcf-c3c3c4f1b9e1 None None] REQUEST_METHOD: `POST` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:21:34.135420 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a07ddcc-a8e9-4766-adcf-c3c3c4f1b9e1 None None] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:21:34.135756 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a07ddcc-a8e9-4766-adcf-c3c3c4f1b9e1 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:21:34.147687 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.notifications [None req-14050da0-3691-4049-b8ba-92e4777083b5 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '7e311300d40945edb61546b619d13f31'} {{(pid=81408) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 13 04:21:34.148726 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.notifications [None req-14050da0-3691-4049-b8ba-92e4777083b5 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '7e311300d40945edb61546b619d13f31'} {{(pid=81408) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 13 04:21:34.150405 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.notifications [None req-14050da0-3691-4049-b8ba-92e4777083b5 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '7e311300d40945edb61546b619d13f31'} {{(pid=81408) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 13 04:21:34.162328 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.notifications [None req-14050da0-3691-4049-b8ba-92e4777083b5 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '7e311300d40945edb61546b619d13f31'} {{(pid=81408) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 13 04:21:34.167002 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.auth.core [None req-7a07ddcc-a8e9-4766-adcf-c3c3c4f1b9e1 None None] MFA Rules not processed for user `0e2d979563d6421995451a0e86bf3022`. Rule list: `[]` (Enabled: `True`). {{(pid=81409) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 13 04:21:34.171777 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.fernet_utils [None req-7a07ddcc-a8e9-4766-adcf-c3c3c4f1b9e1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81409) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:21:34.187535 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 733/1489] 10.222.0.27 () {58 vars in 1224 bytes} [Thu Oct 13 04:21:33 2022] DELETE /identity/v3/users/7e311300d40945edb61546b619d13f31 => generated 0 bytes in 392 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 13 04:21:34.189851 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 757/1490] 10.222.0.27 () {58 vars in 958 bytes} [Thu Oct 13 04:21:34 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 56 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 13 04:21:34.190594 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-14050da0-3691-4049-b8ba-92e4777083b5 admin admin] Authenticating user token {{(pid=81408) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:21:34.198436 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7a07ddcc-a8e9-4766-adcf-c3c3c4f1b9e1 None None] Authenticating user token {{(pid=81409) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:21:34.199941 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.fernet_utils [None req-7a07ddcc-a8e9-4766-adcf-c3c3c4f1b9e1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81409) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:21:34.263816 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG dbcounter [-] [81409] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81409) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:21:34.271468 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7a07ddcc-a8e9-4766-adcf-c3c3c4f1b9e1 None None] Validating token access rules against request {{(pid=81409) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:21:34.322302 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-14050da0-3691-4049-b8ba-92e4777083b5 admin admin] Validating token access rules against request {{(pid=81408) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:21:34.328276 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-59745f0a-08fe-49c0-9c5f-00a560c55ad7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8ba823f313144da3b683bacef8fd62e0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81408) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:21:34.329551 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59745f0a-08fe-49c0-9c5f-00a560c55ad7 admin admin] REQUEST_METHOD: `GET` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:21:34.329551 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59745f0a-08fe-49c0-9c5f-00a560c55ad7 admin admin] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:21:34.329790 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59745f0a-08fe-49c0-9c5f-00a560c55ad7 admin admin] PATH_INFO: `/v3/domains` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:21:34.330338 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-59745f0a-08fe-49c0-9c5f-00a560c55ad7 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81408) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 13 04:21:34.330641 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-59745f0a-08fe-49c0-9c5f-00a560c55ad7 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:21:34.332377 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-59745f0a-08fe-49c0-9c5f-00a560c55ad7 admin admin] RBAC: Authorization granted {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:21:34.346977 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 734/1491] 10.222.0.27 () {58 vars in 1153 bytes} [Thu Oct 13 04:21:34 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 156 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 13 04:21:34.350542 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-59745f0a-08fe-49c0-9c5f-00a560c55ad7 admin admin] Authenticating user token {{(pid=81408) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:21:34.363511 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-59745f0a-08fe-49c0-9c5f-00a560c55ad7 admin admin] Validating token access rules against request {{(pid=81408) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:21:34.368718 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-03fb4ef4-07a4-43b6-8a13-3ff5a4a04d72 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8ba823f313144da3b683bacef8fd62e0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81408) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:21:34.369637 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03fb4ef4-07a4-43b6-8a13-3ff5a4a04d72 admin admin] REQUEST_METHOD: `POST` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:21:34.370020 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03fb4ef4-07a4-43b6-8a13-3ff5a4a04d72 admin admin] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:21:34.370451 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03fb4ef4-07a4-43b6-8a13-3ff5a4a04d72 admin admin] PATH_INFO: `/v3/projects` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:21:34.371198 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-03fb4ef4-07a4-43b6-8a13-3ff5a4a04d72 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:21:34.373347 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-03fb4ef4-07a4-43b6-8a13-3ff5a4a04d72 admin admin] RBAC: Authorization granted {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:21:34.431117 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 735/1492] 10.222.0.27 () {60 vars in 1153 bytes} [Thu Oct 13 04:21:34 2022] POST /identity/v3/projects => generated 383 bytes in 80 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 13 04:21:34.432240 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-03fb4ef4-07a4-43b6-8a13-3ff5a4a04d72 admin admin] Authenticating user token {{(pid=81408) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:21:34.435393 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c8a0fb83-7c62-4e2a-8619-690891483c90 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd8812c3974f84871bb2d761cf0016257', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dcec0431fb6849499d6aec35a234daa6', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81409) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:21:34.437852 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8a0fb83-7c62-4e2a-8619-690891483c90 service neutron] REQUEST_METHOD: `GET` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:21:34.438133 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8a0fb83-7c62-4e2a-8619-690891483c90 service neutron] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:21:34.438436 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8a0fb83-7c62-4e2a-8619-690891483c90 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:21:34.447096 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.fernet_utils [None req-c8a0fb83-7c62-4e2a-8619-690891483c90 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81409) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:21:34.495589 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG dbcounter [-] [81408] Writing DB stats keystone:SELECT=88,keystone:DELETE=10,keystone:INSERT=2 {{(pid=81408) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:21:34.500941 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c8a0fb83-7c62-4e2a-8619-690891483c90 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:21:34.508634 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c8a0fb83-7c62-4e2a-8619-690891483c90 service neutron] RBAC: Authorization granted {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:21:34.539110 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-03fb4ef4-07a4-43b6-8a13-3ff5a4a04d72 admin admin] Validating token access rules against request {{(pid=81408) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:21:34.545025 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 758/1493] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Oct 13 04:21:34 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 347 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 13 04:21:34.552114 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e91fbd28-2779-4e98-b9f6-e99a70a59c32 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8ba823f313144da3b683bacef8fd62e0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81408) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:21:34.552964 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e91fbd28-2779-4e98-b9f6-e99a70a59c32 admin admin] REQUEST_METHOD: `POST` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:21:34.554902 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e91fbd28-2779-4e98-b9f6-e99a70a59c32 admin admin] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:21:34.555390 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e91fbd28-2779-4e98-b9f6-e99a70a59c32 admin admin] PATH_INFO: `/v3/users` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:21:34.558301 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e91fbd28-2779-4e98-b9f6-e99a70a59c32 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:21:34.562788 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e91fbd28-2779-4e98-b9f6-e99a70a59c32 admin admin] RBAC: Authorization granted {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:21:34.599678 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 736/1494] 10.222.0.27 () {60 vars in 1144 bytes} [Thu Oct 13 04:21:34 2022] POST /identity/v3/users => generated 292 bytes in 168 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 13 04:21:34.603376 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c8a0fb83-7c62-4e2a-8619-690891483c90 service neutron] Authenticating user token {{(pid=81409) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:21:34.620406 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c8a0fb83-7c62-4e2a-8619-690891483c90 service neutron] Validating token access rules against request {{(pid=81409) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:21:34.640873 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bc2a2327-4dc3-4d10-9bfb-f496626581c5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8ba823f313144da3b683bacef8fd62e0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81409) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:21:34.641668 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc2a2327-4dc3-4d10-9bfb-f496626581c5 admin admin] REQUEST_METHOD: `GET` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:21:34.641905 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc2a2327-4dc3-4d10-9bfb-f496626581c5 admin admin] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:21:34.642255 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc2a2327-4dc3-4d10-9bfb-f496626581c5 admin admin] PATH_INFO: `/v3/roles` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:21:34.642923 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bc2a2327-4dc3-4d10-9bfb-f496626581c5 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:21:34.647246 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bc2a2327-4dc3-4d10-9bfb-f496626581c5 admin admin] RBAC: Authorization granted {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:21:34.685979 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 759/1495] 10.222.0.27 () {58 vars in 1122 bytes} [Thu Oct 13 04:21:34 2022] GET /identity/v3/roles => generated 1420 bytes in 83 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 13 04:21:34.686585 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e91fbd28-2779-4e98-b9f6-e99a70a59c32 admin admin] Authenticating user token {{(pid=81408) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:21:34.701516 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e91fbd28-2779-4e98-b9f6-e99a70a59c32 admin admin] Validating token access rules against request {{(pid=81408) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:21:34.712934 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5dcc9049-362b-436b-b692-5aba2787a0eb admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8ba823f313144da3b683bacef8fd62e0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81408) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:21:34.713701 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5dcc9049-362b-436b-b692-5aba2787a0eb admin admin] REQUEST_METHOD: `GET` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:21:34.714094 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5dcc9049-362b-436b-b692-5aba2787a0eb admin admin] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:21:34.714465 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5dcc9049-362b-436b-b692-5aba2787a0eb admin admin] PATH_INFO: `/v3/roles` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:21:34.715191 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5dcc9049-362b-436b-b692-5aba2787a0eb admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:21:34.722743 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5dcc9049-362b-436b-b692-5aba2787a0eb admin admin] RBAC: Authorization granted {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:21:34.743584 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 737/1496] 10.222.0.27 () {58 vars in 1122 bytes} [Thu Oct 13 04:21:34 2022] GET /identity/v3/roles => generated 1420 bytes in 55 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Oct 13 04:21:34.745310 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bc2a2327-4dc3-4d10-9bfb-f496626581c5 admin admin] Authenticating user token {{(pid=81409) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:21:34.759916 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bc2a2327-4dc3-4d10-9bfb-f496626581c5 admin admin] Validating token access rules against request {{(pid=81409) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:21:34.770374 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-efd1dfb7-1728-4937-a7b6-e808a92ee9e2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8ba823f313144da3b683bacef8fd62e0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81409) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:21:34.771622 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-efd1dfb7-1728-4937-a7b6-e808a92ee9e2 admin admin] REQUEST_METHOD: `PUT` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:21:34.772155 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-efd1dfb7-1728-4937-a7b6-e808a92ee9e2 admin admin] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:21:34.774074 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-efd1dfb7-1728-4937-a7b6-e808a92ee9e2 admin admin] PATH_INFO: `/v3/projects/234678ebae1c4d3bbb5917bc7a528eb9/users/ffa3b82f53484687b772e830cd68b461/roles/cc56c4ae13a7424ea771c15da4b153c1` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:21:34.820325 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5dcc9049-362b-436b-b692-5aba2787a0eb admin admin] Authenticating user token {{(pid=81408) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:21:34.822963 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.fernet_utils [None req-5dcc9049-362b-436b-b692-5aba2787a0eb admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81408) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:21:34.834499 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-efd1dfb7-1728-4937-a7b6-e808a92ee9e2 admin admin] RBAC: Authorizing `identity:create_grant(project_id=234678ebae1c4d3bbb5917bc7a528eb9, user_id=ffa3b82f53484687b772e830cd68b461, role_id=cc56c4ae13a7424ea771c15da4b153c1)` {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:21:34.838112 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-efd1dfb7-1728-4937-a7b6-e808a92ee9e2 admin admin] RBAC: Authorization granted {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:21:34.847495 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5dcc9049-362b-436b-b692-5aba2787a0eb admin admin] Validating token access rules against request {{(pid=81408) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:21:34.863166 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-56556475-2b1c-4984-8dcb-b2e2de3f86ab admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8ba823f313144da3b683bacef8fd62e0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81408) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:21:34.863816 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56556475-2b1c-4984-8dcb-b2e2de3f86ab admin admin] REQUEST_METHOD: `DELETE` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:21:34.864235 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56556475-2b1c-4984-8dcb-b2e2de3f86ab admin admin] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:21:34.864510 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56556475-2b1c-4984-8dcb-b2e2de3f86ab admin admin] PATH_INFO: `/v3/projects/9e18bc36229e4e118f04206e4b5765a7` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:21:34.876192 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 760/1497] 10.222.0.27 () {60 vars in 1483 bytes} [Thu Oct 13 04:21:34 2022] PUT /identity/v3/projects/234678ebae1c4d3bbb5917bc7a528eb9/users/ffa3b82f53484687b772e830cd68b461/roles/cc56c4ae13a7424ea771c15da4b153c1 => generated 0 bytes in 122 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 13 04:21:34.876403 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-56556475-2b1c-4984-8dcb-b2e2de3f86ab admin admin] RBAC: Authorizing `identity:delete_project(project_id=9e18bc36229e4e118f04206e4b5765a7)` {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:21:34.876403 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-56556475-2b1c-4984-8dcb-b2e2de3f86ab admin admin] RBAC: Authorization granted {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:21:34.878042 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b03f06ed-3afb-483d-8efd-d784622b1e23 None None] REQUEST_METHOD: `POST` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:21:34.878369 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b03f06ed-3afb-483d-8efd-d784622b1e23 None None] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:21:34.878644 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b03f06ed-3afb-483d-8efd-d784622b1e23 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:21:34.913574 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.auth.core [None req-b03f06ed-3afb-483d-8efd-d784622b1e23 None None] MFA Rules not processed for user `ffa3b82f53484687b772e830cd68b461`. Rule list: `[]` (Enabled: `True`). {{(pid=81409) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 13 04:21:34.915139 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.fernet_utils [None req-b03f06ed-3afb-483d-8efd-d784622b1e23 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81409) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:21:34.969634 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.notifications [None req-56556475-2b1c-4984-8dcb-b2e2de3f86ab admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '9e18bc36229e4e118f04206e4b5765a7', 'request_id': 'req-56556475-2b1c-4984-8dcb-b2e2de3f86ab'} {{(pid=81408) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 13 04:21:34.986464 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.notifications [None req-56556475-2b1c-4984-8dcb-b2e2de3f86ab admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '9e18bc36229e4e118f04206e4b5765a7', 'request_id': 'req-56556475-2b1c-4984-8dcb-b2e2de3f86ab'} {{(pid=81408) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 13 04:21:34.999984 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.notifications [None req-56556475-2b1c-4984-8dcb-b2e2de3f86ab admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '9e18bc36229e4e118f04206e4b5765a7', 'request_id': 'req-56556475-2b1c-4984-8dcb-b2e2de3f86ab'} {{(pid=81408) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 13 04:21:35.016525 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.notifications [None req-56556475-2b1c-4984-8dcb-b2e2de3f86ab admin admin] The token cache is being invalidate because project 9e18bc36229e4e118f04206e4b5765a7 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81408) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 13 04:21:35.017512 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.notifications [None req-56556475-2b1c-4984-8dcb-b2e2de3f86ab admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81408) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 13 04:21:35.018629 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.notifications [None req-56556475-2b1c-4984-8dcb-b2e2de3f86ab admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81408) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 13 04:21:35.020755 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 738/1498] 10.222.0.27 () {58 vars in 1233 bytes} [Thu Oct 13 04:21:34 2022] DELETE /identity/v3/projects/9e18bc36229e4e118f04206e4b5765a7 => generated 0 bytes in 201 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 13 04:21:35.026302 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-575b1106-dc3e-42f6-9300-238c02ad1bab None None] REQUEST_METHOD: `POST` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:21:35.026588 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-575b1106-dc3e-42f6-9300-238c02ad1bab None None] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:21:35.026862 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-575b1106-dc3e-42f6-9300-238c02ad1bab None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:21:35.039857 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 761/1499] 10.222.0.27 () {58 vars in 958 bytes} [Thu Oct 13 04:21:34 2022] POST /identity/v3/auth/tokens => generated 3260 bytes in 163 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 13 04:21:35.044632 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b03f06ed-3afb-483d-8efd-d784622b1e23 None None] Authenticating user token {{(pid=81409) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:21:35.069628 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.auth.core [None req-575b1106-dc3e-42f6-9300-238c02ad1bab None None] MFA Rules not processed for user `0e2d979563d6421995451a0e86bf3022`. Rule list: `[]` (Enabled: `True`). {{(pid=81408) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 13 04:21:35.073353 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.fernet_utils [None req-575b1106-dc3e-42f6-9300-238c02ad1bab None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81408) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:21:35.134510 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b03f06ed-3afb-483d-8efd-d784622b1e23 None None] Validating token access rules against request {{(pid=81409) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:21:35.138335 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG dbcounter [-] [81409] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81409) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:21:35.215631 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-20f256f5-2131-4e2e-8a55-c978d3a6a0c8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8ba823f313144da3b683bacef8fd62e0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81409) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:21:35.216753 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20f256f5-2131-4e2e-8a55-c978d3a6a0c8 admin admin] REQUEST_METHOD: `POST` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:21:35.217306 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20f256f5-2131-4e2e-8a55-c978d3a6a0c8 admin admin] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:21:35.217719 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20f256f5-2131-4e2e-8a55-c978d3a6a0c8 admin admin] PATH_INFO: `/v3/projects` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:21:35.218470 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-20f256f5-2131-4e2e-8a55-c978d3a6a0c8 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:21:35.220447 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-20f256f5-2131-4e2e-8a55-c978d3a6a0c8 admin admin] RBAC: Authorization granted {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:21:35.277151 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 739/1500] 10.222.0.27 () {58 vars in 958 bytes} [Thu Oct 13 04:21:35 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 252 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 13 04:21:35.279712 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db001dd9-9165-4ea8-9936-b21519fbb0a8 None None] REQUEST_METHOD: `POST` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:21:35.279712 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db001dd9-9165-4ea8-9936-b21519fbb0a8 None None] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:21:35.279954 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db001dd9-9165-4ea8-9936-b21519fbb0a8 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:21:35.302555 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 762/1501] 10.222.0.27 () {60 vars in 1153 bytes} [Thu Oct 13 04:21:35 2022] POST /identity/v3/projects => generated 383 bytes in 250 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 13 04:21:35.305388 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.auth.core [None req-db001dd9-9165-4ea8-9936-b21519fbb0a8 None None] MFA Rules not processed for user `0e2d979563d6421995451a0e86bf3022`. Rule list: `[]` (Enabled: `True`). {{(pid=81408) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 13 04:21:35.306738 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.fernet_utils [None req-db001dd9-9165-4ea8-9936-b21519fbb0a8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81408) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:21:35.307226 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-20f256f5-2131-4e2e-8a55-c978d3a6a0c8 admin admin] Authenticating user token {{(pid=81409) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:21:35.313920 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.fernet_utils [None req-20f256f5-2131-4e2e-8a55-c978d3a6a0c8 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81409) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:21:35.370598 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG dbcounter [-] [81408] Writing DB stats keystone:SELECT=89,keystone:INSERT=4,keystone:DELETE=7 {{(pid=81408) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:21:35.423633 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 740/1502] 10.222.0.27 () {58 vars in 958 bytes} [Thu Oct 13 04:21:35 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 146 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 13 04:21:35.424592 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-db001dd9-9165-4ea8-9936-b21519fbb0a8 None None] Authenticating user token {{(pid=81408) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:21:35.436101 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-db001dd9-9165-4ea8-9936-b21519fbb0a8 None None] Validating token access rules against request {{(pid=81408) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:21:35.442390 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-945e9190-7b6e-498a-8b1a-dfecd1a5deac admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8ba823f313144da3b683bacef8fd62e0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81408) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:21:35.443602 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-945e9190-7b6e-498a-8b1a-dfecd1a5deac admin admin] REQUEST_METHOD: `GET` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:21:35.443885 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-945e9190-7b6e-498a-8b1a-dfecd1a5deac admin admin] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:21:35.444156 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-945e9190-7b6e-498a-8b1a-dfecd1a5deac admin admin] PATH_INFO: `/v3/domains` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:21:35.444814 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-945e9190-7b6e-498a-8b1a-dfecd1a5deac admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81408) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 13 04:21:35.445117 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-945e9190-7b6e-498a-8b1a-dfecd1a5deac admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:21:35.447373 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-945e9190-7b6e-498a-8b1a-dfecd1a5deac admin admin] RBAC: Authorization granted {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:21:35.454164 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-20f256f5-2131-4e2e-8a55-c978d3a6a0c8 admin admin] Validating token access rules against request {{(pid=81409) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:21:35.460585 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9d28558d-883a-44c5-9b36-b6c1b698dc05 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8ba823f313144da3b683bacef8fd62e0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81409) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:21:35.461348 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d28558d-883a-44c5-9b36-b6c1b698dc05 admin admin] REQUEST_METHOD: `POST` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:21:35.461783 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d28558d-883a-44c5-9b36-b6c1b698dc05 admin admin] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:21:35.462376 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d28558d-883a-44c5-9b36-b6c1b698dc05 admin admin] PATH_INFO: `/v3/users` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:21:35.463135 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9d28558d-883a-44c5-9b36-b6c1b698dc05 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:21:35.465070 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 741/1503] 10.222.0.27 () {58 vars in 1153 bytes} [Thu Oct 13 04:21:35 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 41 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 13 04:21:35.467224 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-945e9190-7b6e-498a-8b1a-dfecd1a5deac admin admin] Authenticating user token {{(pid=81408) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:21:35.467897 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9d28558d-883a-44c5-9b36-b6c1b698dc05 admin admin] RBAC: Authorization granted {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:21:35.496717 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-945e9190-7b6e-498a-8b1a-dfecd1a5deac admin admin] Validating token access rules against request {{(pid=81408) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:21:35.504952 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 763/1504] 10.222.0.27 () {60 vars in 1144 bytes} [Thu Oct 13 04:21:35 2022] POST /identity/v3/users => generated 298 bytes in 200 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 13 04:21:35.507185 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-083a9c10-6c64-40f7-a10c-c603c11ee016 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8ba823f313144da3b683bacef8fd62e0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81408) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:21:35.507829 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-083a9c10-6c64-40f7-a10c-c603c11ee016 admin admin] REQUEST_METHOD: `POST` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:21:35.508977 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9d28558d-883a-44c5-9b36-b6c1b698dc05 admin admin] Authenticating user token {{(pid=81409) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:21:35.510095 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-083a9c10-6c64-40f7-a10c-c603c11ee016 admin admin] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:21:35.511006 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-083a9c10-6c64-40f7-a10c-c603c11ee016 admin admin] PATH_INFO: `/v3/projects` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:21:35.511676 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-083a9c10-6c64-40f7-a10c-c603c11ee016 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:21:35.515887 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-083a9c10-6c64-40f7-a10c-c603c11ee016 admin admin] RBAC: Authorization granted {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:21:35.528439 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9d28558d-883a-44c5-9b36-b6c1b698dc05 admin admin] Validating token access rules against request {{(pid=81409) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:21:35.533668 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a8a62e92-3c9d-4ad5-89cb-6646bf6db2ce admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8ba823f313144da3b683bacef8fd62e0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81409) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:21:35.534509 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8a62e92-3c9d-4ad5-89cb-6646bf6db2ce admin admin] REQUEST_METHOD: `GET` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:21:35.534789 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8a62e92-3c9d-4ad5-89cb-6646bf6db2ce admin admin] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:21:35.535061 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8a62e92-3c9d-4ad5-89cb-6646bf6db2ce admin admin] PATH_INFO: `/v3/roles` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:21:35.535717 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a8a62e92-3c9d-4ad5-89cb-6646bf6db2ce admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:21:35.537337 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a8a62e92-3c9d-4ad5-89cb-6646bf6db2ce admin admin] RBAC: Authorization granted {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:21:35.549489 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 764/1505] 10.222.0.27 () {58 vars in 1122 bytes} [Thu Oct 13 04:21:35 2022] GET /identity/v3/roles => generated 1420 bytes in 41 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 13 04:21:35.553747 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a8a62e92-3c9d-4ad5-89cb-6646bf6db2ce admin admin] Authenticating user token {{(pid=81409) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:21:35.572602 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a8a62e92-3c9d-4ad5-89cb-6646bf6db2ce admin admin] Validating token access rules against request {{(pid=81409) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:21:35.579368 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-812a9fcc-c445-474e-a7ed-c34d766c0fa4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8ba823f313144da3b683bacef8fd62e0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81409) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:21:35.580097 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 742/1506] 10.222.0.27 () {60 vars in 1153 bytes} [Thu Oct 13 04:21:35 2022] POST /identity/v3/projects => generated 371 bytes in 114 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 13 04:21:35.580543 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-812a9fcc-c445-474e-a7ed-c34d766c0fa4 admin admin] REQUEST_METHOD: `PUT` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:21:35.580543 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-812a9fcc-c445-474e-a7ed-c34d766c0fa4 admin admin] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:21:35.581021 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-812a9fcc-c445-474e-a7ed-c34d766c0fa4 admin admin] PATH_INFO: `/v3/projects/26a5281b0094454194695d52d3c2677f/users/111440b9f913470e951d6ad45415b30a/roles/ef55fbd94e1040be83695754075a7ed5` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:21:35.586187 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-083a9c10-6c64-40f7-a10c-c603c11ee016 admin admin] Authenticating user token {{(pid=81408) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:21:35.618327 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-812a9fcc-c445-474e-a7ed-c34d766c0fa4 admin admin] RBAC: Authorizing `identity:create_grant(project_id=26a5281b0094454194695d52d3c2677f, user_id=111440b9f913470e951d6ad45415b30a, role_id=ef55fbd94e1040be83695754075a7ed5)` {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:21:35.624166 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-812a9fcc-c445-474e-a7ed-c34d766c0fa4 admin admin] RBAC: Authorization granted {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:21:35.657989 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 765/1507] 10.222.0.27 () {60 vars in 1483 bytes} [Thu Oct 13 04:21:35 2022] PUT /identity/v3/projects/26a5281b0094454194695d52d3c2677f/users/111440b9f913470e951d6ad45415b30a/roles/ef55fbd94e1040be83695754075a7ed5 => generated 0 bytes in 105 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 13 04:21:35.662648 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-812a9fcc-c445-474e-a7ed-c34d766c0fa4 admin admin] Authenticating user token {{(pid=81409) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:21:35.693250 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-083a9c10-6c64-40f7-a10c-c603c11ee016 admin admin] Validating token access rules against request {{(pid=81408) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:21:35.737533 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG dbcounter [-] [81409] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=81409) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:21:35.758584 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-812a9fcc-c445-474e-a7ed-c34d766c0fa4 admin admin] Validating token access rules against request {{(pid=81409) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:21:35.765170 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0d6ed538-a6a2-408e-b1ad-6564580b5081 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8ba823f313144da3b683bacef8fd62e0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81409) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:21:35.765901 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d6ed538-a6a2-408e-b1ad-6564580b5081 admin admin] REQUEST_METHOD: `GET` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:21:35.766307 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d6ed538-a6a2-408e-b1ad-6564580b5081 admin admin] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:21:35.766680 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d6ed538-a6a2-408e-b1ad-6564580b5081 admin admin] PATH_INFO: `/v3/roles` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:21:35.767435 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0d6ed538-a6a2-408e-b1ad-6564580b5081 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:21:35.770093 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0d6ed538-a6a2-408e-b1ad-6564580b5081 admin admin] RBAC: Authorization granted {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:21:35.783291 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 766/1508] 10.222.0.27 () {58 vars in 1122 bytes} [Thu Oct 13 04:21:35 2022] GET /identity/v3/roles => generated 1420 bytes in 121 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 13 04:21:35.786894 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0d6ed538-a6a2-408e-b1ad-6564580b5081 admin admin] Authenticating user token {{(pid=81409) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:21:35.795179 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bf0cb749-6607-49fe-b594-3a40a30cf0bb admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8ba823f313144da3b683bacef8fd62e0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81408) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:21:35.797198 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf0cb749-6607-49fe-b594-3a40a30cf0bb admin admin] REQUEST_METHOD: `POST` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:21:35.797549 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf0cb749-6607-49fe-b594-3a40a30cf0bb admin admin] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:21:35.797828 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf0cb749-6607-49fe-b594-3a40a30cf0bb admin admin] PATH_INFO: `/v3/users` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:21:35.798503 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bf0cb749-6607-49fe-b594-3a40a30cf0bb admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:21:35.801122 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bf0cb749-6607-49fe-b594-3a40a30cf0bb admin admin] RBAC: Authorization granted {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:21:35.812914 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0d6ed538-a6a2-408e-b1ad-6564580b5081 admin admin] Validating token access rules against request {{(pid=81409) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:21:35.819897 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fb5a3d69-7b0a-42b3-8edc-f36fc2960ec2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8ba823f313144da3b683bacef8fd62e0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81409) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:21:35.820565 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb5a3d69-7b0a-42b3-8edc-f36fc2960ec2 admin admin] REQUEST_METHOD: `PUT` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:21:35.820840 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb5a3d69-7b0a-42b3-8edc-f36fc2960ec2 admin admin] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:21:35.821411 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb5a3d69-7b0a-42b3-8edc-f36fc2960ec2 admin admin] PATH_INFO: `/v3/projects/26a5281b0094454194695d52d3c2677f/users/111440b9f913470e951d6ad45415b30a/roles/ef55fbd94e1040be83695754075a7ed5` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:21:35.835862 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fb5a3d69-7b0a-42b3-8edc-f36fc2960ec2 admin admin] RBAC: Authorizing `identity:create_grant(project_id=26a5281b0094454194695d52d3c2677f, user_id=111440b9f913470e951d6ad45415b30a, role_id=ef55fbd94e1040be83695754075a7ed5)` {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:21:35.837559 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 743/1509] 10.222.0.27 () {60 vars in 1144 bytes} [Thu Oct 13 04:21:35 2022] POST /identity/v3/users => generated 286 bytes in 252 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 13 04:21:35.838549 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fb5a3d69-7b0a-42b3-8edc-f36fc2960ec2 admin admin] RBAC: Authorization granted {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:21:35.842676 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bf0cb749-6607-49fe-b594-3a40a30cf0bb admin admin] Authenticating user token {{(pid=81408) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:21:35.852677 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 767/1510] 10.222.0.27 () {60 vars in 1483 bytes} [Thu Oct 13 04:21:35 2022] PUT /identity/v3/projects/26a5281b0094454194695d52d3c2677f/users/111440b9f913470e951d6ad45415b30a/roles/ef55fbd94e1040be83695754075a7ed5 => generated 0 bytes in 66 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 13 04:21:35.864575 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a005af5-cba3-4a36-9765-2f60bafdc8b9 None None] REQUEST_METHOD: `POST` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:21:35.865271 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a005af5-cba3-4a36-9765-2f60bafdc8b9 None None] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:21:35.865735 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a005af5-cba3-4a36-9765-2f60bafdc8b9 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:21:35.891199 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bf0cb749-6607-49fe-b594-3a40a30cf0bb admin admin] Validating token access rules against request {{(pid=81408) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:21:35.916085 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.auth.core [None req-4a005af5-cba3-4a36-9765-2f60bafdc8b9 None None] MFA Rules not processed for user `111440b9f913470e951d6ad45415b30a`. Rule list: `[]` (Enabled: `True`). {{(pid=81409) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 13 04:21:35.917538 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.fernet_utils [None req-4a005af5-cba3-4a36-9765-2f60bafdc8b9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81409) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:21:35.957496 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG dbcounter [-] [81408] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=81408) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:21:35.982248 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cbfcc38e-70ef-40e9-8ba8-ded6ff18f044 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8ba823f313144da3b683bacef8fd62e0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81408) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:21:35.983110 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cbfcc38e-70ef-40e9-8ba8-ded6ff18f044 admin admin] REQUEST_METHOD: `GET` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:21:35.983598 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cbfcc38e-70ef-40e9-8ba8-ded6ff18f044 admin admin] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:21:35.984064 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cbfcc38e-70ef-40e9-8ba8-ded6ff18f044 admin admin] PATH_INFO: `/v3/roles` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:21:35.985153 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cbfcc38e-70ef-40e9-8ba8-ded6ff18f044 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:21:35.988746 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cbfcc38e-70ef-40e9-8ba8-ded6ff18f044 admin admin] RBAC: Authorization granted {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:21:36.003306 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 744/1511] 10.222.0.27 () {58 vars in 1122 bytes} [Thu Oct 13 04:21:35 2022] GET /identity/v3/roles => generated 1420 bytes in 161 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 13 04:21:36.006880 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cbfcc38e-70ef-40e9-8ba8-ded6ff18f044 admin admin] Authenticating user token {{(pid=81408) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:21:36.020741 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cbfcc38e-70ef-40e9-8ba8-ded6ff18f044 admin admin] Validating token access rules against request {{(pid=81408) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:21:36.030051 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-82b7f0e3-cfa7-4c03-8d13-f3d793618a5d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8ba823f313144da3b683bacef8fd62e0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81408) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:21:36.034278 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 768/1512] 10.222.0.27 () {58 vars in 958 bytes} [Thu Oct 13 04:21:35 2022] POST /identity/v3/auth/tokens => generated 3327 bytes in 167 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 13 04:21:36.035122 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82b7f0e3-cfa7-4c03-8d13-f3d793618a5d admin admin] REQUEST_METHOD: `GET` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:21:36.035587 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82b7f0e3-cfa7-4c03-8d13-f3d793618a5d admin admin] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:21:36.036876 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82b7f0e3-cfa7-4c03-8d13-f3d793618a5d admin admin] PATH_INFO: `/v3/roles` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:21:36.038287 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-82b7f0e3-cfa7-4c03-8d13-f3d793618a5d admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:21:36.042728 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4a005af5-cba3-4a36-9765-2f60bafdc8b9 None None] Authenticating user token {{(pid=81409) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:21:36.043325 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-82b7f0e3-cfa7-4c03-8d13-f3d793618a5d admin admin] RBAC: Authorization granted {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:21:36.045985 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.fernet_utils [None req-4a005af5-cba3-4a36-9765-2f60bafdc8b9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81409) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:21:36.058767 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 745/1513] 10.222.0.27 () {58 vars in 1122 bytes} [Thu Oct 13 04:21:36 2022] GET /identity/v3/roles => generated 1420 bytes in 52 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 13 04:21:36.062631 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-82b7f0e3-cfa7-4c03-8d13-f3d793618a5d admin admin] Authenticating user token {{(pid=81408) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:21:36.079736 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-82b7f0e3-cfa7-4c03-8d13-f3d793618a5d admin admin] Validating token access rules against request {{(pid=81408) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:21:36.097823 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f9ebf8dd-f4c5-46b6-94ab-b7c16a12153d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8ba823f313144da3b683bacef8fd62e0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81408) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:21:36.100191 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9ebf8dd-f4c5-46b6-94ab-b7c16a12153d admin admin] REQUEST_METHOD: `PUT` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:21:36.101200 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9ebf8dd-f4c5-46b6-94ab-b7c16a12153d admin admin] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:21:36.101670 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9ebf8dd-f4c5-46b6-94ab-b7c16a12153d admin admin] PATH_INFO: `/v3/projects/90f0e3b188194915a66e07300a1d7578/users/077c64ce081d4586953e2cfab8af5b9c/roles/cc56c4ae13a7424ea771c15da4b153c1` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:21:36.133982 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f9ebf8dd-f4c5-46b6-94ab-b7c16a12153d admin admin] RBAC: Authorizing `identity:create_grant(project_id=90f0e3b188194915a66e07300a1d7578, user_id=077c64ce081d4586953e2cfab8af5b9c, role_id=cc56c4ae13a7424ea771c15da4b153c1)` {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:21:36.138753 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f9ebf8dd-f4c5-46b6-94ab-b7c16a12153d admin admin] RBAC: Authorization granted {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:21:36.142545 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4a005af5-cba3-4a36-9765-2f60bafdc8b9 None None] Validating token access rules against request {{(pid=81409) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:21:36.149191 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-666051c4-d995-4a83-a2b3-61b50c39f299 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6141b8a6c3d54e179d6879bb019f4d12', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dcec0431fb6849499d6aec35a234daa6', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81409) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:21:36.151538 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-666051c4-d995-4a83-a2b3-61b50c39f299 service cinder] REQUEST_METHOD: `GET` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:21:36.151722 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-666051c4-d995-4a83-a2b3-61b50c39f299 service cinder] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:21:36.152146 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-666051c4-d995-4a83-a2b3-61b50c39f299 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:21:36.165637 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 746/1514] 10.222.0.27 () {60 vars in 1483 bytes} [Thu Oct 13 04:21:36 2022] PUT /identity/v3/projects/90f0e3b188194915a66e07300a1d7578/users/077c64ce081d4586953e2cfab8af5b9c/roles/cc56c4ae13a7424ea771c15da4b153c1 => generated 0 bytes in 103 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 13 04:21:36.167210 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-666051c4-d995-4a83-a2b3-61b50c39f299 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:21:36.170250 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-666051c4-d995-4a83-a2b3-61b50c39f299 service cinder] RBAC: Authorization granted {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:21:36.177763 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e48d491-d4ff-4d6b-9029-4dfa2da93942 None None] REQUEST_METHOD: `POST` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:21:36.178055 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e48d491-d4ff-4d6b-9029-4dfa2da93942 None None] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:21:36.178392 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e48d491-d4ff-4d6b-9029-4dfa2da93942 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:21:36.199912 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 769/1515] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Oct 13 04:21:36 2022] GET /identity/v3/auth/tokens => generated 3327 bytes in 157 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 13 04:21:36.228822 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.auth.core [None req-1e48d491-d4ff-4d6b-9029-4dfa2da93942 None None] MFA Rules not processed for user `077c64ce081d4586953e2cfab8af5b9c`. Rule list: `[]` (Enabled: `True`). {{(pid=81408) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 13 04:21:36.230735 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.fernet_utils [None req-1e48d491-d4ff-4d6b-9029-4dfa2da93942 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81408) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:21:36.328549 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-666051c4-d995-4a83-a2b3-61b50c39f299 service cinder] Authenticating user token {{(pid=81409) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:21:36.343863 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-666051c4-d995-4a83-a2b3-61b50c39f299 service cinder] Validating token access rules against request {{(pid=81409) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:21:36.378741 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 747/1516] 10.222.0.27 () {58 vars in 958 bytes} [Thu Oct 13 04:21:36 2022] POST /identity/v3/auth/tokens => generated 3248 bytes in 199 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 13 04:21:36.381496 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e48d491-d4ff-4d6b-9029-4dfa2da93942 None None] Authenticating user token {{(pid=81408) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:21:36.480049 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f20c1d0b-a771-4ae8-9135-6e54dcf70051 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6141b8a6c3d54e179d6879bb019f4d12', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dcec0431fb6849499d6aec35a234daa6', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81409) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:21:36.480679 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f20c1d0b-a771-4ae8-9135-6e54dcf70051 service cinder] REQUEST_METHOD: `GET` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:21:36.480954 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f20c1d0b-a771-4ae8-9135-6e54dcf70051 service cinder] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:21:36.482126 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f20c1d0b-a771-4ae8-9135-6e54dcf70051 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:21:36.486980 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.fernet_utils [None req-f20c1d0b-a771-4ae8-9135-6e54dcf70051 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81409) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:21:36.508676 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG dbcounter [-] [81409] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81409) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:21:36.549568 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e48d491-d4ff-4d6b-9029-4dfa2da93942 None None] Validating token access rules against request {{(pid=81408) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:21:36.554509 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5d54deaf-8027-4410-b7f8-18fefe8ba62f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8ba823f313144da3b683bacef8fd62e0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81408) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:21:36.555274 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d54deaf-8027-4410-b7f8-18fefe8ba62f admin admin] REQUEST_METHOD: `POST` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:21:36.555657 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d54deaf-8027-4410-b7f8-18fefe8ba62f admin admin] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:21:36.556042 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d54deaf-8027-4410-b7f8-18fefe8ba62f admin admin] PATH_INFO: `/v3/projects` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:21:36.557293 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5d54deaf-8027-4410-b7f8-18fefe8ba62f admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:21:36.559415 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5d54deaf-8027-4410-b7f8-18fefe8ba62f admin admin] RBAC: Authorization granted {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:21:36.607678 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 748/1517] 10.222.0.27 () {60 vars in 1153 bytes} [Thu Oct 13 04:21:36 2022] POST /identity/v3/projects => generated 373 bytes in 227 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 13 04:21:36.608524 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f20c1d0b-a771-4ae8-9135-6e54dcf70051 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:21:36.610311 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f20c1d0b-a771-4ae8-9135-6e54dcf70051 service cinder] RBAC: Authorization granted {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:21:36.612971 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5d54deaf-8027-4410-b7f8-18fefe8ba62f admin admin] Authenticating user token {{(pid=81408) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:21:36.644807 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 770/1518] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Oct 13 04:21:36 2022] GET /identity/v3/auth/tokens => generated 3260 bytes in 320 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Oct 13 04:21:36.668810 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG dbcounter [-] [81408] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=81408) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:21:36.738820 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5d54deaf-8027-4410-b7f8-18fefe8ba62f admin admin] Validating token access rules against request {{(pid=81408) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:21:36.750085 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9083330f-47ee-4fa3-ab81-193c99c4784a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8ba823f313144da3b683bacef8fd62e0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81408) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:21:36.750760 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9083330f-47ee-4fa3-ab81-193c99c4784a admin admin] REQUEST_METHOD: `POST` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:21:36.751107 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9083330f-47ee-4fa3-ab81-193c99c4784a admin admin] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:21:36.751445 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9083330f-47ee-4fa3-ab81-193c99c4784a admin admin] PATH_INFO: `/v3/users` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:21:36.752187 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9083330f-47ee-4fa3-ab81-193c99c4784a admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:21:36.754247 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9083330f-47ee-4fa3-ab81-193c99c4784a admin admin] RBAC: Authorization granted {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:21:36.812420 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 749/1519] 10.222.0.27 () {60 vars in 1144 bytes} [Thu Oct 13 04:21:36 2022] POST /identity/v3/users => generated 293 bytes in 196 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 13 04:21:36.814555 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f20c1d0b-a771-4ae8-9135-6e54dcf70051 service cinder] Authenticating user token {{(pid=81409) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:21:36.846359 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f20c1d0b-a771-4ae8-9135-6e54dcf70051 service cinder] Validating token access rules against request {{(pid=81409) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:21:36.864330 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a61c9431-aea2-4765-ba6d-c9d24a2bd11a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8ba823f313144da3b683bacef8fd62e0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81409) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:21:36.866850 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a61c9431-aea2-4765-ba6d-c9d24a2bd11a admin admin] REQUEST_METHOD: `GET` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:21:36.869427 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a61c9431-aea2-4765-ba6d-c9d24a2bd11a admin admin] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:21:36.869940 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a61c9431-aea2-4765-ba6d-c9d24a2bd11a admin admin] PATH_INFO: `/v3/roles` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:21:36.871903 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a61c9431-aea2-4765-ba6d-c9d24a2bd11a admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:21:36.878030 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a61c9431-aea2-4765-ba6d-c9d24a2bd11a admin admin] RBAC: Authorization granted {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:21:36.907417 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 771/1520] 10.222.0.27 () {58 vars in 1122 bytes} [Thu Oct 13 04:21:36 2022] GET /identity/v3/roles => generated 1420 bytes in 92 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 13 04:21:36.910836 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9083330f-47ee-4fa3-ab81-193c99c4784a admin admin] Authenticating user token {{(pid=81408) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:21:36.916722 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a61c9431-aea2-4765-ba6d-c9d24a2bd11a admin admin] Authenticating user token {{(pid=81409) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:21:36.918528 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.fernet_utils [None req-a61c9431-aea2-4765-ba6d-c9d24a2bd11a admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81409) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:21:36.937664 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9083330f-47ee-4fa3-ab81-193c99c4784a admin admin] Validating token access rules against request {{(pid=81408) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:21:36.952812 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a61c9431-aea2-4765-ba6d-c9d24a2bd11a admin admin] Validating token access rules against request {{(pid=81409) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:21:36.953819 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e1fdfb88-266c-41f6-a265-a3b2ca27c7bd admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8ba823f313144da3b683bacef8fd62e0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81408) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:21:36.956106 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1fdfb88-266c-41f6-a265-a3b2ca27c7bd admin admin] REQUEST_METHOD: `PUT` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:21:36.958068 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1fdfb88-266c-41f6-a265-a3b2ca27c7bd admin admin] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:21:36.958548 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1fdfb88-266c-41f6-a265-a3b2ca27c7bd admin admin] PATH_INFO: `/v3/projects/b701c14392804f2da5513f5b6e50ae35/users/190a831d7e5d4b14b7e72d72b3b4224a/roles/ef55fbd94e1040be83695754075a7ed5` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:21:36.969095 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-863674db-afbf-406d-b2c3-d0551724316e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8ba823f313144da3b683bacef8fd62e0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81409) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:21:36.969756 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-863674db-afbf-406d-b2c3-d0551724316e admin admin] REQUEST_METHOD: `DELETE` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:21:36.970052 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-863674db-afbf-406d-b2c3-d0551724316e admin admin] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:21:36.970326 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-863674db-afbf-406d-b2c3-d0551724316e admin admin] PATH_INFO: `/v3/projects/fb96c1d494324ee9a7cbf76664094cee` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:21:36.972392 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-863674db-afbf-406d-b2c3-d0551724316e admin admin] RBAC: Authorizing `identity:delete_project(project_id=fb96c1d494324ee9a7cbf76664094cee)` {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:21:36.975932 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-863674db-afbf-406d-b2c3-d0551724316e admin admin] RBAC: Authorization granted {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:21:37.004394 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e1fdfb88-266c-41f6-a265-a3b2ca27c7bd admin admin] RBAC: Authorizing `identity:create_grant(project_id=b701c14392804f2da5513f5b6e50ae35, user_id=190a831d7e5d4b14b7e72d72b3b4224a, role_id=ef55fbd94e1040be83695754075a7ed5)` {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:21:37.008257 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e1fdfb88-266c-41f6-a265-a3b2ca27c7bd admin admin] RBAC: Authorization granted {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:21:37.040052 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 750/1521] 10.222.0.27 () {60 vars in 1483 bytes} [Thu Oct 13 04:21:36 2022] PUT /identity/v3/projects/b701c14392804f2da5513f5b6e50ae35/users/190a831d7e5d4b14b7e72d72b3b4224a/roles/ef55fbd94e1040be83695754075a7ed5 => generated 0 bytes in 130 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 13 04:21:37.045497 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e1fdfb88-266c-41f6-a265-a3b2ca27c7bd admin admin] Authenticating user token {{(pid=81408) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:21:37.058950 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.notifications [None req-863674db-afbf-406d-b2c3-d0551724316e admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'fb96c1d494324ee9a7cbf76664094cee', 'request_id': 'req-863674db-afbf-406d-b2c3-d0551724316e'} {{(pid=81409) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 13 04:21:37.073295 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.notifications [None req-863674db-afbf-406d-b2c3-d0551724316e admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'fb96c1d494324ee9a7cbf76664094cee', 'request_id': 'req-863674db-afbf-406d-b2c3-d0551724316e'} {{(pid=81409) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 13 04:21:37.083435 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.notifications [None req-863674db-afbf-406d-b2c3-d0551724316e admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'fb96c1d494324ee9a7cbf76664094cee', 'request_id': 'req-863674db-afbf-406d-b2c3-d0551724316e'} {{(pid=81409) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 13 04:21:37.098022 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.notifications [None req-863674db-afbf-406d-b2c3-d0551724316e admin admin] The token cache is being invalidate because project fb96c1d494324ee9a7cbf76664094cee was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81409) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 13 04:21:37.098644 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.notifications [None req-863674db-afbf-406d-b2c3-d0551724316e admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81409) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 13 04:21:37.099827 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.notifications [None req-863674db-afbf-406d-b2c3-d0551724316e admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81409) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 13 04:21:37.102256 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 772/1522] 10.222.0.27 () {58 vars in 1233 bytes} [Thu Oct 13 04:21:36 2022] DELETE /identity/v3/projects/fb96c1d494324ee9a7cbf76664094cee => generated 0 bytes in 185 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 13 04:21:37.105404 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-863674db-afbf-406d-b2c3-d0551724316e admin admin] Authenticating user token {{(pid=81409) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:21:37.108087 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.fernet_utils [None req-863674db-afbf-406d-b2c3-d0551724316e admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81409) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:21:37.156485 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e1fdfb88-266c-41f6-a265-a3b2ca27c7bd admin admin] Validating token access rules against request {{(pid=81408) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:21:37.227512 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d6047b31-75f8-4664-81d4-2bbc1eae5a8b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8ba823f313144da3b683bacef8fd62e0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81408) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:21:37.228156 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6047b31-75f8-4664-81d4-2bbc1eae5a8b admin admin] REQUEST_METHOD: `GET` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:21:37.228397 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6047b31-75f8-4664-81d4-2bbc1eae5a8b admin admin] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:21:37.228668 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6047b31-75f8-4664-81d4-2bbc1eae5a8b admin admin] PATH_INFO: `/v3/roles` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:21:37.229323 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d6047b31-75f8-4664-81d4-2bbc1eae5a8b admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:21:37.231228 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d6047b31-75f8-4664-81d4-2bbc1eae5a8b admin admin] RBAC: Authorization granted {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:21:37.241022 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-863674db-afbf-406d-b2c3-d0551724316e admin admin] Validating token access rules against request {{(pid=81409) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:21:37.246325 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 751/1523] 10.222.0.27 () {58 vars in 1122 bytes} [Thu Oct 13 04:21:37 2022] GET /identity/v3/roles => generated 1420 bytes in 199 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 13 04:21:37.248258 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-70413cbd-2302-45c5-9fc2-bd6c7cf85ff8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8ba823f313144da3b683bacef8fd62e0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81409) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:21:37.249094 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70413cbd-2302-45c5-9fc2-bd6c7cf85ff8 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:21:37.249501 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70413cbd-2302-45c5-9fc2-bd6c7cf85ff8 admin admin] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:21:37.249924 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70413cbd-2302-45c5-9fc2-bd6c7cf85ff8 admin admin] PATH_INFO: `/v3/users/2100d1f49fb44bc99345bedd2f68ca2d` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:21:37.250033 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d6047b31-75f8-4664-81d4-2bbc1eae5a8b admin admin] Authenticating user token {{(pid=81408) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:21:37.252612 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-70413cbd-2302-45c5-9fc2-bd6c7cf85ff8 admin admin] RBAC: Authorizing `identity:delete_user(user_id=2100d1f49fb44bc99345bedd2f68ca2d)` {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:21:37.252740 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.fernet_utils [None req-d6047b31-75f8-4664-81d4-2bbc1eae5a8b admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81408) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:21:37.253766 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-70413cbd-2302-45c5-9fc2-bd6c7cf85ff8 admin admin] RBAC: Authorization granted {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:21:37.268867 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d6047b31-75f8-4664-81d4-2bbc1eae5a8b admin admin] Validating token access rules against request {{(pid=81408) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:21:37.274592 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-20d6fa13-fbf2-4a49-9b43-17ca45988c16 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8ba823f313144da3b683bacef8fd62e0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81408) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:21:37.276233 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20d6fa13-fbf2-4a49-9b43-17ca45988c16 admin admin] REQUEST_METHOD: `PUT` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:21:37.276664 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20d6fa13-fbf2-4a49-9b43-17ca45988c16 admin admin] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:21:37.276947 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20d6fa13-fbf2-4a49-9b43-17ca45988c16 admin admin] PATH_INFO: `/v3/projects/b701c14392804f2da5513f5b6e50ae35/users/190a831d7e5d4b14b7e72d72b3b4224a/roles/ef55fbd94e1040be83695754075a7ed5` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:21:37.279506 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-20d6fa13-fbf2-4a49-9b43-17ca45988c16 admin admin] RBAC: Authorizing `identity:create_grant(project_id=b701c14392804f2da5513f5b6e50ae35, user_id=190a831d7e5d4b14b7e72d72b3b4224a, role_id=ef55fbd94e1040be83695754075a7ed5)` {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:21:37.282355 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-20d6fa13-fbf2-4a49-9b43-17ca45988c16 admin admin] RBAC: Authorization granted {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:21:37.292763 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG dbcounter [-] [81409] Writing DB stats keystone:SELECT=88,keystone:DELETE=11,keystone:INSERT=1 {{(pid=81409) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:21:37.299416 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 752/1524] 10.222.0.27 () {60 vars in 1483 bytes} [Thu Oct 13 04:21:37 2022] PUT /identity/v3/projects/b701c14392804f2da5513f5b6e50ae35/users/190a831d7e5d4b14b7e72d72b3b4224a/roles/ef55fbd94e1040be83695754075a7ed5 => generated 0 bytes in 50 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 13 04:21:37.318826 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75e4435b-d6c3-4630-965e-19e7d1dfb9f6 None None] REQUEST_METHOD: `POST` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:21:37.320564 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75e4435b-d6c3-4630-965e-19e7d1dfb9f6 None None] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:21:37.320848 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75e4435b-d6c3-4630-965e-19e7d1dfb9f6 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:21:37.346478 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.notifications [None req-70413cbd-2302-45c5-9fc2-bd6c7cf85ff8 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '2100d1f49fb44bc99345bedd2f68ca2d'} {{(pid=81409) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 13 04:21:37.347230 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.notifications [None req-70413cbd-2302-45c5-9fc2-bd6c7cf85ff8 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '2100d1f49fb44bc99345bedd2f68ca2d'} {{(pid=81409) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 13 04:21:37.362756 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.notifications [None req-70413cbd-2302-45c5-9fc2-bd6c7cf85ff8 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '2100d1f49fb44bc99345bedd2f68ca2d'} {{(pid=81409) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 13 04:21:37.364863 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.notifications [None req-70413cbd-2302-45c5-9fc2-bd6c7cf85ff8 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '2100d1f49fb44bc99345bedd2f68ca2d'} {{(pid=81409) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 13 04:21:37.377067 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.auth.core [None req-75e4435b-d6c3-4630-965e-19e7d1dfb9f6 None None] MFA Rules not processed for user `190a831d7e5d4b14b7e72d72b3b4224a`. Rule list: `[]` (Enabled: `True`). {{(pid=81408) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 13 04:21:37.378462 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.fernet_utils [None req-75e4435b-d6c3-4630-965e-19e7d1dfb9f6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81408) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:21:37.378747 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.notifications [None req-70413cbd-2302-45c5-9fc2-bd6c7cf85ff8 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '2100d1f49fb44bc99345bedd2f68ca2d'} {{(pid=81409) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 13 04:21:37.390774 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG dbcounter [-] [81408] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=81408) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:21:37.411785 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 773/1525] 10.222.0.27 () {58 vars in 1224 bytes} [Thu Oct 13 04:21:37 2022] DELETE /identity/v3/users/2100d1f49fb44bc99345bedd2f68ca2d => generated 0 bytes in 307 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 13 04:21:37.507203 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 753/1526] 10.222.0.27 () {58 vars in 958 bytes} [Thu Oct 13 04:21:37 2022] POST /identity/v3/auth/tokens => generated 3317 bytes in 187 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 13 04:21:37.518387 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-70413cbd-2302-45c5-9fc2-bd6c7cf85ff8 admin admin] Authenticating user token {{(pid=81409) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:21:37.519936 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.fernet_utils [None req-70413cbd-2302-45c5-9fc2-bd6c7cf85ff8 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81409) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:21:37.595194 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-70413cbd-2302-45c5-9fc2-bd6c7cf85ff8 admin admin] Validating token access rules against request {{(pid=81409) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:21:37.599408 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fc4def27-2945-4651-a036-de7c34c92bfc service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6141b8a6c3d54e179d6879bb019f4d12', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dcec0431fb6849499d6aec35a234daa6', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81409) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:21:37.600255 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc4def27-2945-4651-a036-de7c34c92bfc service cinder] REQUEST_METHOD: `GET` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:21:37.600755 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc4def27-2945-4651-a036-de7c34c92bfc service cinder] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:21:37.601075 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc4def27-2945-4651-a036-de7c34c92bfc service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:21:37.692906 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fc4def27-2945-4651-a036-de7c34c92bfc service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:21:37.693506 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fc4def27-2945-4651-a036-de7c34c92bfc service cinder] RBAC: Authorization granted {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:21:37.708769 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 774/1527] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Oct 13 04:21:37 2022] GET /identity/v3/auth/tokens => generated 3317 bytes in 191 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 13 04:21:38.416621 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-75e4435b-d6c3-4630-965e-19e7d1dfb9f6 None None] Authenticating user token {{(pid=81408) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:21:38.426565 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.fernet_utils [None req-75e4435b-d6c3-4630-965e-19e7d1dfb9f6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81408) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:21:38.431585 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fc4def27-2945-4651-a036-de7c34c92bfc service cinder] Authenticating user token {{(pid=81409) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:21:38.433780 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.fernet_utils [None req-fc4def27-2945-4651-a036-de7c34c92bfc service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81409) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:21:38.568103 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG dbcounter [-] [81409] Writing DB stats keystone:DELETE=6,keystone:SELECT=93,keystone:INSERT=1 {{(pid=81409) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:21:38.593344 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-75e4435b-d6c3-4630-965e-19e7d1dfb9f6 None None] Validating token access rules against request {{(pid=81408) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:21:38.599223 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fc4def27-2945-4651-a036-de7c34c92bfc service cinder] Validating token access rules against request {{(pid=81409) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:21:38.610244 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1b6395b8-ad8c-48bd-82e8-6e28c8a93dac admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8ba823f313144da3b683bacef8fd62e0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81408) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:21:38.610244 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b6395b8-ad8c-48bd-82e8-6e28c8a93dac admin admin] REQUEST_METHOD: `DELETE` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:21:38.610729 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b6395b8-ad8c-48bd-82e8-6e28c8a93dac admin admin] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:21:38.611105 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b6395b8-ad8c-48bd-82e8-6e28c8a93dac admin admin] PATH_INFO: `/v3/users/077c64ce081d4586953e2cfab8af5b9c` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:21:38.612117 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4e70e655-a9f7-43bd-a033-3836ea66bf64 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8ba823f313144da3b683bacef8fd62e0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81409) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:21:38.612760 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e70e655-a9f7-43bd-a033-3836ea66bf64 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:21:38.613446 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e70e655-a9f7-43bd-a033-3836ea66bf64 admin admin] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:21:38.613446 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e70e655-a9f7-43bd-a033-3836ea66bf64 admin admin] PATH_INFO: `/v3/projects/36d9f444580d4391ae1aa2fe0bee603d` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:21:38.614466 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1b6395b8-ad8c-48bd-82e8-6e28c8a93dac admin admin] RBAC: Authorizing `identity:delete_user(user_id=077c64ce081d4586953e2cfab8af5b9c)` {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:21:38.617286 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4e70e655-a9f7-43bd-a033-3836ea66bf64 admin admin] RBAC: Authorizing `identity:delete_project(project_id=36d9f444580d4391ae1aa2fe0bee603d)` {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:21:38.619504 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1b6395b8-ad8c-48bd-82e8-6e28c8a93dac admin admin] RBAC: Authorization granted {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:21:38.620763 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4e70e655-a9f7-43bd-a033-3836ea66bf64 admin admin] RBAC: Authorization granted {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:21:38.712898 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.notifications [None req-1b6395b8-ad8c-48bd-82e8-6e28c8a93dac admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '077c64ce081d4586953e2cfab8af5b9c'} {{(pid=81408) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 13 04:21:38.726981 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.notifications [None req-1b6395b8-ad8c-48bd-82e8-6e28c8a93dac admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '077c64ce081d4586953e2cfab8af5b9c'} {{(pid=81408) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 13 04:21:38.732329 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.notifications [None req-1b6395b8-ad8c-48bd-82e8-6e28c8a93dac admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '077c64ce081d4586953e2cfab8af5b9c'} {{(pid=81408) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 13 04:21:38.736771 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.notifications [None req-1b6395b8-ad8c-48bd-82e8-6e28c8a93dac admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '077c64ce081d4586953e2cfab8af5b9c'} {{(pid=81408) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 13 04:21:38.759077 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.notifications [None req-1b6395b8-ad8c-48bd-82e8-6e28c8a93dac admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '077c64ce081d4586953e2cfab8af5b9c'} {{(pid=81408) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 13 04:21:38.766020 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.notifications [None req-4e70e655-a9f7-43bd-a033-3836ea66bf64 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '36d9f444580d4391ae1aa2fe0bee603d', 'request_id': 'req-4e70e655-a9f7-43bd-a033-3836ea66bf64'} {{(pid=81409) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 13 04:21:38.785764 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.notifications [None req-4e70e655-a9f7-43bd-a033-3836ea66bf64 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '36d9f444580d4391ae1aa2fe0bee603d', 'request_id': 'req-4e70e655-a9f7-43bd-a033-3836ea66bf64'} {{(pid=81409) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 13 04:21:38.801410 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.notifications [None req-4e70e655-a9f7-43bd-a033-3836ea66bf64 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '36d9f444580d4391ae1aa2fe0bee603d', 'request_id': 'req-4e70e655-a9f7-43bd-a033-3836ea66bf64'} {{(pid=81409) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 13 04:21:38.805222 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 754/1528] 10.222.0.27 () {58 vars in 1224 bytes} [Thu Oct 13 04:21:38 2022] DELETE /identity/v3/users/077c64ce081d4586953e2cfab8af5b9c => generated 0 bytes in 388 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 13 04:21:38.813813 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40e921ba-73ed-420a-aafb-b307032e282f None None] REQUEST_METHOD: `POST` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:21:38.815844 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40e921ba-73ed-420a-aafb-b307032e282f None None] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:21:38.816262 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40e921ba-73ed-420a-aafb-b307032e282f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:21:38.839176 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.notifications [None req-4e70e655-a9f7-43bd-a033-3836ea66bf64 admin admin] The token cache is being invalidate because project 36d9f444580d4391ae1aa2fe0bee603d was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81409) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 13 04:21:38.842762 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.notifications [None req-4e70e655-a9f7-43bd-a033-3836ea66bf64 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81409) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 13 04:21:38.844360 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.notifications [None req-4e70e655-a9f7-43bd-a033-3836ea66bf64 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81409) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 13 04:21:38.848474 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 775/1529] 10.222.0.27 () {58 vars in 1233 bytes} [Thu Oct 13 04:21:38 2022] DELETE /identity/v3/projects/36d9f444580d4391ae1aa2fe0bee603d => generated 0 bytes in 417 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 13 04:21:38.853609 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-871e92bd-350b-4a53-ae34-19617d7fa01e None None] REQUEST_METHOD: `POST` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:21:38.853890 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-871e92bd-350b-4a53-ae34-19617d7fa01e None None] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:21:38.854160 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-871e92bd-350b-4a53-ae34-19617d7fa01e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:21:38.875740 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.auth.core [None req-40e921ba-73ed-420a-aafb-b307032e282f None None] MFA Rules not processed for user `0e2d979563d6421995451a0e86bf3022`. Rule list: `[]` (Enabled: `True`). {{(pid=81408) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 13 04:21:38.878357 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.fernet_utils [None req-40e921ba-73ed-420a-aafb-b307032e282f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81408) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:21:38.909002 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.auth.core [None req-871e92bd-350b-4a53-ae34-19617d7fa01e None None] MFA Rules not processed for user `0e2d979563d6421995451a0e86bf3022`. Rule list: `[]` (Enabled: `True`). {{(pid=81409) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 13 04:21:38.912091 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG dbcounter [-] [81408] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81408) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:21:38.912654 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.fernet_utils [None req-871e92bd-350b-4a53-ae34-19617d7fa01e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81409) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:21:39.031651 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 776/1530] 10.222.0.27 () {58 vars in 958 bytes} [Thu Oct 13 04:21:38 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 179 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 13 04:21:39.034057 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dfd85fe1-ecb9-4b62-9170-cc5ee031247d None None] REQUEST_METHOD: `POST` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:21:39.034381 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dfd85fe1-ecb9-4b62-9170-cc5ee031247d None None] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:21:39.034711 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dfd85fe1-ecb9-4b62-9170-cc5ee031247d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:21:39.048494 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 755/1531] 10.222.0.27 () {58 vars in 958 bytes} [Thu Oct 13 04:21:38 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 237 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 13 04:21:39.064455 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.auth.core [None req-dfd85fe1-ecb9-4b62-9170-cc5ee031247d None None] MFA Rules not processed for user `0e2d979563d6421995451a0e86bf3022`. Rule list: `[]` (Enabled: `True`). {{(pid=81409) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 13 04:21:39.065696 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.fernet_utils [None req-dfd85fe1-ecb9-4b62-9170-cc5ee031247d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81409) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:21:39.068716 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-40e921ba-73ed-420a-aafb-b307032e282f None None] Authenticating user token {{(pid=81408) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:21:39.069629 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.fernet_utils [None req-40e921ba-73ed-420a-aafb-b307032e282f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81408) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:21:39.079205 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 777/1532] 10.222.0.27 () {58 vars in 958 bytes} [Thu Oct 13 04:21:39 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 46 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 13 04:21:39.084642 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dfd85fe1-ecb9-4b62-9170-cc5ee031247d None None] Authenticating user token {{(pid=81409) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:21:39.125460 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dfd85fe1-ecb9-4b62-9170-cc5ee031247d None None] Validating token access rules against request {{(pid=81409) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:21:39.125460 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aa6ece0a-e909-4913-a510-43ba2d0f146d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8ba823f313144da3b683bacef8fd62e0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81409) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:21:39.125460 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa6ece0a-e909-4913-a510-43ba2d0f146d admin admin] REQUEST_METHOD: `GET` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:21:39.125460 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa6ece0a-e909-4913-a510-43ba2d0f146d admin admin] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:21:39.125882 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa6ece0a-e909-4913-a510-43ba2d0f146d admin admin] PATH_INFO: `/v3/domains` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:21:39.125882 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aa6ece0a-e909-4913-a510-43ba2d0f146d admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81409) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 13 04:21:39.125882 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aa6ece0a-e909-4913-a510-43ba2d0f146d admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:21:39.125882 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aa6ece0a-e909-4913-a510-43ba2d0f146d admin admin] RBAC: Authorization granted {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:21:39.154913 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 778/1533] 10.222.0.27 () {58 vars in 1153 bytes} [Thu Oct 13 04:21:39 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 74 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 13 04:21:39.156451 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aa6ece0a-e909-4913-a510-43ba2d0f146d admin admin] Authenticating user token {{(pid=81409) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:21:39.180408 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-40e921ba-73ed-420a-aafb-b307032e282f None None] Validating token access rules against request {{(pid=81408) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:21:39.185040 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-09191907-d718-4de2-bbd4-1991f4836de6 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd8812c3974f84871bb2d761cf0016257', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dcec0431fb6849499d6aec35a234daa6', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81408) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:21:39.185997 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09191907-d718-4de2-bbd4-1991f4836de6 service neutron] REQUEST_METHOD: `GET` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:21:39.186699 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09191907-d718-4de2-bbd4-1991f4836de6 service neutron] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:21:39.186902 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aa6ece0a-e909-4913-a510-43ba2d0f146d admin admin] Validating token access rules against request {{(pid=81409) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:21:39.187311 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09191907-d718-4de2-bbd4-1991f4836de6 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:21:39.192446 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bd394617-0d3d-42bc-96cd-2fddbfda2da5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8ba823f313144da3b683bacef8fd62e0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81409) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:21:39.193200 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd394617-0d3d-42bc-96cd-2fddbfda2da5 admin admin] REQUEST_METHOD: `POST` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:21:39.193579 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd394617-0d3d-42bc-96cd-2fddbfda2da5 admin admin] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:21:39.193951 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd394617-0d3d-42bc-96cd-2fddbfda2da5 admin admin] PATH_INFO: `/v3/projects` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:21:39.195745 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bd394617-0d3d-42bc-96cd-2fddbfda2da5 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:21:39.201044 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bd394617-0d3d-42bc-96cd-2fddbfda2da5 admin admin] RBAC: Authorization granted {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:21:39.204157 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-09191907-d718-4de2-bbd4-1991f4836de6 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:21:39.206136 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-09191907-d718-4de2-bbd4-1991f4836de6 service neutron] RBAC: Authorization granted {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:21:39.222744 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 756/1534] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Oct 13 04:21:39 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 155 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 13 04:21:39.239311 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 779/1535] 10.222.0.27 () {60 vars in 1153 bytes} [Thu Oct 13 04:21:39 2022] POST /identity/v3/projects => generated 367 bytes in 84 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 13 04:21:39.243964 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-09191907-d718-4de2-bbd4-1991f4836de6 service neutron] Authenticating user token {{(pid=81408) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:21:39.258624 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-09191907-d718-4de2-bbd4-1991f4836de6 service neutron] Validating token access rules against request {{(pid=81408) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:21:39.335495 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-41032769-54d7-483e-9418-d98bd0db101f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8ba823f313144da3b683bacef8fd62e0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81408) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:21:39.336201 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41032769-54d7-483e-9418-d98bd0db101f admin admin] REQUEST_METHOD: `POST` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:21:39.336474 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41032769-54d7-483e-9418-d98bd0db101f admin admin] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:21:39.336799 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41032769-54d7-483e-9418-d98bd0db101f admin admin] PATH_INFO: `/v3/users` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:21:39.337464 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-41032769-54d7-483e-9418-d98bd0db101f admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:21:39.340322 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-41032769-54d7-483e-9418-d98bd0db101f admin admin] RBAC: Authorization granted {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:21:39.381949 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bd394617-0d3d-42bc-96cd-2fddbfda2da5 admin admin] Authenticating user token {{(pid=81409) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:21:39.393454 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 757/1536] 10.222.0.27 () {60 vars in 1144 bytes} [Thu Oct 13 04:21:39 2022] POST /identity/v3/users => generated 284 bytes in 150 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 13 04:21:39.395908 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bd394617-0d3d-42bc-96cd-2fddbfda2da5 admin admin] Validating token access rules against request {{(pid=81409) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:21:39.411286 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-78ce88af-76a2-4a82-a88d-6ae67ff2bbe2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8ba823f313144da3b683bacef8fd62e0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81409) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:21:39.414687 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78ce88af-76a2-4a82-a88d-6ae67ff2bbe2 admin admin] REQUEST_METHOD: `GET` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:21:39.414953 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78ce88af-76a2-4a82-a88d-6ae67ff2bbe2 admin admin] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:21:39.415442 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78ce88af-76a2-4a82-a88d-6ae67ff2bbe2 admin admin] PATH_INFO: `/v3/roles` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:21:39.416084 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-78ce88af-76a2-4a82-a88d-6ae67ff2bbe2 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:21:39.418962 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-78ce88af-76a2-4a82-a88d-6ae67ff2bbe2 admin admin] RBAC: Authorization granted {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:21:39.440625 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-41032769-54d7-483e-9418-d98bd0db101f admin admin] Authenticating user token {{(pid=81408) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:21:39.443192 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 780/1537] 10.222.0.27 () {58 vars in 1122 bytes} [Thu Oct 13 04:21:39 2022] GET /identity/v3/roles => generated 1420 bytes in 61 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 13 04:21:39.456209 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-41032769-54d7-483e-9418-d98bd0db101f admin admin] Validating token access rules against request {{(pid=81408) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:21:39.474048 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3b318289-3629-435d-b9e7-e686f9214038 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8ba823f313144da3b683bacef8fd62e0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81408) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:21:39.474613 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b318289-3629-435d-b9e7-e686f9214038 admin admin] REQUEST_METHOD: `GET` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:21:39.475138 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b318289-3629-435d-b9e7-e686f9214038 admin admin] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:21:39.475138 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b318289-3629-435d-b9e7-e686f9214038 admin admin] PATH_INFO: `/v3/roles` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:21:39.475822 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3b318289-3629-435d-b9e7-e686f9214038 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:21:39.477436 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3b318289-3629-435d-b9e7-e686f9214038 admin admin] RBAC: Authorization granted {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:21:39.491324 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 758/1538] 10.222.0.27 () {58 vars in 1122 bytes} [Thu Oct 13 04:21:39 2022] GET /identity/v3/roles => generated 1420 bytes in 53 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Oct 13 04:21:39.495987 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-78ce88af-76a2-4a82-a88d-6ae67ff2bbe2 admin admin] Authenticating user token {{(pid=81409) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:21:39.507231 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-78ce88af-76a2-4a82-a88d-6ae67ff2bbe2 admin admin] Validating token access rules against request {{(pid=81409) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:21:39.513816 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-726c8fdc-a093-49fa-b2da-2477222c72d0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8ba823f313144da3b683bacef8fd62e0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81409) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:21:39.514683 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-726c8fdc-a093-49fa-b2da-2477222c72d0 admin admin] REQUEST_METHOD: `PUT` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:21:39.514960 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-726c8fdc-a093-49fa-b2da-2477222c72d0 admin admin] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:21:39.515225 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-726c8fdc-a093-49fa-b2da-2477222c72d0 admin admin] PATH_INFO: `/v3/projects/55efcb49c8ee4f2abdee379a05261f3e/users/d06360ec39864348a016620181a59739/roles/cc56c4ae13a7424ea771c15da4b153c1` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:21:39.533506 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-726c8fdc-a093-49fa-b2da-2477222c72d0 admin admin] RBAC: Authorizing `identity:create_grant(project_id=55efcb49c8ee4f2abdee379a05261f3e, user_id=d06360ec39864348a016620181a59739, role_id=cc56c4ae13a7424ea771c15da4b153c1)` {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:21:39.536573 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-726c8fdc-a093-49fa-b2da-2477222c72d0 admin admin] RBAC: Authorization granted {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:21:39.551739 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 781/1539] 10.222.0.27 () {60 vars in 1483 bytes} [Thu Oct 13 04:21:39 2022] PUT /identity/v3/projects/55efcb49c8ee4f2abdee379a05261f3e/users/d06360ec39864348a016620181a59739/roles/cc56c4ae13a7424ea771c15da4b153c1 => generated 0 bytes in 56 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 13 04:21:39.562985 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5f647b1-5854-4245-9c9c-d52419b302a1 None None] REQUEST_METHOD: `POST` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:21:39.563268 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5f647b1-5854-4245-9c9c-d52419b302a1 None None] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:21:39.563556 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5f647b1-5854-4245-9c9c-d52419b302a1 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:21:39.600307 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.auth.core [None req-c5f647b1-5854-4245-9c9c-d52419b302a1 None None] MFA Rules not processed for user `d06360ec39864348a016620181a59739`. Rule list: `[]` (Enabled: `True`). {{(pid=81408) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 13 04:21:39.601796 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.fernet_utils [None req-c5f647b1-5854-4245-9c9c-d52419b302a1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81408) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:21:39.648136 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG dbcounter [-] [81408] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81408) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:21:39.688315 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 759/1540] 10.222.0.27 () {58 vars in 958 bytes} [Thu Oct 13 04:21:39 2022] POST /identity/v3/auth/tokens => generated 3244 bytes in 126 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 13 04:21:39.692571 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-726c8fdc-a093-49fa-b2da-2477222c72d0 admin admin] Authenticating user token {{(pid=81409) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:21:39.703448 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG dbcounter [-] [81409] Writing DB stats keystone:SELECT=90,keystone:DELETE=7,keystone:INSERT=3 {{(pid=81409) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:21:39.780997 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c5f647b1-5854-4245-9c9c-d52419b302a1 None None] Authenticating user token {{(pid=81408) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:21:39.782547 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.fernet_utils [None req-c5f647b1-5854-4245-9c9c-d52419b302a1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81408) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:21:39.815708 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-726c8fdc-a093-49fa-b2da-2477222c72d0 admin admin] Validating token access rules against request {{(pid=81409) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:21:39.826523 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f6cc5611-9a51-4ddd-a804-36ddacf0ce21 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8ba823f313144da3b683bacef8fd62e0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81409) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:21:39.827259 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6cc5611-9a51-4ddd-a804-36ddacf0ce21 admin admin] REQUEST_METHOD: `POST` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:21:39.827531 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6cc5611-9a51-4ddd-a804-36ddacf0ce21 admin admin] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:21:39.827799 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6cc5611-9a51-4ddd-a804-36ddacf0ce21 admin admin] PATH_INFO: `/v3/projects` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:21:39.828451 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f6cc5611-9a51-4ddd-a804-36ddacf0ce21 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:21:39.837470 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f6cc5611-9a51-4ddd-a804-36ddacf0ce21 admin admin] RBAC: Authorization granted {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:21:39.905517 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 782/1541] 10.222.0.27 () {60 vars in 1153 bytes} [Thu Oct 13 04:21:39 2022] POST /identity/v3/projects => generated 367 bytes in 213 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 13 04:21:39.910006 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f6cc5611-9a51-4ddd-a804-36ddacf0ce21 admin admin] Authenticating user token {{(pid=81409) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:21:39.944030 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c5f647b1-5854-4245-9c9c-d52419b302a1 None None] Validating token access rules against request {{(pid=81408) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:21:40.023851 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-057e3b92-e54c-4c2d-9a4f-e4e7c9d942ab admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8ba823f313144da3b683bacef8fd62e0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81408) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:21:40.024983 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-057e3b92-e54c-4c2d-9a4f-e4e7c9d942ab admin admin] REQUEST_METHOD: `DELETE` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:21:40.025253 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-057e3b92-e54c-4c2d-9a4f-e4e7c9d942ab admin admin] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:21:40.025522 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-057e3b92-e54c-4c2d-9a4f-e4e7c9d942ab admin admin] PATH_INFO: `/v3/users/ffa3b82f53484687b772e830cd68b461` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:21:40.027107 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-057e3b92-e54c-4c2d-9a4f-e4e7c9d942ab admin admin] RBAC: Authorizing `identity:delete_user(user_id=ffa3b82f53484687b772e830cd68b461)` {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:21:40.030460 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-057e3b92-e54c-4c2d-9a4f-e4e7c9d942ab admin admin] RBAC: Authorization granted {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:21:40.032033 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f6cc5611-9a51-4ddd-a804-36ddacf0ce21 admin admin] Validating token access rules against request {{(pid=81409) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:21:40.038565 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9f743328-4c82-465e-83e8-83f78a6cdd44 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8ba823f313144da3b683bacef8fd62e0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81409) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:21:40.039569 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f743328-4c82-465e-83e8-83f78a6cdd44 admin admin] REQUEST_METHOD: `POST` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:21:40.040073 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f743328-4c82-465e-83e8-83f78a6cdd44 admin admin] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:21:40.040662 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f743328-4c82-465e-83e8-83f78a6cdd44 admin admin] PATH_INFO: `/v3/users` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:21:40.041523 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9f743328-4c82-465e-83e8-83f78a6cdd44 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:21:40.044234 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9f743328-4c82-465e-83e8-83f78a6cdd44 admin admin] RBAC: Authorization granted {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:21:40.074804 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 783/1542] 10.222.0.27 () {60 vars in 1144 bytes} [Thu Oct 13 04:21:39 2022] POST /identity/v3/users => generated 290 bytes in 165 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 13 04:21:40.079821 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9f743328-4c82-465e-83e8-83f78a6cdd44 admin admin] Authenticating user token {{(pid=81409) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:21:40.093125 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9f743328-4c82-465e-83e8-83f78a6cdd44 admin admin] Validating token access rules against request {{(pid=81409) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:21:40.098917 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.notifications [None req-057e3b92-e54c-4c2d-9a4f-e4e7c9d942ab admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'ffa3b82f53484687b772e830cd68b461'} {{(pid=81408) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 13 04:21:40.100441 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8a72b084-b428-4a2c-8ca1-3c190765e745 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8ba823f313144da3b683bacef8fd62e0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81409) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:21:40.101451 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a72b084-b428-4a2c-8ca1-3c190765e745 admin admin] REQUEST_METHOD: `GET` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:21:40.101786 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a72b084-b428-4a2c-8ca1-3c190765e745 admin admin] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:21:40.102235 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a72b084-b428-4a2c-8ca1-3c190765e745 admin admin] PATH_INFO: `/v3/roles` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:21:40.102890 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8a72b084-b428-4a2c-8ca1-3c190765e745 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:21:40.104573 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8a72b084-b428-4a2c-8ca1-3c190765e745 admin admin] RBAC: Authorization granted {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:21:40.112051 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.notifications [None req-057e3b92-e54c-4c2d-9a4f-e4e7c9d942ab admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'ffa3b82f53484687b772e830cd68b461'} {{(pid=81408) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 13 04:21:40.113243 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.notifications [None req-057e3b92-e54c-4c2d-9a4f-e4e7c9d942ab admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'ffa3b82f53484687b772e830cd68b461'} {{(pid=81408) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 13 04:21:40.114166 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.notifications [None req-057e3b92-e54c-4c2d-9a4f-e4e7c9d942ab admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'ffa3b82f53484687b772e830cd68b461'} {{(pid=81408) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 13 04:21:40.118474 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 784/1543] 10.222.0.27 () {58 vars in 1122 bytes} [Thu Oct 13 04:21:40 2022] GET /identity/v3/roles => generated 1420 bytes in 39 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 13 04:21:40.123353 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8a72b084-b428-4a2c-8ca1-3c190765e745 admin admin] Authenticating user token {{(pid=81409) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:21:40.124577 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.notifications [None req-057e3b92-e54c-4c2d-9a4f-e4e7c9d942ab admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'ffa3b82f53484687b772e830cd68b461'} {{(pid=81408) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 13 04:21:40.125414 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.fernet_utils [None req-8a72b084-b428-4a2c-8ca1-3c190765e745 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81409) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:21:40.145270 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG dbcounter [-] [81408] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81408) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:21:40.157096 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 760/1544] 10.222.0.27 () {58 vars in 1224 bytes} [Thu Oct 13 04:21:39 2022] DELETE /identity/v3/users/ffa3b82f53484687b772e830cd68b461 => generated 0 bytes in 372 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 13 04:21:40.158604 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-254b6db5-c4ee-4a1f-a1e4-d44f7c1344a9 None None] REQUEST_METHOD: `POST` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:21:40.158892 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-254b6db5-c4ee-4a1f-a1e4-d44f7c1344a9 None None] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:21:40.159170 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-254b6db5-c4ee-4a1f-a1e4-d44f7c1344a9 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:21:40.163754 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8a72b084-b428-4a2c-8ca1-3c190765e745 admin admin] Validating token access rules against request {{(pid=81409) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:21:40.195916 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.auth.core [None req-254b6db5-c4ee-4a1f-a1e4-d44f7c1344a9 None None] MFA Rules not processed for user `0e2d979563d6421995451a0e86bf3022`. Rule list: `[]` (Enabled: `True`). {{(pid=81408) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 13 04:21:40.197478 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.fernet_utils [None req-254b6db5-c4ee-4a1f-a1e4-d44f7c1344a9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81408) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:21:40.269127 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7807a7be-499b-46b9-aa02-49fc170ca5e2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8ba823f313144da3b683bacef8fd62e0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81409) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:21:40.269983 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7807a7be-499b-46b9-aa02-49fc170ca5e2 admin admin] REQUEST_METHOD: `PUT` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:21:40.271259 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7807a7be-499b-46b9-aa02-49fc170ca5e2 admin admin] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:21:40.271543 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7807a7be-499b-46b9-aa02-49fc170ca5e2 admin admin] PATH_INFO: `/v3/projects/822488016a48437894fbd19cb330ea35/users/bf48436529cc4889916d9a432b95bb84/roles/ef55fbd94e1040be83695754075a7ed5` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:21:40.282415 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 761/1545] 10.222.0.27 () {58 vars in 958 bytes} [Thu Oct 13 04:21:40 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 125 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 13 04:21:40.283395 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-254b6db5-c4ee-4a1f-a1e4-d44f7c1344a9 None None] Authenticating user token {{(pid=81408) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:21:40.292091 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.fernet_utils [None req-254b6db5-c4ee-4a1f-a1e4-d44f7c1344a9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81408) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:21:40.295411 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7807a7be-499b-46b9-aa02-49fc170ca5e2 admin admin] RBAC: Authorizing `identity:create_grant(project_id=822488016a48437894fbd19cb330ea35, user_id=bf48436529cc4889916d9a432b95bb84, role_id=ef55fbd94e1040be83695754075a7ed5)` {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:21:40.298271 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7807a7be-499b-46b9-aa02-49fc170ca5e2 admin admin] RBAC: Authorization granted {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:21:40.301138 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG dbcounter [-] [81409] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=81409) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:21:40.317956 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 785/1546] 10.222.0.27 () {60 vars in 1483 bytes} [Thu Oct 13 04:21:40 2022] PUT /identity/v3/projects/822488016a48437894fbd19cb330ea35/users/bf48436529cc4889916d9a432b95bb84/roles/ef55fbd94e1040be83695754075a7ed5 => generated 0 bytes in 196 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 13 04:21:40.318949 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7807a7be-499b-46b9-aa02-49fc170ca5e2 admin admin] Authenticating user token {{(pid=81409) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:21:40.321963 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.fernet_utils [None req-7807a7be-499b-46b9-aa02-49fc170ca5e2 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81409) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:21:40.323236 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-254b6db5-c4ee-4a1f-a1e4-d44f7c1344a9 None None] Validating token access rules against request {{(pid=81408) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:21:40.420700 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7807a7be-499b-46b9-aa02-49fc170ca5e2 admin admin] Validating token access rules against request {{(pid=81409) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:21:40.421047 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-63f87718-cb8f-418a-afb3-0ba406ff5b9a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8ba823f313144da3b683bacef8fd62e0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81408) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:21:40.422991 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63f87718-cb8f-418a-afb3-0ba406ff5b9a admin admin] REQUEST_METHOD: `DELETE` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:21:40.423680 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63f87718-cb8f-418a-afb3-0ba406ff5b9a admin admin] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:21:40.424097 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63f87718-cb8f-418a-afb3-0ba406ff5b9a admin admin] PATH_INFO: `/v3/projects/90f0e3b188194915a66e07300a1d7578` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:21:40.425027 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e994452e-4a91-4363-981e-d019da62eb54 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd8812c3974f84871bb2d761cf0016257', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dcec0431fb6849499d6aec35a234daa6', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81409) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:21:40.425546 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-63f87718-cb8f-418a-afb3-0ba406ff5b9a admin admin] RBAC: Authorizing `identity:delete_project(project_id=90f0e3b188194915a66e07300a1d7578)` {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:21:40.425768 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e994452e-4a91-4363-981e-d019da62eb54 service neutron] REQUEST_METHOD: `GET` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:21:40.426831 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e994452e-4a91-4363-981e-d019da62eb54 service neutron] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:21:40.427171 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e994452e-4a91-4363-981e-d019da62eb54 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:21:40.428432 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-63f87718-cb8f-418a-afb3-0ba406ff5b9a admin admin] RBAC: Authorization granted {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:21:40.438447 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e994452e-4a91-4363-981e-d019da62eb54 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:21:40.439986 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e994452e-4a91-4363-981e-d019da62eb54 service neutron] RBAC: Authorization granted {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:21:40.452712 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 786/1547] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Oct 13 04:21:40 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 134 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 13 04:21:40.453473 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e994452e-4a91-4363-981e-d019da62eb54 service neutron] Authenticating user token {{(pid=81409) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:21:40.464229 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e994452e-4a91-4363-981e-d019da62eb54 service neutron] Validating token access rules against request {{(pid=81409) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:21:40.479644 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-506918e0-0d72-4be6-b720-371854c6f206 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8ba823f313144da3b683bacef8fd62e0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81409) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:21:40.480460 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-506918e0-0d72-4be6-b720-371854c6f206 admin admin] REQUEST_METHOD: `GET` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:21:40.480883 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-506918e0-0d72-4be6-b720-371854c6f206 admin admin] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:21:40.481573 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-506918e0-0d72-4be6-b720-371854c6f206 admin admin] PATH_INFO: `/v3/roles` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:21:40.482558 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-506918e0-0d72-4be6-b720-371854c6f206 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:21:40.484837 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-506918e0-0d72-4be6-b720-371854c6f206 admin admin] RBAC: Authorization granted {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:21:40.500768 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 787/1548] 10.222.0.27 () {58 vars in 1122 bytes} [Thu Oct 13 04:21:40 2022] GET /identity/v3/roles => generated 1420 bytes in 48 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 13 04:21:40.502179 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b7637ce-5460-4227-9076-eb24857db639 None None] REQUEST_METHOD: `GET` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:21:40.502559 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b7637ce-5460-4227-9076-eb24857db639 None None] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:21:40.502904 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b7637ce-5460-4227-9076-eb24857db639 None None] PATH_INFO: `/` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:21:40.504448 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 788/1549] 10.222.0.27 () {54 vars in 921 bytes} [Thu Oct 13 04:21:40 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 13 04:21:40.505211 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1b7637ce-5460-4227-9076-eb24857db639 None None] Authenticating user token {{(pid=81409) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:21:40.511631 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.notifications [None req-63f87718-cb8f-418a-afb3-0ba406ff5b9a admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '90f0e3b188194915a66e07300a1d7578', 'request_id': 'req-63f87718-cb8f-418a-afb3-0ba406ff5b9a'} {{(pid=81408) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 13 04:21:40.522653 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.notifications [None req-63f87718-cb8f-418a-afb3-0ba406ff5b9a admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '90f0e3b188194915a66e07300a1d7578', 'request_id': 'req-63f87718-cb8f-418a-afb3-0ba406ff5b9a'} {{(pid=81408) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 13 04:21:40.534408 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.notifications [None req-63f87718-cb8f-418a-afb3-0ba406ff5b9a admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '90f0e3b188194915a66e07300a1d7578', 'request_id': 'req-63f87718-cb8f-418a-afb3-0ba406ff5b9a'} {{(pid=81408) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 13 04:21:40.550329 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.notifications [None req-63f87718-cb8f-418a-afb3-0ba406ff5b9a admin admin] The token cache is being invalidate because project 90f0e3b188194915a66e07300a1d7578 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81408) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 13 04:21:40.551058 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.notifications [None req-63f87718-cb8f-418a-afb3-0ba406ff5b9a admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81408) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 13 04:21:40.551845 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.notifications [None req-63f87718-cb8f-418a-afb3-0ba406ff5b9a admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81408) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 13 04:21:40.557088 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 762/1550] 10.222.0.27 () {58 vars in 1233 bytes} [Thu Oct 13 04:21:40 2022] DELETE /identity/v3/projects/90f0e3b188194915a66e07300a1d7578 => generated 0 bytes in 271 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 13 04:21:40.557088 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0b395b7-c0b2-4b50-9d29-4d3fcbb1387a None None] REQUEST_METHOD: `POST` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:21:40.557088 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0b395b7-c0b2-4b50-9d29-4d3fcbb1387a None None] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:21:40.557088 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0b395b7-c0b2-4b50-9d29-4d3fcbb1387a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:21:40.587978 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.auth.core [None req-f0b395b7-c0b2-4b50-9d29-4d3fcbb1387a None None] MFA Rules not processed for user `cb740ed7f7684b079e45f887be83f795`. Rule list: `[]` (Enabled: `True`). {{(pid=81408) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 13 04:21:40.592235 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.fernet_utils [None req-f0b395b7-c0b2-4b50-9d29-4d3fcbb1387a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81408) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:21:40.617425 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1b7637ce-5460-4227-9076-eb24857db639 None None] Validating token access rules against request {{(pid=81409) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:21:40.623275 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d33cc416-e948-42cf-a14a-98a10bf1fe4f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8ba823f313144da3b683bacef8fd62e0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81409) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:21:40.624753 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d33cc416-e948-42cf-a14a-98a10bf1fe4f admin admin] REQUEST_METHOD: `PUT` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:21:40.625203 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d33cc416-e948-42cf-a14a-98a10bf1fe4f admin admin] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:21:40.625708 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d33cc416-e948-42cf-a14a-98a10bf1fe4f admin admin] PATH_INFO: `/v3/projects/822488016a48437894fbd19cb330ea35/users/bf48436529cc4889916d9a432b95bb84/roles/ef55fbd94e1040be83695754075a7ed5` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:21:40.628386 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d33cc416-e948-42cf-a14a-98a10bf1fe4f admin admin] RBAC: Authorizing `identity:create_grant(project_id=822488016a48437894fbd19cb330ea35, user_id=bf48436529cc4889916d9a432b95bb84, role_id=ef55fbd94e1040be83695754075a7ed5)` {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:21:40.632333 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d33cc416-e948-42cf-a14a-98a10bf1fe4f admin admin] RBAC: Authorization granted {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:21:40.634615 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG dbcounter [-] [81408] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=81408) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:21:40.656938 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 789/1551] 10.222.0.27 () {60 vars in 1483 bytes} [Thu Oct 13 04:21:40 2022] PUT /identity/v3/projects/822488016a48437894fbd19cb330ea35/users/bf48436529cc4889916d9a432b95bb84/roles/ef55fbd94e1040be83695754075a7ed5 => generated 0 bytes in 152 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 13 04:21:40.658636 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d33cc416-e948-42cf-a14a-98a10bf1fe4f admin admin] Authenticating user token {{(pid=81409) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:21:40.661309 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.fernet_utils [None req-d33cc416-e948-42cf-a14a-98a10bf1fe4f admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81409) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:21:40.695753 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 763/1552] 10.222.0.27 () {58 vars in 1018 bytes} [Thu Oct 13 04:21:40 2022] POST /identity/v3/auth/tokens => generated 3372 bytes in 140 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 13 04:21:40.705225 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fda849b0-3306-41ef-b9ca-cd6c765c4313 None None] REQUEST_METHOD: `POST` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:21:40.705878 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fda849b0-3306-41ef-b9ca-cd6c765c4313 None None] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:21:40.706314 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fda849b0-3306-41ef-b9ca-cd6c765c4313 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:21:40.773443 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.auth.core [None req-fda849b0-3306-41ef-b9ca-cd6c765c4313 None None] MFA Rules not processed for user `bf48436529cc4889916d9a432b95bb84`. Rule list: `[]` (Enabled: `True`). {{(pid=81408) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 13 04:21:40.779415 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.fernet_utils [None req-fda849b0-3306-41ef-b9ca-cd6c765c4313 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81408) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:21:40.796635 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d33cc416-e948-42cf-a14a-98a10bf1fe4f admin admin] Validating token access rules against request {{(pid=81409) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:21:40.802302 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2ca03cb4-0aab-441d-900c-ec2122091b25 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8ba823f313144da3b683bacef8fd62e0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81409) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:21:40.803576 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ca03cb4-0aab-441d-900c-ec2122091b25 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:21:40.804052 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ca03cb4-0aab-441d-900c-ec2122091b25 admin admin] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:21:40.804703 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ca03cb4-0aab-441d-900c-ec2122091b25 admin admin] PATH_INFO: `/v3/users/190a831d7e5d4b14b7e72d72b3b4224a` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:21:40.806320 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2ca03cb4-0aab-441d-900c-ec2122091b25 admin admin] RBAC: Authorizing `identity:delete_user(user_id=190a831d7e5d4b14b7e72d72b3b4224a)` {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:21:40.808812 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2ca03cb4-0aab-441d-900c-ec2122091b25 admin admin] RBAC: Authorization granted {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:21:40.852714 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG dbcounter [-] [81409] Writing DB stats keystone:INSERT=2,keystone:SELECT=95,keystone:DELETE=3 {{(pid=81409) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:21:40.941039 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.notifications [None req-2ca03cb4-0aab-441d-900c-ec2122091b25 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '190a831d7e5d4b14b7e72d72b3b4224a'} {{(pid=81409) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 13 04:21:40.942159 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.notifications [None req-2ca03cb4-0aab-441d-900c-ec2122091b25 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '190a831d7e5d4b14b7e72d72b3b4224a'} {{(pid=81409) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 13 04:21:40.960416 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.notifications [None req-2ca03cb4-0aab-441d-900c-ec2122091b25 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '190a831d7e5d4b14b7e72d72b3b4224a'} {{(pid=81409) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 13 04:21:40.961552 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.notifications [None req-2ca03cb4-0aab-441d-900c-ec2122091b25 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '190a831d7e5d4b14b7e72d72b3b4224a'} {{(pid=81409) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 13 04:21:40.971582 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.notifications [None req-2ca03cb4-0aab-441d-900c-ec2122091b25 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '190a831d7e5d4b14b7e72d72b3b4224a'} {{(pid=81409) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 13 04:21:40.996768 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 790/1553] 10.222.0.27 () {58 vars in 1224 bytes} [Thu Oct 13 04:21:40 2022] DELETE /identity/v3/users/190a831d7e5d4b14b7e72d72b3b4224a => generated 0 bytes in 339 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 13 04:21:41.002780 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2ca03cb4-0aab-441d-900c-ec2122091b25 admin admin] Authenticating user token {{(pid=81409) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:21:41.004465 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.fernet_utils [None req-2ca03cb4-0aab-441d-900c-ec2122091b25 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81409) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:21:41.014831 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 764/1554] 10.222.0.27 () {58 vars in 958 bytes} [Thu Oct 13 04:21:40 2022] POST /identity/v3/auth/tokens => generated 3311 bytes in 319 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 13 04:21:41.027530 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fda849b0-3306-41ef-b9ca-cd6c765c4313 None None] Authenticating user token {{(pid=81408) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:21:41.029905 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.fernet_utils [None req-fda849b0-3306-41ef-b9ca-cd6c765c4313 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81408) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:21:41.107419 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2ca03cb4-0aab-441d-900c-ec2122091b25 admin admin] Validating token access rules against request {{(pid=81409) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:21:41.122982 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8a84505c-4df8-4b69-9801-afd47835ff48 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cb740ed7f7684b079e45f887be83f795', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dcec0431fb6849499d6aec35a234daa6', 'project_domain_id': 'default', 'roles': ['ResellerAdmin', 'admin', 'reader', 'service', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81409) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:21:41.126450 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a84505c-4df8-4b69-9801-afd47835ff48 service nova] REQUEST_METHOD: `GET` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:21:41.126818 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a84505c-4df8-4b69-9801-afd47835ff48 service nova] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:21:41.127179 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a84505c-4df8-4b69-9801-afd47835ff48 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:21:41.130966 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.fernet_utils [None req-8a84505c-4df8-4b69-9801-afd47835ff48 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81409) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:21:41.136402 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fda849b0-3306-41ef-b9ca-cd6c765c4313 None None] Validating token access rules against request {{(pid=81408) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:21:41.147279 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8a84505c-4df8-4b69-9801-afd47835ff48 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:21:41.149349 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8a84505c-4df8-4b69-9801-afd47835ff48 service nova] RBAC: Authorization granted {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:21:41.166350 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 791/1555] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Oct 13 04:21:40 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 168 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 13 04:21:41.228164 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2854003a-815d-41ed-83eb-2172cdd5e7b2 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6141b8a6c3d54e179d6879bb019f4d12', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dcec0431fb6849499d6aec35a234daa6', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81408) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:21:41.228798 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2854003a-815d-41ed-83eb-2172cdd5e7b2 service cinder] REQUEST_METHOD: `GET` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:21:41.229079 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2854003a-815d-41ed-83eb-2172cdd5e7b2 service cinder] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:21:41.229353 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2854003a-815d-41ed-83eb-2172cdd5e7b2 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:21:41.278969 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG dbcounter [-] [81408] Writing DB stats keystone:SELECT=100 {{(pid=81408) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:21:41.361218 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2854003a-815d-41ed-83eb-2172cdd5e7b2 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:21:41.362698 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2854003a-815d-41ed-83eb-2172cdd5e7b2 service cinder] RBAC: Authorization granted {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:21:41.374764 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 765/1556] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Oct 13 04:21:41 2022] GET /identity/v3/auth/tokens => generated 3311 bytes in 347 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Oct 13 04:21:41.465440 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8a84505c-4df8-4b69-9801-afd47835ff48 service nova] Authenticating user token {{(pid=81409) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:21:41.468333 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.fernet_utils [None req-8a84505c-4df8-4b69-9801-afd47835ff48 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81409) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:21:41.494381 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2854003a-815d-41ed-83eb-2172cdd5e7b2 service cinder] Authenticating user token {{(pid=81408) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:21:41.506161 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2854003a-815d-41ed-83eb-2172cdd5e7b2 service cinder] Validating token access rules against request {{(pid=81408) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:21:41.514952 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-002f8d34-7306-444a-8deb-60621b3ba9f6 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6141b8a6c3d54e179d6879bb019f4d12', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dcec0431fb6849499d6aec35a234daa6', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81408) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:21:41.515629 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-002f8d34-7306-444a-8deb-60621b3ba9f6 service cinder] REQUEST_METHOD: `GET` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:21:41.516087 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-002f8d34-7306-444a-8deb-60621b3ba9f6 service cinder] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:21:41.517191 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-002f8d34-7306-444a-8deb-60621b3ba9f6 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:21:41.521952 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.fernet_utils [None req-002f8d34-7306-444a-8deb-60621b3ba9f6 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81408) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:21:41.611095 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8a84505c-4df8-4b69-9801-afd47835ff48 service nova] Validating token access rules against request {{(pid=81409) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:21:41.616641 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a97f4941-98cc-4114-a76e-f952c27456a5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8ba823f313144da3b683bacef8fd62e0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81409) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:21:41.617302 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a97f4941-98cc-4114-a76e-f952c27456a5 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:21:41.617573 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a97f4941-98cc-4114-a76e-f952c27456a5 admin admin] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:21:41.617837 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a97f4941-98cc-4114-a76e-f952c27456a5 admin admin] PATH_INFO: `/v3/projects/234678ebae1c4d3bbb5917bc7a528eb9` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:21:41.619667 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a97f4941-98cc-4114-a76e-f952c27456a5 admin admin] RBAC: Authorizing `identity:delete_project(project_id=234678ebae1c4d3bbb5917bc7a528eb9)` {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:21:41.621728 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a97f4941-98cc-4114-a76e-f952c27456a5 admin admin] RBAC: Authorization granted {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:21:41.673950 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG dbcounter [-] [81409] Writing DB stats keystone:DELETE=8,keystone:SELECT=91,keystone:INSERT=1 {{(pid=81409) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:21:41.698022 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.notifications [None req-a97f4941-98cc-4114-a76e-f952c27456a5 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '234678ebae1c4d3bbb5917bc7a528eb9', 'request_id': 'req-a97f4941-98cc-4114-a76e-f952c27456a5'} {{(pid=81409) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 13 04:21:41.710910 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.notifications [None req-a97f4941-98cc-4114-a76e-f952c27456a5 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '234678ebae1c4d3bbb5917bc7a528eb9', 'request_id': 'req-a97f4941-98cc-4114-a76e-f952c27456a5'} {{(pid=81409) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 13 04:21:41.720371 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.notifications [None req-a97f4941-98cc-4114-a76e-f952c27456a5 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '234678ebae1c4d3bbb5917bc7a528eb9', 'request_id': 'req-a97f4941-98cc-4114-a76e-f952c27456a5'} {{(pid=81409) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 13 04:21:41.732929 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.notifications [None req-a97f4941-98cc-4114-a76e-f952c27456a5 admin admin] The token cache is being invalidate because project 234678ebae1c4d3bbb5917bc7a528eb9 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81409) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 13 04:21:41.733210 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.notifications [None req-a97f4941-98cc-4114-a76e-f952c27456a5 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81409) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 13 04:21:41.733877 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.notifications [None req-a97f4941-98cc-4114-a76e-f952c27456a5 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81409) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 13 04:21:41.735407 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 792/1557] 10.222.0.27 () {58 vars in 1233 bytes} [Thu Oct 13 04:21:41 2022] DELETE /identity/v3/projects/234678ebae1c4d3bbb5917bc7a528eb9 => generated 0 bytes in 271 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 13 04:21:41.736318 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a97f4941-98cc-4114-a76e-f952c27456a5 admin admin] Authenticating user token {{(pid=81409) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:21:41.737623 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.fernet_utils [None req-a97f4941-98cc-4114-a76e-f952c27456a5 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81409) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:21:41.749365 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-002f8d34-7306-444a-8deb-60621b3ba9f6 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:21:41.750864 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-002f8d34-7306-444a-8deb-60621b3ba9f6 service cinder] RBAC: Authorization granted {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:21:41.760920 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 766/1558] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Oct 13 04:21:41 2022] GET /identity/v3/auth/tokens => generated 3244 bytes in 268 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Oct 13 04:21:41.761768 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-002f8d34-7306-444a-8deb-60621b3ba9f6 service cinder] Authenticating user token {{(pid=81408) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:21:41.775914 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-002f8d34-7306-444a-8deb-60621b3ba9f6 service cinder] Validating token access rules against request {{(pid=81408) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:21:41.839357 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a97f4941-98cc-4114-a76e-f952c27456a5 admin admin] Validating token access rules against request {{(pid=81409) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:21:41.844917 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-81ea0288-a9cb-4f57-981a-a04a7cc1ea53 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8ba823f313144da3b683bacef8fd62e0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81409) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:21:41.845565 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81ea0288-a9cb-4f57-981a-a04a7cc1ea53 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:21:41.845948 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81ea0288-a9cb-4f57-981a-a04a7cc1ea53 admin admin] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:21:41.846165 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81ea0288-a9cb-4f57-981a-a04a7cc1ea53 admin admin] PATH_INFO: `/v3/projects/b701c14392804f2da5513f5b6e50ae35` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:21:41.847500 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-81ea0288-a9cb-4f57-981a-a04a7cc1ea53 admin admin] RBAC: Authorizing `identity:delete_project(project_id=b701c14392804f2da5513f5b6e50ae35)` {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:21:41.851360 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-81ea0288-a9cb-4f57-981a-a04a7cc1ea53 admin admin] RBAC: Authorization granted {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:21:41.854585 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cac0126a-a643-42ee-97d1-f6cae86ec989 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8ba823f313144da3b683bacef8fd62e0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81408) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:21:41.855475 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cac0126a-a643-42ee-97d1-f6cae86ec989 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:21:41.855822 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cac0126a-a643-42ee-97d1-f6cae86ec989 admin admin] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:21:41.856586 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cac0126a-a643-42ee-97d1-f6cae86ec989 admin admin] PATH_INFO: `/v3/users/111440b9f913470e951d6ad45415b30a` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:21:41.858939 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cac0126a-a643-42ee-97d1-f6cae86ec989 admin admin] RBAC: Authorizing `identity:delete_user(user_id=111440b9f913470e951d6ad45415b30a)` {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:21:41.861087 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cac0126a-a643-42ee-97d1-f6cae86ec989 admin admin] RBAC: Authorization granted {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:21:41.922241 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.notifications [None req-81ea0288-a9cb-4f57-981a-a04a7cc1ea53 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'b701c14392804f2da5513f5b6e50ae35', 'request_id': 'req-81ea0288-a9cb-4f57-981a-a04a7cc1ea53'} {{(pid=81409) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 13 04:21:41.933661 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.notifications [None req-81ea0288-a9cb-4f57-981a-a04a7cc1ea53 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'b701c14392804f2da5513f5b6e50ae35', 'request_id': 'req-81ea0288-a9cb-4f57-981a-a04a7cc1ea53'} {{(pid=81409) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 13 04:21:41.947692 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.notifications [None req-cac0126a-a643-42ee-97d1-f6cae86ec989 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '111440b9f913470e951d6ad45415b30a'} {{(pid=81408) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 13 04:21:41.947846 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.notifications [None req-81ea0288-a9cb-4f57-981a-a04a7cc1ea53 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'b701c14392804f2da5513f5b6e50ae35', 'request_id': 'req-81ea0288-a9cb-4f57-981a-a04a7cc1ea53'} {{(pid=81409) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 13 04:21:41.965285 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG dbcounter [-] [81408] Writing DB stats keystone:SELECT=93,keystone:DELETE=7 {{(pid=81408) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:21:41.970911 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.notifications [None req-cac0126a-a643-42ee-97d1-f6cae86ec989 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '111440b9f913470e951d6ad45415b30a'} {{(pid=81408) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 13 04:21:41.973116 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.notifications [None req-cac0126a-a643-42ee-97d1-f6cae86ec989 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '111440b9f913470e951d6ad45415b30a'} {{(pid=81408) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 13 04:21:41.973240 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.notifications [None req-81ea0288-a9cb-4f57-981a-a04a7cc1ea53 admin admin] The token cache is being invalidate because project b701c14392804f2da5513f5b6e50ae35 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81409) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 13 04:21:41.974043 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.notifications [None req-81ea0288-a9cb-4f57-981a-a04a7cc1ea53 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81409) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 13 04:21:41.974256 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.notifications [None req-cac0126a-a643-42ee-97d1-f6cae86ec989 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '111440b9f913470e951d6ad45415b30a'} {{(pid=81408) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 13 04:21:41.976133 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.notifications [None req-81ea0288-a9cb-4f57-981a-a04a7cc1ea53 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81409) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 13 04:21:41.980003 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 793/1559] 10.222.0.27 () {58 vars in 1233 bytes} [Thu Oct 13 04:21:41 2022] DELETE /identity/v3/projects/b701c14392804f2da5513f5b6e50ae35 => generated 0 bytes in 244 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 13 04:21:41.991811 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1b25ac6-1ebc-4231-8ce4-fe97120b1329 None None] REQUEST_METHOD: `POST` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:21:41.992652 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1b25ac6-1ebc-4231-8ce4-fe97120b1329 None None] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:21:41.992882 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.notifications [None req-cac0126a-a643-42ee-97d1-f6cae86ec989 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '111440b9f913470e951d6ad45415b30a'} {{(pid=81408) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 13 04:21:41.993026 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1b25ac6-1ebc-4231-8ce4-fe97120b1329 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:21:42.024484 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.auth.core [None req-c1b25ac6-1ebc-4231-8ce4-fe97120b1329 None None] MFA Rules not processed for user `0e2d979563d6421995451a0e86bf3022`. Rule list: `[]` (Enabled: `True`). {{(pid=81409) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 13 04:21:42.026268 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.fernet_utils [None req-c1b25ac6-1ebc-4231-8ce4-fe97120b1329 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81409) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:21:42.043710 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 767/1560] 10.222.0.27 () {58 vars in 1224 bytes} [Thu Oct 13 04:21:41 2022] DELETE /identity/v3/users/111440b9f913470e951d6ad45415b30a => generated 0 bytes in 282 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 13 04:21:42.126335 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG dbcounter [-] [81409] Writing DB stats keystone:DELETE=13,keystone:SELECT=85,keystone:INSERT=2 {{(pid=81409) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:21:42.214025 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 794/1561] 10.222.0.27 () {58 vars in 958 bytes} [Thu Oct 13 04:21:41 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 223 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 13 04:21:42.216515 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7689f1d6-411a-488f-80e0-86823a35f9b2 None None] REQUEST_METHOD: `POST` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:21:42.216916 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7689f1d6-411a-488f-80e0-86823a35f9b2 None None] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:21:42.217356 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7689f1d6-411a-488f-80e0-86823a35f9b2 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:21:42.290395 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.auth.core [None req-7689f1d6-411a-488f-80e0-86823a35f9b2 None None] MFA Rules not processed for user `0e2d979563d6421995451a0e86bf3022`. Rule list: `[]` (Enabled: `True`). {{(pid=81408) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 13 04:21:42.290395 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.fernet_utils [None req-7689f1d6-411a-488f-80e0-86823a35f9b2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81408) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:21:42.405867 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 768/1562] 10.222.0.27 () {58 vars in 958 bytes} [Thu Oct 13 04:21:42 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 188 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 13 04:21:42.418406 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c1b25ac6-1ebc-4231-8ce4-fe97120b1329 None None] Authenticating user token {{(pid=81409) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:21:42.431587 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c1b25ac6-1ebc-4231-8ce4-fe97120b1329 None None] Validating token access rules against request {{(pid=81409) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:21:42.437521 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-982451ab-ed0c-4e58-9ab4-85a53b27e8b5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8ba823f313144da3b683bacef8fd62e0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81409) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:21:42.438581 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-982451ab-ed0c-4e58-9ab4-85a53b27e8b5 admin admin] REQUEST_METHOD: `GET` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:21:42.439025 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-982451ab-ed0c-4e58-9ab4-85a53b27e8b5 admin admin] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:21:42.439427 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-982451ab-ed0c-4e58-9ab4-85a53b27e8b5 admin admin] PATH_INFO: `/v3/domains` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:21:42.440224 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-982451ab-ed0c-4e58-9ab4-85a53b27e8b5 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81409) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 13 04:21:42.440686 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-982451ab-ed0c-4e58-9ab4-85a53b27e8b5 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:21:42.442470 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-982451ab-ed0c-4e58-9ab4-85a53b27e8b5 admin admin] RBAC: Authorization granted {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:21:42.462001 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7689f1d6-411a-488f-80e0-86823a35f9b2 None None] Authenticating user token {{(pid=81408) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:21:42.469668 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 795/1563] 10.222.0.27 () {58 vars in 1153 bytes} [Thu Oct 13 04:21:42 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 61 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 13 04:21:42.474823 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7689f1d6-411a-488f-80e0-86823a35f9b2 None None] Validating token access rules against request {{(pid=81408) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:21:42.480677 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7dae640b-9f25-4ecb-a1cf-d273e87ae3c7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8ba823f313144da3b683bacef8fd62e0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81408) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:21:42.482087 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7dae640b-9f25-4ecb-a1cf-d273e87ae3c7 admin admin] REQUEST_METHOD: `POST` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:21:42.482576 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7dae640b-9f25-4ecb-a1cf-d273e87ae3c7 admin admin] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:21:42.483054 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7dae640b-9f25-4ecb-a1cf-d273e87ae3c7 admin admin] PATH_INFO: `/v3/projects` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:21:42.483859 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7dae640b-9f25-4ecb-a1cf-d273e87ae3c7 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:21:42.485963 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7dae640b-9f25-4ecb-a1cf-d273e87ae3c7 admin admin] RBAC: Authorization granted {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:21:42.530444 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-982451ab-ed0c-4e58-9ab4-85a53b27e8b5 admin admin] Authenticating user token {{(pid=81409) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:21:42.539987 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 769/1564] 10.222.0.27 () {60 vars in 1153 bytes} [Thu Oct 13 04:21:42 2022] POST /identity/v3/projects => generated 365 bytes in 77 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 13 04:21:42.545115 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-982451ab-ed0c-4e58-9ab4-85a53b27e8b5 admin admin] Validating token access rules against request {{(pid=81409) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:21:42.615780 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b1012f69-dc6f-4a5f-9a82-8ccb95ae780e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8ba823f313144da3b683bacef8fd62e0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81409) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:21:42.616549 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1012f69-dc6f-4a5f-9a82-8ccb95ae780e admin admin] REQUEST_METHOD: `POST` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:21:42.616926 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1012f69-dc6f-4a5f-9a82-8ccb95ae780e admin admin] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:21:42.617303 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1012f69-dc6f-4a5f-9a82-8ccb95ae780e admin admin] PATH_INFO: `/v3/users` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:21:42.618046 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b1012f69-dc6f-4a5f-9a82-8ccb95ae780e admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:21:42.620128 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b1012f69-dc6f-4a5f-9a82-8ccb95ae780e admin admin] RBAC: Authorization granted {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:21:42.645051 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 796/1565] 10.222.0.27 () {60 vars in 1144 bytes} [Thu Oct 13 04:21:42 2022] POST /identity/v3/users => generated 283 bytes in 115 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 13 04:21:42.649160 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7dae640b-9f25-4ecb-a1cf-d273e87ae3c7 admin admin] Authenticating user token {{(pid=81408) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:21:42.659453 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7dae640b-9f25-4ecb-a1cf-d273e87ae3c7 admin admin] Validating token access rules against request {{(pid=81408) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:21:42.665483 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ae22d6ad-4907-4523-ae6f-3380ef8606ec admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8ba823f313144da3b683bacef8fd62e0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81408) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:21:42.665992 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae22d6ad-4907-4523-ae6f-3380ef8606ec admin admin] REQUEST_METHOD: `GET` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:21:42.666631 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae22d6ad-4907-4523-ae6f-3380ef8606ec admin admin] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:21:42.666949 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae22d6ad-4907-4523-ae6f-3380ef8606ec admin admin] PATH_INFO: `/v3/roles` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:21:42.667922 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ae22d6ad-4907-4523-ae6f-3380ef8606ec admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:21:42.670629 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ae22d6ad-4907-4523-ae6f-3380ef8606ec admin admin] RBAC: Authorization granted {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:21:42.681843 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 770/1566] 10.222.0.27 () {58 vars in 1122 bytes} [Thu Oct 13 04:21:42 2022] GET /identity/v3/roles => generated 1420 bytes in 33 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 13 04:21:42.685985 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b1012f69-dc6f-4a5f-9a82-8ccb95ae780e admin admin] Authenticating user token {{(pid=81409) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:21:42.696478 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b1012f69-dc6f-4a5f-9a82-8ccb95ae780e admin admin] Validating token access rules against request {{(pid=81409) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:21:42.709301 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-305baea9-1fa0-4dcb-8949-92a7102bf582 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8ba823f313144da3b683bacef8fd62e0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81409) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:21:42.710139 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-305baea9-1fa0-4dcb-8949-92a7102bf582 admin admin] REQUEST_METHOD: `GET` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:21:42.710463 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-305baea9-1fa0-4dcb-8949-92a7102bf582 admin admin] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:21:42.710799 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-305baea9-1fa0-4dcb-8949-92a7102bf582 admin admin] PATH_INFO: `/v3/roles` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:21:42.713396 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-305baea9-1fa0-4dcb-8949-92a7102bf582 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:21:42.715506 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-305baea9-1fa0-4dcb-8949-92a7102bf582 admin admin] RBAC: Authorization granted {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:21:42.734774 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 797/1567] 10.222.0.27 () {58 vars in 1122 bytes} [Thu Oct 13 04:21:42 2022] GET /identity/v3/roles => generated 1420 bytes in 49 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 13 04:21:42.741690 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ae22d6ad-4907-4523-ae6f-3380ef8606ec admin admin] Authenticating user token {{(pid=81408) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:21:42.757164 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ae22d6ad-4907-4523-ae6f-3380ef8606ec admin admin] Validating token access rules against request {{(pid=81408) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:21:42.762641 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-13ac7444-1383-441b-9d06-1b9e9a941248 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8ba823f313144da3b683bacef8fd62e0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81408) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:21:42.763249 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13ac7444-1383-441b-9d06-1b9e9a941248 admin admin] REQUEST_METHOD: `PUT` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:21:42.763430 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13ac7444-1383-441b-9d06-1b9e9a941248 admin admin] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:21:42.763966 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13ac7444-1383-441b-9d06-1b9e9a941248 admin admin] PATH_INFO: `/v3/projects/8d029efee04a45549aa744c9c7e09acf/users/c7d32e81e9ad4563a4d6b7f3ef8456df/roles/cc56c4ae13a7424ea771c15da4b153c1` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:21:42.782018 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-13ac7444-1383-441b-9d06-1b9e9a941248 admin admin] RBAC: Authorizing `identity:create_grant(project_id=8d029efee04a45549aa744c9c7e09acf, user_id=c7d32e81e9ad4563a4d6b7f3ef8456df, role_id=cc56c4ae13a7424ea771c15da4b153c1)` {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:21:42.785130 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-13ac7444-1383-441b-9d06-1b9e9a941248 admin admin] RBAC: Authorization granted {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:21:42.813412 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 771/1568] 10.222.0.27 () {60 vars in 1483 bytes} [Thu Oct 13 04:21:42 2022] PUT /identity/v3/projects/8d029efee04a45549aa744c9c7e09acf/users/c7d32e81e9ad4563a4d6b7f3ef8456df/roles/cc56c4ae13a7424ea771c15da4b153c1 => generated 0 bytes in 64 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 13 04:21:42.815201 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d867aca-af2f-4b77-a4a1-451887b667bf None None] REQUEST_METHOD: `POST` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:21:42.815613 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d867aca-af2f-4b77-a4a1-451887b667bf None None] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:21:42.815886 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d867aca-af2f-4b77-a4a1-451887b667bf None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:21:42.845724 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.auth.core [None req-1d867aca-af2f-4b77-a4a1-451887b667bf None None] MFA Rules not processed for user `c7d32e81e9ad4563a4d6b7f3ef8456df`. Rule list: `[]` (Enabled: `True`). {{(pid=81409) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 13 04:21:42.846987 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.fernet_utils [None req-1d867aca-af2f-4b77-a4a1-451887b667bf None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81409) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:21:42.927280 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 798/1569] 10.222.0.27 () {58 vars in 958 bytes} [Thu Oct 13 04:21:42 2022] POST /identity/v3/auth/tokens => generated 3242 bytes in 113 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 13 04:21:42.931911 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-13ac7444-1383-441b-9d06-1b9e9a941248 admin admin] Authenticating user token {{(pid=81408) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:21:43.020823 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-13ac7444-1383-441b-9d06-1b9e9a941248 admin admin] Validating token access rules against request {{(pid=81408) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:21:43.026634 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-328f3cab-07d9-4dd6-ba13-7a98ca91a5dd admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8ba823f313144da3b683bacef8fd62e0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81408) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:21:43.027953 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-328f3cab-07d9-4dd6-ba13-7a98ca91a5dd admin admin] REQUEST_METHOD: `POST` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:21:43.028706 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-328f3cab-07d9-4dd6-ba13-7a98ca91a5dd admin admin] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:21:43.029142 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-328f3cab-07d9-4dd6-ba13-7a98ca91a5dd admin admin] PATH_INFO: `/v3/projects` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:21:43.030300 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-328f3cab-07d9-4dd6-ba13-7a98ca91a5dd admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:21:43.033164 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-328f3cab-07d9-4dd6-ba13-7a98ca91a5dd admin admin] RBAC: Authorization granted {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:21:43.063703 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG dbcounter [-] [81408] Writing DB stats keystone:SELECT=93,keystone:INSERT=4,keystone:DELETE=3 {{(pid=81408) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:21:43.071818 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 772/1570] 10.222.0.27 () {60 vars in 1153 bytes} [Thu Oct 13 04:21:42 2022] POST /identity/v3/projects => generated 365 bytes in 140 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 13 04:21:43.076720 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1d867aca-af2f-4b77-a4a1-451887b667bf None None] Authenticating user token {{(pid=81409) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:21:43.093216 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1d867aca-af2f-4b77-a4a1-451887b667bf None None] Validating token access rules against request {{(pid=81409) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:21:43.157081 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG dbcounter [-] [81409] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81409) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:21:43.188740 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-863d2e92-e69c-4600-bd99-0e9c51024b6d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8ba823f313144da3b683bacef8fd62e0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81409) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:21:43.189640 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-863d2e92-e69c-4600-bd99-0e9c51024b6d admin admin] REQUEST_METHOD: `POST` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:21:43.190237 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-863d2e92-e69c-4600-bd99-0e9c51024b6d admin admin] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:21:43.190649 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-863d2e92-e69c-4600-bd99-0e9c51024b6d admin admin] PATH_INFO: `/v3/users` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:21:43.191617 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-863d2e92-e69c-4600-bd99-0e9c51024b6d admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:21:43.194177 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-863d2e92-e69c-4600-bd99-0e9c51024b6d admin admin] RBAC: Authorization granted {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:21:43.228652 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 799/1571] 10.222.0.27 () {60 vars in 1144 bytes} [Thu Oct 13 04:21:43 2022] POST /identity/v3/users => generated 289 bytes in 153 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 13 04:21:43.234608 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-328f3cab-07d9-4dd6-ba13-7a98ca91a5dd admin admin] Authenticating user token {{(pid=81408) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:21:43.254106 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-328f3cab-07d9-4dd6-ba13-7a98ca91a5dd admin admin] Validating token access rules against request {{(pid=81408) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:21:43.270533 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-50cf94d4-91e8-4db7-9311-6733d33df84e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8ba823f313144da3b683bacef8fd62e0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81408) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:21:43.271795 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50cf94d4-91e8-4db7-9311-6733d33df84e admin admin] REQUEST_METHOD: `GET` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:21:43.272085 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50cf94d4-91e8-4db7-9311-6733d33df84e admin admin] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:21:43.272359 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50cf94d4-91e8-4db7-9311-6733d33df84e admin admin] PATH_INFO: `/v3/roles` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:21:43.276020 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-50cf94d4-91e8-4db7-9311-6733d33df84e admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:21:43.282286 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-50cf94d4-91e8-4db7-9311-6733d33df84e admin admin] RBAC: Authorization granted {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:21:43.318858 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 773/1572] 10.222.0.27 () {58 vars in 1122 bytes} [Thu Oct 13 04:21:43 2022] GET /identity/v3/roles => generated 1420 bytes in 85 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 13 04:21:43.320211 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-863d2e92-e69c-4600-bd99-0e9c51024b6d admin admin] Authenticating user token {{(pid=81409) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:21:43.346046 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-863d2e92-e69c-4600-bd99-0e9c51024b6d admin admin] Validating token access rules against request {{(pid=81409) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:21:43.354986 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d062d86d-ffca-416d-9f2a-44520e151aea admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8ba823f313144da3b683bacef8fd62e0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81409) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:21:43.355671 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d062d86d-ffca-416d-9f2a-44520e151aea admin admin] REQUEST_METHOD: `PUT` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:21:43.355938 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d062d86d-ffca-416d-9f2a-44520e151aea admin admin] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:21:43.362543 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d062d86d-ffca-416d-9f2a-44520e151aea admin admin] PATH_INFO: `/v3/projects/28438e126c82422882635ce40802a418/users/dd72d097cd8642b48de6e3bc2753d2f9/roles/ef55fbd94e1040be83695754075a7ed5` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:21:43.423676 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d062d86d-ffca-416d-9f2a-44520e151aea admin admin] RBAC: Authorizing `identity:create_grant(project_id=28438e126c82422882635ce40802a418, user_id=dd72d097cd8642b48de6e3bc2753d2f9, role_id=ef55fbd94e1040be83695754075a7ed5)` {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:21:43.426720 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d062d86d-ffca-416d-9f2a-44520e151aea admin admin] RBAC: Authorization granted {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:21:43.469980 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 800/1573] 10.222.0.27 () {60 vars in 1483 bytes} [Thu Oct 13 04:21:43 2022] PUT /identity/v3/projects/28438e126c82422882635ce40802a418/users/dd72d097cd8642b48de6e3bc2753d2f9/roles/ef55fbd94e1040be83695754075a7ed5 => generated 0 bytes in 147 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 13 04:21:43.470226 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-50cf94d4-91e8-4db7-9311-6733d33df84e admin admin] Authenticating user token {{(pid=81408) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:21:43.485042 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-50cf94d4-91e8-4db7-9311-6733d33df84e admin admin] Validating token access rules against request {{(pid=81408) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:21:43.529947 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d062d86d-ffca-416d-9f2a-44520e151aea admin admin] Authenticating user token {{(pid=81409) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:21:43.534055 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.fernet_utils [None req-d062d86d-ffca-416d-9f2a-44520e151aea admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81409) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:21:43.641061 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0ae34c58-a022-4b25-8a63-f6fdda64b0bc admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8ba823f313144da3b683bacef8fd62e0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81408) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:21:43.643071 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ae34c58-a022-4b25-8a63-f6fdda64b0bc admin admin] REQUEST_METHOD: `GET` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:21:43.643359 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ae34c58-a022-4b25-8a63-f6fdda64b0bc admin admin] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:21:43.643639 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ae34c58-a022-4b25-8a63-f6fdda64b0bc admin admin] PATH_INFO: `/v3/roles` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:21:43.646691 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0ae34c58-a022-4b25-8a63-f6fdda64b0bc admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:21:43.650488 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0ae34c58-a022-4b25-8a63-f6fdda64b0bc admin admin] RBAC: Authorization granted {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:21:43.661132 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d062d86d-ffca-416d-9f2a-44520e151aea admin admin] Validating token access rules against request {{(pid=81409) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:21:43.667441 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-01cb5ea6-5eda-4604-bb79-8ba5c1bf7066 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8ba823f313144da3b683bacef8fd62e0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81409) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:21:43.668275 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01cb5ea6-5eda-4604-bb79-8ba5c1bf7066 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:21:43.669246 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01cb5ea6-5eda-4604-bb79-8ba5c1bf7066 admin admin] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:21:43.669760 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01cb5ea6-5eda-4604-bb79-8ba5c1bf7066 admin admin] PATH_INFO: `/v3/projects/26a5281b0094454194695d52d3c2677f` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:21:43.671833 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-01cb5ea6-5eda-4604-bb79-8ba5c1bf7066 admin admin] RBAC: Authorizing `identity:delete_project(project_id=26a5281b0094454194695d52d3c2677f)` {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:21:43.675815 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-01cb5ea6-5eda-4604-bb79-8ba5c1bf7066 admin admin] RBAC: Authorization granted {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:21:43.677192 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 774/1574] 10.222.0.27 () {58 vars in 1122 bytes} [Thu Oct 13 04:21:43 2022] GET /identity/v3/roles => generated 1420 bytes in 207 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 13 04:21:43.680951 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0ae34c58-a022-4b25-8a63-f6fdda64b0bc admin admin] Authenticating user token {{(pid=81408) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:21:43.705044 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0ae34c58-a022-4b25-8a63-f6fdda64b0bc admin admin] Validating token access rules against request {{(pid=81408) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:21:43.710122 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-62a5a0d3-4a14-4df7-8a25-f447a42d060b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8ba823f313144da3b683bacef8fd62e0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81408) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:21:43.710844 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62a5a0d3-4a14-4df7-8a25-f447a42d060b admin admin] REQUEST_METHOD: `PUT` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:21:43.711127 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62a5a0d3-4a14-4df7-8a25-f447a42d060b admin admin] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:21:43.711401 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62a5a0d3-4a14-4df7-8a25-f447a42d060b admin admin] PATH_INFO: `/v3/projects/28438e126c82422882635ce40802a418/users/dd72d097cd8642b48de6e3bc2753d2f9/roles/ef55fbd94e1040be83695754075a7ed5` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:21:43.713421 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-62a5a0d3-4a14-4df7-8a25-f447a42d060b admin admin] RBAC: Authorizing `identity:create_grant(project_id=28438e126c82422882635ce40802a418, user_id=dd72d097cd8642b48de6e3bc2753d2f9, role_id=ef55fbd94e1040be83695754075a7ed5)` {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:21:43.716821 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-62a5a0d3-4a14-4df7-8a25-f447a42d060b admin admin] RBAC: Authorization granted {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:21:43.742250 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 775/1575] 10.222.0.27 () {60 vars in 1483 bytes} [Thu Oct 13 04:21:43 2022] PUT /identity/v3/projects/28438e126c82422882635ce40802a418/users/dd72d097cd8642b48de6e3bc2753d2f9/roles/ef55fbd94e1040be83695754075a7ed5 => generated 0 bytes in 53 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 13 04:21:43.743927 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f69d127-3cfd-4fd6-924d-74b0746e7d2b None None] REQUEST_METHOD: `POST` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:21:43.744283 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f69d127-3cfd-4fd6-924d-74b0746e7d2b None None] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:21:43.744649 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f69d127-3cfd-4fd6-924d-74b0746e7d2b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:21:43.753950 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.notifications [None req-01cb5ea6-5eda-4604-bb79-8ba5c1bf7066 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '26a5281b0094454194695d52d3c2677f', 'request_id': 'req-01cb5ea6-5eda-4604-bb79-8ba5c1bf7066'} {{(pid=81409) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 13 04:21:43.767210 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.notifications [None req-01cb5ea6-5eda-4604-bb79-8ba5c1bf7066 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '26a5281b0094454194695d52d3c2677f', 'request_id': 'req-01cb5ea6-5eda-4604-bb79-8ba5c1bf7066'} {{(pid=81409) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 13 04:21:43.777150 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.notifications [None req-01cb5ea6-5eda-4604-bb79-8ba5c1bf7066 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '26a5281b0094454194695d52d3c2677f', 'request_id': 'req-01cb5ea6-5eda-4604-bb79-8ba5c1bf7066'} {{(pid=81409) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 13 04:21:43.799743 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.auth.core [None req-7f69d127-3cfd-4fd6-924d-74b0746e7d2b None None] MFA Rules not processed for user `dd72d097cd8642b48de6e3bc2753d2f9`. Rule list: `[]` (Enabled: `True`). {{(pid=81408) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 13 04:21:43.803016 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.fernet_utils [None req-7f69d127-3cfd-4fd6-924d-74b0746e7d2b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81408) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:21:43.805219 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.notifications [None req-01cb5ea6-5eda-4604-bb79-8ba5c1bf7066 admin admin] The token cache is being invalidate because project 26a5281b0094454194695d52d3c2677f was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81409) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 13 04:21:43.805521 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.notifications [None req-01cb5ea6-5eda-4604-bb79-8ba5c1bf7066 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81409) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 13 04:21:43.807890 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.notifications [None req-01cb5ea6-5eda-4604-bb79-8ba5c1bf7066 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81409) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 13 04:21:43.810755 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 801/1576] 10.222.0.27 () {58 vars in 1233 bytes} [Thu Oct 13 04:21:43 2022] DELETE /identity/v3/projects/26a5281b0094454194695d52d3c2677f => generated 0 bytes in 281 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 13 04:21:43.814819 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a77d1ba9-0e1f-4428-bcc4-36e1e2b78929 None None] REQUEST_METHOD: `POST` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:21:43.815107 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a77d1ba9-0e1f-4428-bcc4-36e1e2b78929 None None] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:21:43.816768 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a77d1ba9-0e1f-4428-bcc4-36e1e2b78929 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:21:43.880562 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.auth.core [None req-a77d1ba9-0e1f-4428-bcc4-36e1e2b78929 None None] MFA Rules not processed for user `0e2d979563d6421995451a0e86bf3022`. Rule list: `[]` (Enabled: `True`). {{(pid=81409) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 13 04:21:43.881971 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.fernet_utils [None req-a77d1ba9-0e1f-4428-bcc4-36e1e2b78929 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81409) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:21:43.930821 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG dbcounter [-] [81409] Writing DB stats keystone:SELECT=88,keystone:INSERT=5,keystone:DELETE=7 {{(pid=81409) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:21:43.945842 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 776/1577] 10.222.0.27 () {58 vars in 958 bytes} [Thu Oct 13 04:21:43 2022] POST /identity/v3/auth/tokens => generated 3309 bytes in 200 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 13 04:21:43.953197 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7f69d127-3cfd-4fd6-924d-74b0746e7d2b None None] Authenticating user token {{(pid=81408) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:21:43.954719 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.fernet_utils [None req-7f69d127-3cfd-4fd6-924d-74b0746e7d2b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81408) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:21:43.972327 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 802/1578] 10.222.0.27 () {58 vars in 958 bytes} [Thu Oct 13 04:21:43 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 158 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 13 04:21:43.979943 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3f84e11-bc32-4aed-9ef8-ffc73048a593 None None] REQUEST_METHOD: `POST` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:21:43.980225 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3f84e11-bc32-4aed-9ef8-ffc73048a593 None None] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:21:43.980489 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3f84e11-bc32-4aed-9ef8-ffc73048a593 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:21:44.014236 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.auth.core [None req-e3f84e11-bc32-4aed-9ef8-ffc73048a593 None None] MFA Rules not processed for user `0e2d979563d6421995451a0e86bf3022`. Rule list: `[]` (Enabled: `True`). {{(pid=81409) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 13 04:21:44.016278 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.fernet_utils [None req-e3f84e11-bc32-4aed-9ef8-ffc73048a593 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81409) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:21:44.032263 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7f69d127-3cfd-4fd6-924d-74b0746e7d2b None None] Validating token access rules against request {{(pid=81408) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:21:44.032541 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 803/1579] 10.222.0.27 () {58 vars in 958 bytes} [Thu Oct 13 04:21:43 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 53 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 13 04:21:44.036524 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e3f84e11-bc32-4aed-9ef8-ffc73048a593 None None] Authenticating user token {{(pid=81409) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:21:44.037533 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a2b49d08-a33d-4e7a-8889-d0ad837479f5 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6141b8a6c3d54e179d6879bb019f4d12', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dcec0431fb6849499d6aec35a234daa6', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81408) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:21:44.038162 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2b49d08-a33d-4e7a-8889-d0ad837479f5 service cinder] REQUEST_METHOD: `GET` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:21:44.038504 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2b49d08-a33d-4e7a-8889-d0ad837479f5 service cinder] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:21:44.038756 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2b49d08-a33d-4e7a-8889-d0ad837479f5 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:21:44.052203 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e3f84e11-bc32-4aed-9ef8-ffc73048a593 None None] Validating token access rules against request {{(pid=81409) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:21:44.052367 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a2b49d08-a33d-4e7a-8889-d0ad837479f5 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:21:44.053746 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a2b49d08-a33d-4e7a-8889-d0ad837479f5 service cinder] RBAC: Authorization granted {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:21:44.060004 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG dbcounter [-] [81408] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81408) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:21:44.063631 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-28d6a32e-c34c-4a12-8cd9-485e0feac3a9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8ba823f313144da3b683bacef8fd62e0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81409) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:21:44.064448 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28d6a32e-c34c-4a12-8cd9-485e0feac3a9 admin admin] REQUEST_METHOD: `GET` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:21:44.064821 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28d6a32e-c34c-4a12-8cd9-485e0feac3a9 admin admin] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:21:44.065190 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28d6a32e-c34c-4a12-8cd9-485e0feac3a9 admin admin] PATH_INFO: `/v3/domains` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:21:44.065943 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-28d6a32e-c34c-4a12-8cd9-485e0feac3a9 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81409) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 13 04:21:44.066452 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-28d6a32e-c34c-4a12-8cd9-485e0feac3a9 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:21:44.068272 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-28d6a32e-c34c-4a12-8cd9-485e0feac3a9 admin admin] RBAC: Authorization granted {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:21:44.080604 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 777/1580] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Oct 13 04:21:43 2022] GET /identity/v3/auth/tokens => generated 3309 bytes in 128 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Oct 13 04:21:44.090384 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 804/1581] 10.222.0.27 () {58 vars in 1153 bytes} [Thu Oct 13 04:21:44 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 54 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 13 04:21:44.092654 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a2b49d08-a33d-4e7a-8889-d0ad837479f5 service cinder] Authenticating user token {{(pid=81408) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:21:44.108310 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a2b49d08-a33d-4e7a-8889-d0ad837479f5 service cinder] Validating token access rules against request {{(pid=81408) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:21:44.115585 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9a519439-b50e-4e43-a336-45ab0b43054b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8ba823f313144da3b683bacef8fd62e0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81408) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:21:44.116205 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a519439-b50e-4e43-a336-45ab0b43054b admin admin] REQUEST_METHOD: `POST` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:21:44.116482 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a519439-b50e-4e43-a336-45ab0b43054b admin admin] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:21:44.116767 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a519439-b50e-4e43-a336-45ab0b43054b admin admin] PATH_INFO: `/v3/projects` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:21:44.117776 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9a519439-b50e-4e43-a336-45ab0b43054b admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:21:44.125876 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9a519439-b50e-4e43-a336-45ab0b43054b admin admin] RBAC: Authorization granted {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:21:44.163260 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 778/1582] 10.222.0.27 () {60 vars in 1153 bytes} [Thu Oct 13 04:21:44 2022] POST /identity/v3/projects => generated 379 bytes in 71 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 13 04:21:44.167314 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-28d6a32e-c34c-4a12-8cd9-485e0feac3a9 admin admin] Authenticating user token {{(pid=81409) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:21:44.195482 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-28d6a32e-c34c-4a12-8cd9-485e0feac3a9 admin admin] Validating token access rules against request {{(pid=81409) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:21:44.209064 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9a519439-b50e-4e43-a336-45ab0b43054b admin admin] Authenticating user token {{(pid=81408) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:21:44.269002 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e3995343-3e65-4b97-af5d-ed3015f6484c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8ba823f313144da3b683bacef8fd62e0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81409) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:21:44.270236 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3995343-3e65-4b97-af5d-ed3015f6484c admin admin] REQUEST_METHOD: `POST` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:21:44.270979 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3995343-3e65-4b97-af5d-ed3015f6484c admin admin] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:21:44.271488 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3995343-3e65-4b97-af5d-ed3015f6484c admin admin] PATH_INFO: `/v3/users` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:21:44.272335 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e3995343-3e65-4b97-af5d-ed3015f6484c admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:21:44.274577 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e3995343-3e65-4b97-af5d-ed3015f6484c admin admin] RBAC: Authorization granted {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:21:44.281907 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9a519439-b50e-4e43-a336-45ab0b43054b admin admin] Validating token access rules against request {{(pid=81408) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:21:44.286095 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-040f23fe-b6cb-454a-b966-4cd56ff49620 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6141b8a6c3d54e179d6879bb019f4d12', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dcec0431fb6849499d6aec35a234daa6', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81408) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:21:44.286749 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-040f23fe-b6cb-454a-b966-4cd56ff49620 service cinder] REQUEST_METHOD: `GET` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:21:44.287025 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-040f23fe-b6cb-454a-b966-4cd56ff49620 service cinder] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:21:44.287291 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-040f23fe-b6cb-454a-b966-4cd56ff49620 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:21:44.289488 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.fernet_utils [None req-040f23fe-b6cb-454a-b966-4cd56ff49620 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81408) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:21:44.312782 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 805/1583] 10.222.0.27 () {60 vars in 1144 bytes} [Thu Oct 13 04:21:44 2022] POST /identity/v3/users => generated 290 bytes in 146 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 13 04:21:44.317646 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e3995343-3e65-4b97-af5d-ed3015f6484c admin admin] Authenticating user token {{(pid=81409) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:21:44.332523 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e3995343-3e65-4b97-af5d-ed3015f6484c admin admin] Validating token access rules against request {{(pid=81409) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:21:44.339742 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e49944c8-cdf6-48fc-b902-e8e89ec2d6b7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8ba823f313144da3b683bacef8fd62e0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81409) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:21:44.341474 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e49944c8-cdf6-48fc-b902-e8e89ec2d6b7 admin admin] REQUEST_METHOD: `GET` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:21:44.341989 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e49944c8-cdf6-48fc-b902-e8e89ec2d6b7 admin admin] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:21:44.342772 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e49944c8-cdf6-48fc-b902-e8e89ec2d6b7 admin admin] PATH_INFO: `/v3/roles` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:21:44.343709 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e49944c8-cdf6-48fc-b902-e8e89ec2d6b7 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:21:44.345846 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e49944c8-cdf6-48fc-b902-e8e89ec2d6b7 admin admin] RBAC: Authorization granted {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:21:44.361852 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 806/1584] 10.222.0.27 () {58 vars in 1122 bytes} [Thu Oct 13 04:21:44 2022] GET /identity/v3/roles => generated 1420 bytes in 46 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 13 04:21:44.365932 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e49944c8-cdf6-48fc-b902-e8e89ec2d6b7 admin admin] Authenticating user token {{(pid=81409) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:21:44.366671 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-040f23fe-b6cb-454a-b966-4cd56ff49620 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:21:44.368396 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-040f23fe-b6cb-454a-b966-4cd56ff49620 service cinder] RBAC: Authorization granted {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:21:44.380909 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e49944c8-cdf6-48fc-b902-e8e89ec2d6b7 admin admin] Validating token access rules against request {{(pid=81409) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:21:44.388652 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-edeb0c8d-2887-4d71-88f7-b2ceb4c07895 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8ba823f313144da3b683bacef8fd62e0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81409) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:21:44.389245 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 779/1585] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Oct 13 04:21:44 2022] GET /identity/v3/auth/tokens => generated 3242 bytes in 181 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Oct 13 04:21:44.390134 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-edeb0c8d-2887-4d71-88f7-b2ceb4c07895 admin admin] REQUEST_METHOD: `GET` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:21:44.390701 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-edeb0c8d-2887-4d71-88f7-b2ceb4c07895 admin admin] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:21:44.391196 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-edeb0c8d-2887-4d71-88f7-b2ceb4c07895 admin admin] PATH_INFO: `/v3/roles` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:21:44.392407 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-edeb0c8d-2887-4d71-88f7-b2ceb4c07895 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:21:44.394446 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-edeb0c8d-2887-4d71-88f7-b2ceb4c07895 admin admin] RBAC: Authorization granted {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:21:44.424586 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 807/1586] 10.222.0.27 () {58 vars in 1122 bytes} [Thu Oct 13 04:21:44 2022] GET /identity/v3/roles => generated 1420 bytes in 57 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 13 04:21:44.425776 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-040f23fe-b6cb-454a-b966-4cd56ff49620 service cinder] Authenticating user token {{(pid=81408) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:21:44.437360 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-040f23fe-b6cb-454a-b966-4cd56ff49620 service cinder] Validating token access rules against request {{(pid=81408) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:21:44.441975 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-63506def-8936-47eb-ac36-50482aa56256 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8ba823f313144da3b683bacef8fd62e0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81408) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:21:44.442776 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63506def-8936-47eb-ac36-50482aa56256 admin admin] REQUEST_METHOD: `PUT` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:21:44.443165 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63506def-8936-47eb-ac36-50482aa56256 admin admin] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:21:44.443506 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63506def-8936-47eb-ac36-50482aa56256 admin admin] PATH_INFO: `/v3/projects/95e7139479ed41fa8e7a1cf6e3e874d7/users/01498f56b62246339c7cd9d3cead50ac/roles/cc56c4ae13a7424ea771c15da4b153c1` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:21:44.460763 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-63506def-8936-47eb-ac36-50482aa56256 admin admin] RBAC: Authorizing `identity:create_grant(project_id=95e7139479ed41fa8e7a1cf6e3e874d7, user_id=01498f56b62246339c7cd9d3cead50ac, role_id=cc56c4ae13a7424ea771c15da4b153c1)` {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:21:44.463882 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-63506def-8936-47eb-ac36-50482aa56256 admin admin] RBAC: Authorization granted {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:21:44.482086 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 780/1587] 10.222.0.27 () {60 vars in 1483 bytes} [Thu Oct 13 04:21:44 2022] PUT /identity/v3/projects/95e7139479ed41fa8e7a1cf6e3e874d7/users/01498f56b62246339c7cd9d3cead50ac/roles/cc56c4ae13a7424ea771c15da4b153c1 => generated 0 bytes in 57 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 13 04:21:44.493902 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0735d0d-e7be-4f52-bd57-2f1c82973729 None None] REQUEST_METHOD: `POST` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:21:44.494261 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0735d0d-e7be-4f52-bd57-2f1c82973729 None None] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:21:44.494649 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0735d0d-e7be-4f52-bd57-2f1c82973729 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:21:44.532164 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.auth.core [None req-d0735d0d-e7be-4f52-bd57-2f1c82973729 None None] MFA Rules not processed for user `01498f56b62246339c7cd9d3cead50ac`. Rule list: `[]` (Enabled: `True`). {{(pid=81409) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 13 04:21:44.533543 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.fernet_utils [None req-d0735d0d-e7be-4f52-bd57-2f1c82973729 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81409) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:21:44.618253 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 808/1588] 10.222.0.27 () {58 vars in 958 bytes} [Thu Oct 13 04:21:44 2022] POST /identity/v3/auth/tokens => generated 3256 bytes in 125 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 13 04:21:44.622909 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-63506def-8936-47eb-ac36-50482aa56256 admin admin] Authenticating user token {{(pid=81408) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:21:44.689595 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-63506def-8936-47eb-ac36-50482aa56256 admin admin] Validating token access rules against request {{(pid=81408) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:21:44.694600 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a86d2d37-6c9d-48fb-a5b9-6c0ad919b1c8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8ba823f313144da3b683bacef8fd62e0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81408) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:21:44.695216 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a86d2d37-6c9d-48fb-a5b9-6c0ad919b1c8 admin admin] REQUEST_METHOD: `POST` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:21:44.695547 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a86d2d37-6c9d-48fb-a5b9-6c0ad919b1c8 admin admin] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:21:44.695826 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a86d2d37-6c9d-48fb-a5b9-6c0ad919b1c8 admin admin] PATH_INFO: `/v3/projects` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:21:44.696502 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a86d2d37-6c9d-48fb-a5b9-6c0ad919b1c8 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:21:44.698492 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a86d2d37-6c9d-48fb-a5b9-6c0ad919b1c8 admin admin] RBAC: Authorization granted {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:21:44.714973 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG dbcounter [-] [81408] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=81408) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:21:44.737153 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 781/1589] 10.222.0.27 () {60 vars in 1153 bytes} [Thu Oct 13 04:21:44 2022] POST /identity/v3/projects => generated 379 bytes in 112 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 13 04:21:44.738304 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d0735d0d-e7be-4f52-bd57-2f1c82973729 None None] Authenticating user token {{(pid=81409) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:21:44.749385 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d0735d0d-e7be-4f52-bd57-2f1c82973729 None None] Validating token access rules against request {{(pid=81409) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:21:44.760255 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG dbcounter [-] [81409] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81409) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:21:44.816213 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-19e938b5-0ff4-43ff-85c3-e84c289a2bbd admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8ba823f313144da3b683bacef8fd62e0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81409) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:21:44.816836 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19e938b5-0ff4-43ff-85c3-e84c289a2bbd admin admin] REQUEST_METHOD: `POST` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:21:44.817114 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19e938b5-0ff4-43ff-85c3-e84c289a2bbd admin admin] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:21:44.817381 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19e938b5-0ff4-43ff-85c3-e84c289a2bbd admin admin] PATH_INFO: `/v3/users` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:21:44.818024 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-19e938b5-0ff4-43ff-85c3-e84c289a2bbd admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:21:44.820143 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-19e938b5-0ff4-43ff-85c3-e84c289a2bbd admin admin] RBAC: Authorization granted {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:21:44.843645 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 809/1590] 10.222.0.27 () {60 vars in 1144 bytes} [Thu Oct 13 04:21:44 2022] POST /identity/v3/users => generated 290 bytes in 106 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 13 04:21:44.847545 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a86d2d37-6c9d-48fb-a5b9-6c0ad919b1c8 admin admin] Authenticating user token {{(pid=81408) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:21:44.859435 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a86d2d37-6c9d-48fb-a5b9-6c0ad919b1c8 admin admin] Validating token access rules against request {{(pid=81408) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:21:44.864886 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-845800a2-a90e-4615-b11d-ce57f7d7060e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8ba823f313144da3b683bacef8fd62e0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81408) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:21:44.865690 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-845800a2-a90e-4615-b11d-ce57f7d7060e admin admin] REQUEST_METHOD: `GET` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:21:44.866151 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-845800a2-a90e-4615-b11d-ce57f7d7060e admin admin] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:21:44.866555 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-845800a2-a90e-4615-b11d-ce57f7d7060e admin admin] PATH_INFO: `/v3/roles` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:21:44.867477 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-845800a2-a90e-4615-b11d-ce57f7d7060e admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:21:44.869446 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-845800a2-a90e-4615-b11d-ce57f7d7060e admin admin] RBAC: Authorization granted {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:21:44.887486 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-19e938b5-0ff4-43ff-85c3-e84c289a2bbd admin admin] Authenticating user token {{(pid=81409) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:21:44.896131 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 782/1591] 10.222.0.27 () {58 vars in 1122 bytes} [Thu Oct 13 04:21:44 2022] GET /identity/v3/roles => generated 1420 bytes in 39 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 13 04:21:44.915483 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-19e938b5-0ff4-43ff-85c3-e84c289a2bbd admin admin] Validating token access rules against request {{(pid=81409) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:21:44.921617 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0e4caab4-df28-40ef-9458-e80dd70fb05b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8ba823f313144da3b683bacef8fd62e0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81409) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:21:44.922425 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e4caab4-df28-40ef-9458-e80dd70fb05b admin admin] REQUEST_METHOD: `GET` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:21:44.922880 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e4caab4-df28-40ef-9458-e80dd70fb05b admin admin] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:21:44.923315 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e4caab4-df28-40ef-9458-e80dd70fb05b admin admin] PATH_INFO: `/v3/roles` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:21:44.924529 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0e4caab4-df28-40ef-9458-e80dd70fb05b admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:21:44.926442 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0e4caab4-df28-40ef-9458-e80dd70fb05b admin admin] RBAC: Authorization granted {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:21:44.944399 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 810/1592] 10.222.0.27 () {58 vars in 1122 bytes} [Thu Oct 13 04:21:44 2022] GET /identity/v3/roles => generated 1420 bytes in 57 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Oct 13 04:21:44.953088 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-845800a2-a90e-4615-b11d-ce57f7d7060e admin admin] Authenticating user token {{(pid=81408) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:21:44.975979 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-845800a2-a90e-4615-b11d-ce57f7d7060e admin admin] Validating token access rules against request {{(pid=81408) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:21:44.985218 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ce167d9c-df5c-45b6-9b5d-48928501b41a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8ba823f313144da3b683bacef8fd62e0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81408) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:21:44.986515 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce167d9c-df5c-45b6-9b5d-48928501b41a admin admin] REQUEST_METHOD: `PUT` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:21:44.987022 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce167d9c-df5c-45b6-9b5d-48928501b41a admin admin] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:21:44.987546 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce167d9c-df5c-45b6-9b5d-48928501b41a admin admin] PATH_INFO: `/v3/projects/1ceae26ece6c4ad0a47610d8af5047c1/users/1e075f9fd76d46eda970c96b03c1cec2/roles/cc56c4ae13a7424ea771c15da4b153c1` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:21:45.013731 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ce167d9c-df5c-45b6-9b5d-48928501b41a admin admin] RBAC: Authorizing `identity:create_grant(project_id=1ceae26ece6c4ad0a47610d8af5047c1, user_id=1e075f9fd76d46eda970c96b03c1cec2, role_id=cc56c4ae13a7424ea771c15da4b153c1)` {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:21:45.017010 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ce167d9c-df5c-45b6-9b5d-48928501b41a admin admin] RBAC: Authorization granted {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:21:45.032575 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 783/1593] 10.222.0.27 () {60 vars in 1483 bytes} [Thu Oct 13 04:21:44 2022] PUT /identity/v3/projects/1ceae26ece6c4ad0a47610d8af5047c1/users/1e075f9fd76d46eda970c96b03c1cec2/roles/cc56c4ae13a7424ea771c15da4b153c1 => generated 0 bytes in 81 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 13 04:21:45.044748 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31d39b44-991a-49a4-811d-130b7e9ff45b None None] REQUEST_METHOD: `POST` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:21:45.045040 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31d39b44-991a-49a4-811d-130b7e9ff45b None None] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:21:45.045327 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31d39b44-991a-49a4-811d-130b7e9ff45b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:21:45.077910 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.auth.core [None req-31d39b44-991a-49a4-811d-130b7e9ff45b None None] MFA Rules not processed for user `1e075f9fd76d46eda970c96b03c1cec2`. Rule list: `[]` (Enabled: `True`). {{(pid=81409) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 13 04:21:45.079595 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.fernet_utils [None req-31d39b44-991a-49a4-811d-130b7e9ff45b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81409) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:21:45.163821 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 811/1594] 10.222.0.27 () {58 vars in 958 bytes} [Thu Oct 13 04:21:45 2022] POST /identity/v3/auth/tokens => generated 3256 bytes in 121 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 13 04:21:45.168359 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ce167d9c-df5c-45b6-9b5d-48928501b41a admin admin] Authenticating user token {{(pid=81408) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:21:45.243294 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ce167d9c-df5c-45b6-9b5d-48928501b41a admin admin] Validating token access rules against request {{(pid=81408) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:21:45.248859 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fd81be9a-e323-4376-a70c-5da02878354b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8ba823f313144da3b683bacef8fd62e0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81408) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:21:45.249480 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd81be9a-e323-4376-a70c-5da02878354b admin admin] REQUEST_METHOD: `POST` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:21:45.249754 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd81be9a-e323-4376-a70c-5da02878354b admin admin] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:21:45.250019 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd81be9a-e323-4376-a70c-5da02878354b admin admin] PATH_INFO: `/v3/projects` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:21:45.251172 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fd81be9a-e323-4376-a70c-5da02878354b admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:21:45.253188 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fd81be9a-e323-4376-a70c-5da02878354b admin admin] RBAC: Authorization granted {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:21:45.288607 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 784/1595] 10.222.0.27 () {60 vars in 1153 bytes} [Thu Oct 13 04:21:45 2022] POST /identity/v3/projects => generated 379 bytes in 118 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 13 04:21:45.290338 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-31d39b44-991a-49a4-811d-130b7e9ff45b None None] Authenticating user token {{(pid=81409) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:21:45.303835 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-31d39b44-991a-49a4-811d-130b7e9ff45b None None] Validating token access rules against request {{(pid=81409) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:21:45.392919 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e0c28a6b-f0a5-4b0f-9227-1a189741085f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8ba823f313144da3b683bacef8fd62e0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81409) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:21:45.393544 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0c28a6b-f0a5-4b0f-9227-1a189741085f admin admin] REQUEST_METHOD: `POST` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:21:45.393827 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0c28a6b-f0a5-4b0f-9227-1a189741085f admin admin] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:21:45.394126 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0c28a6b-f0a5-4b0f-9227-1a189741085f admin admin] PATH_INFO: `/v3/users` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:21:45.394757 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e0c28a6b-f0a5-4b0f-9227-1a189741085f admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:21:45.396677 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e0c28a6b-f0a5-4b0f-9227-1a189741085f admin admin] RBAC: Authorization granted {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:21:45.439855 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 812/1596] 10.222.0.27 () {60 vars in 1144 bytes} [Thu Oct 13 04:21:45 2022] POST /identity/v3/users => generated 296 bytes in 150 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 13 04:21:45.443741 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fd81be9a-e323-4376-a70c-5da02878354b admin admin] Authenticating user token {{(pid=81408) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:21:45.454678 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fd81be9a-e323-4376-a70c-5da02878354b admin admin] Validating token access rules against request {{(pid=81408) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:21:45.459640 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f23a31f6-84a5-4f4f-ba2c-e7b26d18e343 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8ba823f313144da3b683bacef8fd62e0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81408) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:21:45.468173 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f23a31f6-84a5-4f4f-ba2c-e7b26d18e343 admin admin] REQUEST_METHOD: `GET` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:21:45.468466 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f23a31f6-84a5-4f4f-ba2c-e7b26d18e343 admin admin] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:21:45.468740 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f23a31f6-84a5-4f4f-ba2c-e7b26d18e343 admin admin] PATH_INFO: `/v3/roles` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:21:45.469416 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f23a31f6-84a5-4f4f-ba2c-e7b26d18e343 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:21:45.471093 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f23a31f6-84a5-4f4f-ba2c-e7b26d18e343 admin admin] RBAC: Authorization granted {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:21:45.487745 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 785/1597] 10.222.0.27 () {58 vars in 1122 bytes} [Thu Oct 13 04:21:45 2022] GET /identity/v3/roles => generated 1420 bytes in 40 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 13 04:21:45.487927 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e0c28a6b-f0a5-4b0f-9227-1a189741085f admin admin] Authenticating user token {{(pid=81409) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:21:45.498466 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e0c28a6b-f0a5-4b0f-9227-1a189741085f admin admin] Validating token access rules against request {{(pid=81409) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:21:45.503130 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9653e941-fdae-4cc3-b567-82b88d97693a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8ba823f313144da3b683bacef8fd62e0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81409) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:21:45.503790 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9653e941-fdae-4cc3-b567-82b88d97693a admin admin] REQUEST_METHOD: `PUT` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:21:45.504066 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9653e941-fdae-4cc3-b567-82b88d97693a admin admin] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:21:45.504337 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9653e941-fdae-4cc3-b567-82b88d97693a admin admin] PATH_INFO: `/v3/projects/610bb38e1f5e4bd2ad2afe210df75757/users/987d5e2972cf4038a3ba585a4d4b6f07/roles/ef55fbd94e1040be83695754075a7ed5` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:21:45.514077 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG dbcounter [-] [81409] Writing DB stats keystone:SELECT=94,keystone:INSERT=6 {{(pid=81409) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:21:45.521885 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9653e941-fdae-4cc3-b567-82b88d97693a admin admin] RBAC: Authorizing `identity:create_grant(project_id=610bb38e1f5e4bd2ad2afe210df75757, user_id=987d5e2972cf4038a3ba585a4d4b6f07, role_id=ef55fbd94e1040be83695754075a7ed5)` {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:21:45.524648 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9653e941-fdae-4cc3-b567-82b88d97693a admin admin] RBAC: Authorization granted {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:21:45.554684 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 813/1598] 10.222.0.27 () {60 vars in 1483 bytes} [Thu Oct 13 04:21:45 2022] PUT /identity/v3/projects/610bb38e1f5e4bd2ad2afe210df75757/users/987d5e2972cf4038a3ba585a4d4b6f07/roles/ef55fbd94e1040be83695754075a7ed5 => generated 0 bytes in 64 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 13 04:21:45.556908 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f23a31f6-84a5-4f4f-ba2c-e7b26d18e343 admin admin] Authenticating user token {{(pid=81408) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:21:45.570739 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f23a31f6-84a5-4f4f-ba2c-e7b26d18e343 admin admin] Validating token access rules against request {{(pid=81408) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:21:45.671183 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-99774186-8bcd-43fd-a393-a755e5cc625f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8ba823f313144da3b683bacef8fd62e0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81408) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:21:45.671801 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99774186-8bcd-43fd-a393-a755e5cc625f admin admin] REQUEST_METHOD: `GET` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:21:45.672080 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99774186-8bcd-43fd-a393-a755e5cc625f admin admin] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:21:45.672353 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99774186-8bcd-43fd-a393-a755e5cc625f admin admin] PATH_INFO: `/v3/roles` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:21:45.672996 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-99774186-8bcd-43fd-a393-a755e5cc625f admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:21:45.674955 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-99774186-8bcd-43fd-a393-a755e5cc625f admin admin] RBAC: Authorization granted {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:21:45.688975 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 786/1599] 10.222.0.27 () {58 vars in 1122 bytes} [Thu Oct 13 04:21:45 2022] GET /identity/v3/roles => generated 1420 bytes in 133 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 13 04:21:45.697362 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9653e941-fdae-4cc3-b567-82b88d97693a admin admin] Authenticating user token {{(pid=81409) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:21:45.714505 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9653e941-fdae-4cc3-b567-82b88d97693a admin admin] Validating token access rules against request {{(pid=81409) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:21:45.722091 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4c314137-d945-44e4-b80b-a3c0dfb31b55 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8ba823f313144da3b683bacef8fd62e0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81409) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:21:45.729005 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c314137-d945-44e4-b80b-a3c0dfb31b55 admin admin] REQUEST_METHOD: `PUT` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:21:45.729005 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c314137-d945-44e4-b80b-a3c0dfb31b55 admin admin] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:21:45.729005 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c314137-d945-44e4-b80b-a3c0dfb31b55 admin admin] PATH_INFO: `/v3/projects/610bb38e1f5e4bd2ad2afe210df75757/users/987d5e2972cf4038a3ba585a4d4b6f07/roles/ef55fbd94e1040be83695754075a7ed5` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:21:45.729005 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4c314137-d945-44e4-b80b-a3c0dfb31b55 admin admin] RBAC: Authorizing `identity:create_grant(project_id=610bb38e1f5e4bd2ad2afe210df75757, user_id=987d5e2972cf4038a3ba585a4d4b6f07, role_id=ef55fbd94e1040be83695754075a7ed5)` {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:21:45.732620 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4c314137-d945-44e4-b80b-a3c0dfb31b55 admin admin] RBAC: Authorization granted {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:21:45.750034 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 814/1600] 10.222.0.27 () {60 vars in 1483 bytes} [Thu Oct 13 04:21:45 2022] PUT /identity/v3/projects/610bb38e1f5e4bd2ad2afe210df75757/users/987d5e2972cf4038a3ba585a4d4b6f07/roles/ef55fbd94e1040be83695754075a7ed5 => generated 0 bytes in 53 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 13 04:21:45.761533 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9b9fb4d-5c6b-43d2-956c-71750c7ca02d None None] REQUEST_METHOD: `POST` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:21:45.762099 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9b9fb4d-5c6b-43d2-956c-71750c7ca02d None None] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:21:45.762335 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9b9fb4d-5c6b-43d2-956c-71750c7ca02d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:21:45.822648 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.auth.core [None req-c9b9fb4d-5c6b-43d2-956c-71750c7ca02d None None] MFA Rules not processed for user `987d5e2972cf4038a3ba585a4d4b6f07`. Rule list: `[]` (Enabled: `True`). {{(pid=81408) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 13 04:21:45.823897 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.fernet_utils [None req-c9b9fb4d-5c6b-43d2-956c-71750c7ca02d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81408) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:21:45.853105 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG dbcounter [-] [81408] Writing DB stats keystone:INSERT=3,keystone:SELECT=97 {{(pid=81408) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:21:45.965702 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 787/1601] 10.222.0.27 () {58 vars in 958 bytes} [Thu Oct 13 04:21:45 2022] POST /identity/v3/auth/tokens => generated 3323 bytes in 205 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 13 04:21:45.994829 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4c314137-d945-44e4-b80b-a3c0dfb31b55 admin admin] Authenticating user token {{(pid=81409) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:21:45.999991 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.fernet_utils [None req-4c314137-d945-44e4-b80b-a3c0dfb31b55 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81409) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:21:46.086639 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4c314137-d945-44e4-b80b-a3c0dfb31b55 admin admin] Validating token access rules against request {{(pid=81409) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:21:46.092447 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f1a933c2-f2b7-4b0f-91e4-4605758d5e26 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6141b8a6c3d54e179d6879bb019f4d12', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dcec0431fb6849499d6aec35a234daa6', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81409) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:21:46.093283 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1a933c2-f2b7-4b0f-91e4-4605758d5e26 service cinder] REQUEST_METHOD: `GET` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:21:46.093695 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1a933c2-f2b7-4b0f-91e4-4605758d5e26 service cinder] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:21:46.094160 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1a933c2-f2b7-4b0f-91e4-4605758d5e26 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:21:46.131955 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f1a933c2-f2b7-4b0f-91e4-4605758d5e26 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:21:46.134707 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f1a933c2-f2b7-4b0f-91e4-4605758d5e26 service cinder] RBAC: Authorization granted {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:21:46.161346 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 815/1602] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Oct 13 04:21:45 2022] GET /identity/v3/auth/tokens => generated 3323 bytes in 167 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 13 04:21:46.271055 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c9b9fb4d-5c6b-43d2-956c-71750c7ca02d None None] Authenticating user token {{(pid=81408) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:21:46.281902 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c9b9fb4d-5c6b-43d2-956c-71750c7ca02d None None] Validating token access rules against request {{(pid=81408) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:21:46.286055 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b5bc9aaa-a645-4908-ac21-0f7bec15e771 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6141b8a6c3d54e179d6879bb019f4d12', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dcec0431fb6849499d6aec35a234daa6', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81408) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:21:46.286753 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5bc9aaa-a645-4908-ac21-0f7bec15e771 service cinder] REQUEST_METHOD: `GET` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:21:46.287058 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5bc9aaa-a645-4908-ac21-0f7bec15e771 service cinder] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:21:46.287335 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5bc9aaa-a645-4908-ac21-0f7bec15e771 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:21:46.360444 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b5bc9aaa-a645-4908-ac21-0f7bec15e771 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:21:46.361896 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b5bc9aaa-a645-4908-ac21-0f7bec15e771 service cinder] RBAC: Authorization granted {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:21:46.371439 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 788/1603] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Oct 13 04:21:46 2022] GET /identity/v3/auth/tokens => generated 3256 bytes in 101 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 13 04:21:47.670413 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f1a933c2-f2b7-4b0f-91e4-4605758d5e26 service cinder] Authenticating user token {{(pid=81409) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:21:47.672354 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.fernet_utils [None req-f1a933c2-f2b7-4b0f-91e4-4605758d5e26 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81409) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:21:47.739531 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f1a933c2-f2b7-4b0f-91e4-4605758d5e26 service cinder] Validating token access rules against request {{(pid=81409) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:21:47.744745 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1257e3dc-0a06-433f-ad21-1bf7114d8f7f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8ba823f313144da3b683bacef8fd62e0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81409) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:21:47.745524 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1257e3dc-0a06-433f-ad21-1bf7114d8f7f admin admin] REQUEST_METHOD: `DELETE` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:21:47.745859 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1257e3dc-0a06-433f-ad21-1bf7114d8f7f admin admin] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:21:47.746193 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1257e3dc-0a06-433f-ad21-1bf7114d8f7f admin admin] PATH_INFO: `/v3/users/d06360ec39864348a016620181a59739` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:21:47.747487 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1257e3dc-0a06-433f-ad21-1bf7114d8f7f admin admin] RBAC: Authorizing `identity:delete_user(user_id=d06360ec39864348a016620181a59739)` {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:21:47.749392 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1257e3dc-0a06-433f-ad21-1bf7114d8f7f admin admin] RBAC: Authorization granted {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:21:47.801520 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.notifications [None req-1257e3dc-0a06-433f-ad21-1bf7114d8f7f admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'd06360ec39864348a016620181a59739'} {{(pid=81409) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 13 04:21:47.802191 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.notifications [None req-1257e3dc-0a06-433f-ad21-1bf7114d8f7f admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'd06360ec39864348a016620181a59739'} {{(pid=81409) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 13 04:21:47.810844 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.notifications [None req-1257e3dc-0a06-433f-ad21-1bf7114d8f7f admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'd06360ec39864348a016620181a59739'} {{(pid=81409) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 13 04:21:47.811685 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.notifications [None req-1257e3dc-0a06-433f-ad21-1bf7114d8f7f admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'd06360ec39864348a016620181a59739'} {{(pid=81409) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 13 04:21:47.820127 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.notifications [None req-1257e3dc-0a06-433f-ad21-1bf7114d8f7f admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'd06360ec39864348a016620181a59739'} {{(pid=81409) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 13 04:21:47.836556 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG dbcounter [-] [81409] Writing DB stats keystone:SELECT=87,keystone:INSERT=3,keystone:DELETE=10 {{(pid=81409) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:21:47.851590 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 816/1604] 10.222.0.27 () {58 vars in 1224 bytes} [Thu Oct 13 04:21:47 2022] DELETE /identity/v3/users/d06360ec39864348a016620181a59739 => generated 0 bytes in 182 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 13 04:21:47.855931 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61ff93c4-5eb9-46f8-9f52-e5aabb276152 None None] REQUEST_METHOD: `POST` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:21:47.857507 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61ff93c4-5eb9-46f8-9f52-e5aabb276152 None None] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:21:47.857794 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61ff93c4-5eb9-46f8-9f52-e5aabb276152 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:21:47.883100 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.auth.core [None req-61ff93c4-5eb9-46f8-9f52-e5aabb276152 None None] MFA Rules not processed for user `0e2d979563d6421995451a0e86bf3022`. Rule list: `[]` (Enabled: `True`). {{(pid=81408) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 13 04:21:47.885324 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.fernet_utils [None req-61ff93c4-5eb9-46f8-9f52-e5aabb276152 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81408) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:21:48.041468 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 789/1605] 10.222.0.27 () {58 vars in 958 bytes} [Thu Oct 13 04:21:47 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 186 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 13 04:21:48.052524 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1257e3dc-0a06-433f-ad21-1bf7114d8f7f admin admin] Authenticating user token {{(pid=81409) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:21:48.054151 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.fernet_utils [None req-1257e3dc-0a06-433f-ad21-1bf7114d8f7f admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81409) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:21:48.209597 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1257e3dc-0a06-433f-ad21-1bf7114d8f7f admin admin] Validating token access rules against request {{(pid=81409) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:21:48.220124 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b62a0657-b40f-497e-9de1-549d4b6562a1 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd8812c3974f84871bb2d761cf0016257', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dcec0431fb6849499d6aec35a234daa6', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81409) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:21:48.221046 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b62a0657-b40f-497e-9de1-549d4b6562a1 service neutron] REQUEST_METHOD: `GET` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:21:48.221459 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b62a0657-b40f-497e-9de1-549d4b6562a1 service neutron] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:21:48.222087 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b62a0657-b40f-497e-9de1-549d4b6562a1 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:21:48.243438 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b62a0657-b40f-497e-9de1-549d4b6562a1 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:21:48.244886 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b62a0657-b40f-497e-9de1-549d4b6562a1 service neutron] RBAC: Authorization granted {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:21:48.270333 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 817/1606] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Oct 13 04:21:48 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 218 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 13 04:21:48.777772 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-61ff93c4-5eb9-46f8-9f52-e5aabb276152 None None] Authenticating user token {{(pid=81408) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:21:48.779372 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.fernet_utils [None req-61ff93c4-5eb9-46f8-9f52-e5aabb276152 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81408) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:21:48.791188 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-61ff93c4-5eb9-46f8-9f52-e5aabb276152 None None] Validating token access rules against request {{(pid=81408) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:21:48.796470 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6cfbcbdd-ae5e-464c-a37d-3f2c86d859ec admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8ba823f313144da3b683bacef8fd62e0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81408) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:21:48.797104 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6cfbcbdd-ae5e-464c-a37d-3f2c86d859ec admin admin] REQUEST_METHOD: `DELETE` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:21:48.797384 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6cfbcbdd-ae5e-464c-a37d-3f2c86d859ec admin admin] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:21:48.797655 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6cfbcbdd-ae5e-464c-a37d-3f2c86d859ec admin admin] PATH_INFO: `/v3/projects/55efcb49c8ee4f2abdee379a05261f3e` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:21:48.798814 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6cfbcbdd-ae5e-464c-a37d-3f2c86d859ec admin admin] RBAC: Authorizing `identity:delete_project(project_id=55efcb49c8ee4f2abdee379a05261f3e)` {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:21:48.800727 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6cfbcbdd-ae5e-464c-a37d-3f2c86d859ec admin admin] RBAC: Authorization granted {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:21:48.861954 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG dbcounter [-] [81408] Writing DB stats keystone:SELECT=96,keystone:DELETE=4 {{(pid=81408) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:21:48.864289 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.notifications [None req-6cfbcbdd-ae5e-464c-a37d-3f2c86d859ec admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '55efcb49c8ee4f2abdee379a05261f3e', 'request_id': 'req-6cfbcbdd-ae5e-464c-a37d-3f2c86d859ec'} {{(pid=81408) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 13 04:21:48.873385 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.notifications [None req-6cfbcbdd-ae5e-464c-a37d-3f2c86d859ec admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '55efcb49c8ee4f2abdee379a05261f3e', 'request_id': 'req-6cfbcbdd-ae5e-464c-a37d-3f2c86d859ec'} {{(pid=81408) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 13 04:21:48.880296 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.notifications [None req-6cfbcbdd-ae5e-464c-a37d-3f2c86d859ec admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '55efcb49c8ee4f2abdee379a05261f3e', 'request_id': 'req-6cfbcbdd-ae5e-464c-a37d-3f2c86d859ec'} {{(pid=81408) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 13 04:21:48.891470 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.notifications [None req-6cfbcbdd-ae5e-464c-a37d-3f2c86d859ec admin admin] The token cache is being invalidate because project 55efcb49c8ee4f2abdee379a05261f3e was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81408) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 13 04:21:48.891470 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.notifications [None req-6cfbcbdd-ae5e-464c-a37d-3f2c86d859ec admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81408) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 13 04:21:48.891802 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.notifications [None req-6cfbcbdd-ae5e-464c-a37d-3f2c86d859ec admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81408) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 13 04:21:48.896113 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 790/1607] 10.222.0.27 () {58 vars in 1233 bytes} [Thu Oct 13 04:21:48 2022] DELETE /identity/v3/projects/55efcb49c8ee4f2abdee379a05261f3e => generated 0 bytes in 116 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 13 04:21:48.897890 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b62a0657-b40f-497e-9de1-549d4b6562a1 service neutron] Authenticating user token {{(pid=81409) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:21:48.908088 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b62a0657-b40f-497e-9de1-549d4b6562a1 service neutron] Validating token access rules against request {{(pid=81409) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:21:48.963794 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-78c3bea9-107f-4603-8245-765771534f09 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8ba823f313144da3b683bacef8fd62e0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81409) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:21:48.964419 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78c3bea9-107f-4603-8245-765771534f09 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:21:48.964697 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78c3bea9-107f-4603-8245-765771534f09 admin admin] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:21:48.964970 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78c3bea9-107f-4603-8245-765771534f09 admin admin] PATH_INFO: `/v3/users/bf48436529cc4889916d9a432b95bb84` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:21:48.966138 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-78c3bea9-107f-4603-8245-765771534f09 admin admin] RBAC: Authorizing `identity:delete_user(user_id=bf48436529cc4889916d9a432b95bb84)` {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:21:48.968089 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-78c3bea9-107f-4603-8245-765771534f09 admin admin] RBAC: Authorization granted {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:21:49.014067 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.notifications [None req-78c3bea9-107f-4603-8245-765771534f09 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'bf48436529cc4889916d9a432b95bb84'} {{(pid=81409) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 13 04:21:49.014738 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.notifications [None req-78c3bea9-107f-4603-8245-765771534f09 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'bf48436529cc4889916d9a432b95bb84'} {{(pid=81409) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 13 04:21:49.021364 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.notifications [None req-78c3bea9-107f-4603-8245-765771534f09 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'bf48436529cc4889916d9a432b95bb84'} {{(pid=81409) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 13 04:21:49.021972 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.notifications [None req-78c3bea9-107f-4603-8245-765771534f09 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'bf48436529cc4889916d9a432b95bb84'} {{(pid=81409) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 13 04:21:49.029775 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.notifications [None req-78c3bea9-107f-4603-8245-765771534f09 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'bf48436529cc4889916d9a432b95bb84'} {{(pid=81409) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 13 04:21:49.047240 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 818/1608] 10.222.0.27 () {58 vars in 1224 bytes} [Thu Oct 13 04:21:48 2022] DELETE /identity/v3/users/bf48436529cc4889916d9a432b95bb84 => generated 0 bytes in 150 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 13 04:21:49.316538 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6cfbcbdd-ae5e-464c-a37d-3f2c86d859ec admin admin] Authenticating user token {{(pid=81408) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:21:49.318189 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.fernet_utils [None req-6cfbcbdd-ae5e-464c-a37d-3f2c86d859ec admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81408) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:21:49.332545 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6cfbcbdd-ae5e-464c-a37d-3f2c86d859ec admin admin] Validating token access rules against request {{(pid=81408) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:21:49.394609 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9e8a9f6d-0906-472a-9e63-468435258eea admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8ba823f313144da3b683bacef8fd62e0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81408) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:21:49.395179 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e8a9f6d-0906-472a-9e63-468435258eea admin admin] REQUEST_METHOD: `DELETE` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:21:49.395458 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e8a9f6d-0906-472a-9e63-468435258eea admin admin] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:21:49.395730 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e8a9f6d-0906-472a-9e63-468435258eea admin admin] PATH_INFO: `/v3/projects/822488016a48437894fbd19cb330ea35` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:21:49.397115 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9e8a9f6d-0906-472a-9e63-468435258eea admin admin] RBAC: Authorizing `identity:delete_project(project_id=822488016a48437894fbd19cb330ea35)` {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:21:49.399085 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9e8a9f6d-0906-472a-9e63-468435258eea admin admin] RBAC: Authorization granted {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:21:49.458928 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.notifications [None req-9e8a9f6d-0906-472a-9e63-468435258eea admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '822488016a48437894fbd19cb330ea35', 'request_id': 'req-9e8a9f6d-0906-472a-9e63-468435258eea'} {{(pid=81408) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 13 04:21:49.466861 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.notifications [None req-9e8a9f6d-0906-472a-9e63-468435258eea admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '822488016a48437894fbd19cb330ea35', 'request_id': 'req-9e8a9f6d-0906-472a-9e63-468435258eea'} {{(pid=81408) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 13 04:21:49.473430 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.notifications [None req-9e8a9f6d-0906-472a-9e63-468435258eea admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '822488016a48437894fbd19cb330ea35', 'request_id': 'req-9e8a9f6d-0906-472a-9e63-468435258eea'} {{(pid=81408) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 13 04:21:49.486446 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.notifications [None req-9e8a9f6d-0906-472a-9e63-468435258eea admin admin] The token cache is being invalidate because project 822488016a48437894fbd19cb330ea35 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81408) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 13 04:21:49.486741 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.notifications [None req-9e8a9f6d-0906-472a-9e63-468435258eea admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81408) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 13 04:21:49.487402 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.notifications [None req-9e8a9f6d-0906-472a-9e63-468435258eea admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81408) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 13 04:21:49.488915 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 791/1609] 10.222.0.27 () {58 vars in 1233 bytes} [Thu Oct 13 04:21:49 2022] DELETE /identity/v3/projects/822488016a48437894fbd19cb330ea35 => generated 0 bytes in 173 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 13 04:21:49.493858 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-043774e9-9a5e-46be-8d55-15880c3567b8 None None] REQUEST_METHOD: `POST` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:21:49.494180 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-043774e9-9a5e-46be-8d55-15880c3567b8 None None] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:21:49.494580 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-043774e9-9a5e-46be-8d55-15880c3567b8 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:21:49.515039 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.auth.core [None req-043774e9-9a5e-46be-8d55-15880c3567b8 None None] MFA Rules not processed for user `0e2d979563d6421995451a0e86bf3022`. Rule list: `[]` (Enabled: `True`). {{(pid=81409) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 13 04:21:49.516238 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.fernet_utils [None req-043774e9-9a5e-46be-8d55-15880c3567b8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81409) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:21:49.528543 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG dbcounter [-] [81409] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81409) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:21:49.587950 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 819/1610] 10.222.0.27 () {58 vars in 958 bytes} [Thu Oct 13 04:21:49 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 89 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 13 04:21:49.589755 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b782e4d0-d58d-4c04-a409-80c917a27666 None None] REQUEST_METHOD: `POST` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:21:49.590041 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b782e4d0-d58d-4c04-a409-80c917a27666 None None] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:21:49.595074 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b782e4d0-d58d-4c04-a409-80c917a27666 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:21:49.627204 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.auth.core [None req-b782e4d0-d58d-4c04-a409-80c917a27666 None None] MFA Rules not processed for user `0e2d979563d6421995451a0e86bf3022`. Rule list: `[]` (Enabled: `True`). {{(pid=81408) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 13 04:21:49.631963 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.fernet_utils [None req-b782e4d0-d58d-4c04-a409-80c917a27666 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81408) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:21:49.641256 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 792/1611] 10.222.0.27 () {58 vars in 958 bytes} [Thu Oct 13 04:21:49 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 52 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 13 04:21:49.647626 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-043774e9-9a5e-46be-8d55-15880c3567b8 None None] Authenticating user token {{(pid=81409) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:21:49.658314 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-043774e9-9a5e-46be-8d55-15880c3567b8 None None] Validating token access rules against request {{(pid=81409) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:21:49.663267 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d10ee4ab-c906-40a3-9d81-f285139ec463 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8ba823f313144da3b683bacef8fd62e0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81409) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:21:49.664034 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d10ee4ab-c906-40a3-9d81-f285139ec463 admin admin] REQUEST_METHOD: `GET` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:21:49.664482 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d10ee4ab-c906-40a3-9d81-f285139ec463 admin admin] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:21:49.664888 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d10ee4ab-c906-40a3-9d81-f285139ec463 admin admin] PATH_INFO: `/v3/domains` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:21:49.665668 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d10ee4ab-c906-40a3-9d81-f285139ec463 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81409) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 13 04:21:49.666105 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d10ee4ab-c906-40a3-9d81-f285139ec463 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:21:49.667879 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d10ee4ab-c906-40a3-9d81-f285139ec463 admin admin] RBAC: Authorization granted {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:21:49.682572 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 820/1612] 10.222.0.27 () {58 vars in 1153 bytes} [Thu Oct 13 04:21:49 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 35 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 13 04:21:49.687835 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b782e4d0-d58d-4c04-a409-80c917a27666 None None] Authenticating user token {{(pid=81408) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:21:49.697515 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b782e4d0-d58d-4c04-a409-80c917a27666 None None] Validating token access rules against request {{(pid=81408) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:21:49.702712 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-417791cb-058e-4674-a32f-ea3b3eb7143f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8ba823f313144da3b683bacef8fd62e0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81408) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:21:49.703338 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-417791cb-058e-4674-a32f-ea3b3eb7143f admin admin] REQUEST_METHOD: `POST` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:21:49.703610 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-417791cb-058e-4674-a32f-ea3b3eb7143f admin admin] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:21:49.703876 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-417791cb-058e-4674-a32f-ea3b3eb7143f admin admin] PATH_INFO: `/v3/projects` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:21:49.704715 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-417791cb-058e-4674-a32f-ea3b3eb7143f admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:21:49.706696 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-417791cb-058e-4674-a32f-ea3b3eb7143f admin admin] RBAC: Authorization granted {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:21:49.740201 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 793/1613] 10.222.0.27 () {60 vars in 1153 bytes} [Thu Oct 13 04:21:49 2022] POST /identity/v3/projects => generated 387 bytes in 53 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 13 04:21:49.745719 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d10ee4ab-c906-40a3-9d81-f285139ec463 admin admin] Authenticating user token {{(pid=81409) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:21:49.757251 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d10ee4ab-c906-40a3-9d81-f285139ec463 admin admin] Validating token access rules against request {{(pid=81409) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:21:49.821122 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-34a73af6-94ea-4be2-a1ed-d3dbfd30c791 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8ba823f313144da3b683bacef8fd62e0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81409) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:21:49.821735 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34a73af6-94ea-4be2-a1ed-d3dbfd30c791 admin admin] REQUEST_METHOD: `POST` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:21:49.822008 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34a73af6-94ea-4be2-a1ed-d3dbfd30c791 admin admin] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:21:49.822304 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34a73af6-94ea-4be2-a1ed-d3dbfd30c791 admin admin] PATH_INFO: `/v3/users` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:21:49.822947 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-34a73af6-94ea-4be2-a1ed-d3dbfd30c791 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:21:49.824941 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-34a73af6-94ea-4be2-a1ed-d3dbfd30c791 admin admin] RBAC: Authorization granted {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:21:49.854429 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 821/1614] 10.222.0.27 () {60 vars in 1144 bytes} [Thu Oct 13 04:21:49 2022] POST /identity/v3/users => generated 294 bytes in 109 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 13 04:21:49.858785 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-417791cb-058e-4674-a32f-ea3b3eb7143f admin admin] Authenticating user token {{(pid=81408) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:21:49.870285 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-417791cb-058e-4674-a32f-ea3b3eb7143f admin admin] Validating token access rules against request {{(pid=81408) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:21:49.875437 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ccb3c6a8-54f6-46cd-a50f-d2de2b25eb65 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8ba823f313144da3b683bacef8fd62e0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81408) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:21:49.876057 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ccb3c6a8-54f6-46cd-a50f-d2de2b25eb65 admin admin] REQUEST_METHOD: `GET` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:21:49.876335 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ccb3c6a8-54f6-46cd-a50f-d2de2b25eb65 admin admin] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:21:49.876608 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ccb3c6a8-54f6-46cd-a50f-d2de2b25eb65 admin admin] PATH_INFO: `/v3/roles` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:21:49.877254 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ccb3c6a8-54f6-46cd-a50f-d2de2b25eb65 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:21:49.878941 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ccb3c6a8-54f6-46cd-a50f-d2de2b25eb65 admin admin] RBAC: Authorization granted {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:21:49.890465 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 794/1615] 10.222.0.27 () {58 vars in 1122 bytes} [Thu Oct 13 04:21:49 2022] GET /identity/v3/roles => generated 1420 bytes in 36 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Oct 13 04:21:49.894601 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-34a73af6-94ea-4be2-a1ed-d3dbfd30c791 admin admin] Authenticating user token {{(pid=81409) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:21:49.906801 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-34a73af6-94ea-4be2-a1ed-d3dbfd30c791 admin admin] Validating token access rules against request {{(pid=81409) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:21:49.911605 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-02e6d933-42be-404c-9e32-dbadb3c851e7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8ba823f313144da3b683bacef8fd62e0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81409) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:21:49.912241 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-02e6d933-42be-404c-9e32-dbadb3c851e7 admin admin] REQUEST_METHOD: `GET` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:21:49.912778 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-02e6d933-42be-404c-9e32-dbadb3c851e7 admin admin] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:21:49.913051 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-02e6d933-42be-404c-9e32-dbadb3c851e7 admin admin] PATH_INFO: `/v3/roles` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:21:49.913692 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-02e6d933-42be-404c-9e32-dbadb3c851e7 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:21:49.915322 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-02e6d933-42be-404c-9e32-dbadb3c851e7 admin admin] RBAC: Authorization granted {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:21:49.927756 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 822/1616] 10.222.0.27 () {58 vars in 1122 bytes} [Thu Oct 13 04:21:49 2022] GET /identity/v3/roles => generated 1420 bytes in 34 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 13 04:21:49.931957 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ccb3c6a8-54f6-46cd-a50f-d2de2b25eb65 admin admin] Authenticating user token {{(pid=81408) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:21:49.943040 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ccb3c6a8-54f6-46cd-a50f-d2de2b25eb65 admin admin] Validating token access rules against request {{(pid=81408) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:21:49.948354 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9c45b9ab-62b4-4fc9-aaa6-9cabf0e0670f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8ba823f313144da3b683bacef8fd62e0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81408) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:21:49.949186 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c45b9ab-62b4-4fc9-aaa6-9cabf0e0670f admin admin] REQUEST_METHOD: `PUT` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:21:49.949617 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c45b9ab-62b4-4fc9-aaa6-9cabf0e0670f admin admin] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:21:49.950196 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c45b9ab-62b4-4fc9-aaa6-9cabf0e0670f admin admin] PATH_INFO: `/v3/projects/f5cff077cac34052ad41eb6d02ba1f59/users/0c99078cec6148139f4ef59a54d3ad12/roles/cc56c4ae13a7424ea771c15da4b153c1` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:21:49.968598 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9c45b9ab-62b4-4fc9-aaa6-9cabf0e0670f admin admin] RBAC: Authorizing `identity:create_grant(project_id=f5cff077cac34052ad41eb6d02ba1f59, user_id=0c99078cec6148139f4ef59a54d3ad12, role_id=cc56c4ae13a7424ea771c15da4b153c1)` {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:21:49.971455 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9c45b9ab-62b4-4fc9-aaa6-9cabf0e0670f admin admin] RBAC: Authorization granted {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:21:49.999071 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 795/1617] 10.222.0.27 () {60 vars in 1483 bytes} [Thu Oct 13 04:21:49 2022] PUT /identity/v3/projects/f5cff077cac34052ad41eb6d02ba1f59/users/0c99078cec6148139f4ef59a54d3ad12/roles/cc56c4ae13a7424ea771c15da4b153c1 => generated 0 bytes in 68 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 13 04:21:50.010399 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a62e3a4f-83ae-4f29-8770-98fda88dd284 None None] REQUEST_METHOD: `POST` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:21:50.010685 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a62e3a4f-83ae-4f29-8770-98fda88dd284 None None] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:21:50.010960 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a62e3a4f-83ae-4f29-8770-98fda88dd284 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:21:50.043357 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.auth.core [None req-a62e3a4f-83ae-4f29-8770-98fda88dd284 None None] MFA Rules not processed for user `0c99078cec6148139f4ef59a54d3ad12`. Rule list: `[]` (Enabled: `True`). {{(pid=81409) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 13 04:21:50.046129 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.fernet_utils [None req-a62e3a4f-83ae-4f29-8770-98fda88dd284 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81409) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:21:50.122687 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 823/1618] 10.222.0.27 () {58 vars in 958 bytes} [Thu Oct 13 04:21:50 2022] POST /identity/v3/auth/tokens => generated 3264 bytes in 113 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 13 04:21:50.127835 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9c45b9ab-62b4-4fc9-aaa6-9cabf0e0670f admin admin] Authenticating user token {{(pid=81408) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:21:50.132231 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG dbcounter [-] [81408] Writing DB stats keystone:SELECT=86,keystone:INSERT=4,keystone:DELETE=10 {{(pid=81408) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:21:50.196367 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9c45b9ab-62b4-4fc9-aaa6-9cabf0e0670f admin admin] Validating token access rules against request {{(pid=81408) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:21:50.202142 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2aa2e39d-63fa-46e8-bf55-d6ff05e8ac97 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8ba823f313144da3b683bacef8fd62e0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81408) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:21:50.202777 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2aa2e39d-63fa-46e8-bf55-d6ff05e8ac97 admin admin] REQUEST_METHOD: `POST` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:21:50.203057 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2aa2e39d-63fa-46e8-bf55-d6ff05e8ac97 admin admin] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:21:50.203319 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2aa2e39d-63fa-46e8-bf55-d6ff05e8ac97 admin admin] PATH_INFO: `/v3/projects` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:21:50.203948 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2aa2e39d-63fa-46e8-bf55-d6ff05e8ac97 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:21:50.205846 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2aa2e39d-63fa-46e8-bf55-d6ff05e8ac97 admin admin] RBAC: Authorization granted {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:21:50.240594 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 796/1619] 10.222.0.27 () {60 vars in 1153 bytes} [Thu Oct 13 04:21:50 2022] POST /identity/v3/projects => generated 387 bytes in 110 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 13 04:21:50.243093 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a62e3a4f-83ae-4f29-8770-98fda88dd284 None None] Authenticating user token {{(pid=81409) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:21:50.255050 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a62e3a4f-83ae-4f29-8770-98fda88dd284 None None] Validating token access rules against request {{(pid=81409) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:21:50.266478 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG dbcounter [-] [81409] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81409) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:21:50.322625 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-64680bc4-be84-4896-8e2e-36a2bc9cae53 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8ba823f313144da3b683bacef8fd62e0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81409) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:21:50.323241 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64680bc4-be84-4896-8e2e-36a2bc9cae53 admin admin] REQUEST_METHOD: `POST` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:21:50.323514 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64680bc4-be84-4896-8e2e-36a2bc9cae53 admin admin] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:21:50.323798 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64680bc4-be84-4896-8e2e-36a2bc9cae53 admin admin] PATH_INFO: `/v3/users` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:21:50.324426 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-64680bc4-be84-4896-8e2e-36a2bc9cae53 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:21:50.326949 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-64680bc4-be84-4896-8e2e-36a2bc9cae53 admin admin] RBAC: Authorization granted {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:21:50.349165 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 824/1620] 10.222.0.27 () {60 vars in 1144 bytes} [Thu Oct 13 04:21:50 2022] POST /identity/v3/users => generated 300 bytes in 107 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 13 04:21:50.353350 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2aa2e39d-63fa-46e8-bf55-d6ff05e8ac97 admin admin] Authenticating user token {{(pid=81408) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:21:50.364788 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2aa2e39d-63fa-46e8-bf55-d6ff05e8ac97 admin admin] Validating token access rules against request {{(pid=81408) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:21:50.369628 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6ea3f478-9075-4b85-9308-70dab90b16cf admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8ba823f313144da3b683bacef8fd62e0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81408) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:21:50.370263 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ea3f478-9075-4b85-9308-70dab90b16cf admin admin] REQUEST_METHOD: `GET` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:21:50.370542 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ea3f478-9075-4b85-9308-70dab90b16cf admin admin] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:21:50.370808 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ea3f478-9075-4b85-9308-70dab90b16cf admin admin] PATH_INFO: `/v3/roles` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:21:50.371449 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6ea3f478-9075-4b85-9308-70dab90b16cf admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:21:50.373317 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6ea3f478-9075-4b85-9308-70dab90b16cf admin admin] RBAC: Authorization granted {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:21:50.384095 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 797/1621] 10.222.0.27 () {58 vars in 1122 bytes} [Thu Oct 13 04:21:50 2022] GET /identity/v3/roles => generated 1420 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 13 04:21:50.389186 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-64680bc4-be84-4896-8e2e-36a2bc9cae53 admin admin] Authenticating user token {{(pid=81409) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:21:50.399227 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-64680bc4-be84-4896-8e2e-36a2bc9cae53 admin admin] Validating token access rules against request {{(pid=81409) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:21:50.404938 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4fe65c17-ab81-4834-9e04-c6d49e532914 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8ba823f313144da3b683bacef8fd62e0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81409) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:21:50.405596 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4fe65c17-ab81-4834-9e04-c6d49e532914 admin admin] REQUEST_METHOD: `PUT` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:21:50.405866 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4fe65c17-ab81-4834-9e04-c6d49e532914 admin admin] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:21:50.406131 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4fe65c17-ab81-4834-9e04-c6d49e532914 admin admin] PATH_INFO: `/v3/projects/663ccd212ee6498996d9ca2ff9c77927/users/1750c6bde3ae401dbb44e2db6fb4fc37/roles/ef55fbd94e1040be83695754075a7ed5` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:21:50.421887 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4fe65c17-ab81-4834-9e04-c6d49e532914 admin admin] RBAC: Authorizing `identity:create_grant(project_id=663ccd212ee6498996d9ca2ff9c77927, user_id=1750c6bde3ae401dbb44e2db6fb4fc37, role_id=ef55fbd94e1040be83695754075a7ed5)` {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:21:50.424603 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4fe65c17-ab81-4834-9e04-c6d49e532914 admin admin] RBAC: Authorization granted {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:21:50.438090 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 825/1622] 10.222.0.27 () {60 vars in 1483 bytes} [Thu Oct 13 04:21:50 2022] PUT /identity/v3/projects/663ccd212ee6498996d9ca2ff9c77927/users/1750c6bde3ae401dbb44e2db6fb4fc37/roles/ef55fbd94e1040be83695754075a7ed5 => generated 0 bytes in 50 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 13 04:21:50.441962 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6ea3f478-9075-4b85-9308-70dab90b16cf admin admin] Authenticating user token {{(pid=81408) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:21:50.453070 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6ea3f478-9075-4b85-9308-70dab90b16cf admin admin] Validating token access rules against request {{(pid=81408) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:21:50.511816 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-380a8e2c-504f-4e99-87b1-e3c74ee8d26a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8ba823f313144da3b683bacef8fd62e0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81408) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:21:50.512968 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-380a8e2c-504f-4e99-87b1-e3c74ee8d26a admin admin] REQUEST_METHOD: `GET` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:21:50.513480 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-380a8e2c-504f-4e99-87b1-e3c74ee8d26a admin admin] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:21:50.513868 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-380a8e2c-504f-4e99-87b1-e3c74ee8d26a admin admin] PATH_INFO: `/v3/roles` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:21:50.514541 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-380a8e2c-504f-4e99-87b1-e3c74ee8d26a admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:21:50.516155 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-380a8e2c-504f-4e99-87b1-e3c74ee8d26a admin admin] RBAC: Authorization granted {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:21:50.526370 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 798/1623] 10.222.0.27 () {58 vars in 1122 bytes} [Thu Oct 13 04:21:50 2022] GET /identity/v3/roles => generated 1420 bytes in 85 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 13 04:21:50.531402 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4fe65c17-ab81-4834-9e04-c6d49e532914 admin admin] Authenticating user token {{(pid=81409) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:21:50.542309 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4fe65c17-ab81-4834-9e04-c6d49e532914 admin admin] Validating token access rules against request {{(pid=81409) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:21:50.547289 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f5cf8ef5-15b3-4382-a802-6821498634f9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8ba823f313144da3b683bacef8fd62e0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81409) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:21:50.547935 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5cf8ef5-15b3-4382-a802-6821498634f9 admin admin] REQUEST_METHOD: `PUT` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:21:50.548205 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5cf8ef5-15b3-4382-a802-6821498634f9 admin admin] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:21:50.548470 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5cf8ef5-15b3-4382-a802-6821498634f9 admin admin] PATH_INFO: `/v3/projects/663ccd212ee6498996d9ca2ff9c77927/users/1750c6bde3ae401dbb44e2db6fb4fc37/roles/ef55fbd94e1040be83695754075a7ed5` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:21:50.550380 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f5cf8ef5-15b3-4382-a802-6821498634f9 admin admin] RBAC: Authorizing `identity:create_grant(project_id=663ccd212ee6498996d9ca2ff9c77927, user_id=1750c6bde3ae401dbb44e2db6fb4fc37, role_id=ef55fbd94e1040be83695754075a7ed5)` {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:21:50.552993 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f5cf8ef5-15b3-4382-a802-6821498634f9 admin admin] RBAC: Authorization granted {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:21:50.564424 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 826/1624] 10.222.0.27 () {60 vars in 1483 bytes} [Thu Oct 13 04:21:50 2022] PUT /identity/v3/projects/663ccd212ee6498996d9ca2ff9c77927/users/1750c6bde3ae401dbb44e2db6fb4fc37/roles/ef55fbd94e1040be83695754075a7ed5 => generated 0 bytes in 34 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 13 04:21:50.575506 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc6fc58c-30c0-4232-8ec8-c6c32202e90c None None] REQUEST_METHOD: `POST` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:21:50.575829 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc6fc58c-30c0-4232-8ec8-c6c32202e90c None None] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:21:50.576106 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc6fc58c-30c0-4232-8ec8-c6c32202e90c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:21:50.603368 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.auth.core [None req-bc6fc58c-30c0-4232-8ec8-c6c32202e90c None None] MFA Rules not processed for user `1750c6bde3ae401dbb44e2db6fb4fc37`. Rule list: `[]` (Enabled: `True`). {{(pid=81408) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 13 04:21:50.604581 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.fernet_utils [None req-bc6fc58c-30c0-4232-8ec8-c6c32202e90c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81408) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:21:50.673520 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG dbcounter [-] [81408] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81408) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:21:50.682116 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 799/1625] 10.222.0.27 () {58 vars in 958 bytes} [Thu Oct 13 04:21:50 2022] POST /identity/v3/auth/tokens => generated 3331 bytes in 108 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 13 04:21:50.695886 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f5cf8ef5-15b3-4382-a802-6821498634f9 admin admin] Authenticating user token {{(pid=81409) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:21:50.698191 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.fernet_utils [None req-f5cf8ef5-15b3-4382-a802-6821498634f9 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81409) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:21:50.765802 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f5cf8ef5-15b3-4382-a802-6821498634f9 admin admin] Validating token access rules against request {{(pid=81409) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:21:50.770916 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-34cf7c5d-b90d-47b8-ac23-81b24aae784f service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6141b8a6c3d54e179d6879bb019f4d12', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dcec0431fb6849499d6aec35a234daa6', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81409) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:21:50.771638 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34cf7c5d-b90d-47b8-ac23-81b24aae784f service cinder] REQUEST_METHOD: `GET` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:21:50.771998 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34cf7c5d-b90d-47b8-ac23-81b24aae784f service cinder] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:21:50.772516 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34cf7c5d-b90d-47b8-ac23-81b24aae784f service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:21:50.786309 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-34cf7c5d-b90d-47b8-ac23-81b24aae784f service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:21:50.788247 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-34cf7c5d-b90d-47b8-ac23-81b24aae784f service cinder] RBAC: Authorization granted {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:21:50.806847 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 827/1626] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Oct 13 04:21:50 2022] GET /identity/v3/auth/tokens => generated 3331 bytes in 111 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 13 04:21:51.660663 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bc6fc58c-30c0-4232-8ec8-c6c32202e90c None None] Authenticating user token {{(pid=81408) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:21:51.741677 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bc6fc58c-30c0-4232-8ec8-c6c32202e90c None None] Validating token access rules against request {{(pid=81408) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:21:51.753471 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1500b522-5519-462d-afde-5ff92706cd55 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8ba823f313144da3b683bacef8fd62e0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81408) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:21:51.754339 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1500b522-5519-462d-afde-5ff92706cd55 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:21:51.754691 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1500b522-5519-462d-afde-5ff92706cd55 admin admin] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:21:51.755039 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1500b522-5519-462d-afde-5ff92706cd55 admin admin] PATH_INFO: `/v3/users/0c99078cec6148139f4ef59a54d3ad12` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:21:51.756683 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1500b522-5519-462d-afde-5ff92706cd55 admin admin] RBAC: Authorizing `identity:delete_user(user_id=0c99078cec6148139f4ef59a54d3ad12)` {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:21:51.781387 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1500b522-5519-462d-afde-5ff92706cd55 admin admin] RBAC: Authorization granted {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:21:51.847167 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-34cf7c5d-b90d-47b8-ac23-81b24aae784f service cinder] Authenticating user token {{(pid=81409) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:21:51.849447 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.fernet_utils [None req-34cf7c5d-b90d-47b8-ac23-81b24aae784f service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81409) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:21:51.873729 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-34cf7c5d-b90d-47b8-ac23-81b24aae784f service cinder] Validating token access rules against request {{(pid=81409) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:21:51.877833 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.notifications [None req-1500b522-5519-462d-afde-5ff92706cd55 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '0c99078cec6148139f4ef59a54d3ad12'} {{(pid=81408) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 13 04:21:51.880950 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c0c96442-016d-4756-b11b-5d1ac66a6fbb admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8ba823f313144da3b683bacef8fd62e0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81409) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:21:51.881813 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0c96442-016d-4756-b11b-5d1ac66a6fbb admin admin] REQUEST_METHOD: `DELETE` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:21:51.882088 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0c96442-016d-4756-b11b-5d1ac66a6fbb admin admin] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:21:51.882533 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0c96442-016d-4756-b11b-5d1ac66a6fbb admin admin] PATH_INFO: `/v3/users/01498f56b62246339c7cd9d3cead50ac` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:21:51.884388 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c0c96442-016d-4756-b11b-5d1ac66a6fbb admin admin] RBAC: Authorizing `identity:delete_user(user_id=01498f56b62246339c7cd9d3cead50ac)` {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:21:51.888838 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.notifications [None req-1500b522-5519-462d-afde-5ff92706cd55 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '0c99078cec6148139f4ef59a54d3ad12'} {{(pid=81408) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 13 04:21:51.889520 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c0c96442-016d-4756-b11b-5d1ac66a6fbb admin admin] RBAC: Authorization granted {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:21:51.889627 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.notifications [None req-1500b522-5519-462d-afde-5ff92706cd55 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '0c99078cec6148139f4ef59a54d3ad12'} {{(pid=81408) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 13 04:21:51.890157 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.notifications [None req-1500b522-5519-462d-afde-5ff92706cd55 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '0c99078cec6148139f4ef59a54d3ad12'} {{(pid=81408) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 13 04:21:51.900442 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.notifications [None req-1500b522-5519-462d-afde-5ff92706cd55 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '0c99078cec6148139f4ef59a54d3ad12'} {{(pid=81408) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 13 04:21:51.932305 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 800/1627] 10.222.0.27 () {58 vars in 1224 bytes} [Thu Oct 13 04:21:51 2022] DELETE /identity/v3/users/0c99078cec6148139f4ef59a54d3ad12 => generated 0 bytes in 272 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 13 04:21:51.941857 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65b6fa0f-1dbc-4438-8765-05bac37e555f None None] REQUEST_METHOD: `POST` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:21:51.941857 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65b6fa0f-1dbc-4438-8765-05bac37e555f None None] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:21:51.941857 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65b6fa0f-1dbc-4438-8765-05bac37e555f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:21:51.961361 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.notifications [None req-c0c96442-016d-4756-b11b-5d1ac66a6fbb admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '01498f56b62246339c7cd9d3cead50ac'} {{(pid=81409) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 13 04:21:51.962512 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.notifications [None req-c0c96442-016d-4756-b11b-5d1ac66a6fbb admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '01498f56b62246339c7cd9d3cead50ac'} {{(pid=81409) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 13 04:21:51.971598 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.notifications [None req-c0c96442-016d-4756-b11b-5d1ac66a6fbb admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '01498f56b62246339c7cd9d3cead50ac'} {{(pid=81409) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 13 04:21:51.972691 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.notifications [None req-c0c96442-016d-4756-b11b-5d1ac66a6fbb admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '01498f56b62246339c7cd9d3cead50ac'} {{(pid=81409) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 13 04:21:51.973918 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.auth.core [None req-65b6fa0f-1dbc-4438-8765-05bac37e555f None None] MFA Rules not processed for user `0e2d979563d6421995451a0e86bf3022`. Rule list: `[]` (Enabled: `True`). {{(pid=81408) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 13 04:21:51.975552 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.fernet_utils [None req-65b6fa0f-1dbc-4438-8765-05bac37e555f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81408) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:21:51.984989 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.notifications [None req-c0c96442-016d-4756-b11b-5d1ac66a6fbb admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '01498f56b62246339c7cd9d3cead50ac'} {{(pid=81409) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 13 04:21:51.987131 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG dbcounter [-] [81409] Writing DB stats keystone:SELECT=86,keystone:INSERT=6,keystone:DELETE=8 {{(pid=81409) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:21:52.014567 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 828/1628] 10.222.0.27 () {58 vars in 1224 bytes} [Thu Oct 13 04:21:51 2022] DELETE /identity/v3/users/01498f56b62246339c7cd9d3cead50ac => generated 0 bytes in 169 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 13 04:21:52.020148 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b587feb9-cde8-47c2-ba42-12df5eaf5ed5 None None] REQUEST_METHOD: `POST` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:21:52.020508 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b587feb9-cde8-47c2-ba42-12df5eaf5ed5 None None] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:21:52.020694 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b587feb9-cde8-47c2-ba42-12df5eaf5ed5 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:21:52.043842 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.auth.core [None req-b587feb9-cde8-47c2-ba42-12df5eaf5ed5 None None] MFA Rules not processed for user `0e2d979563d6421995451a0e86bf3022`. Rule list: `[]` (Enabled: `True`). {{(pid=81409) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 13 04:21:52.045063 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.fernet_utils [None req-b587feb9-cde8-47c2-ba42-12df5eaf5ed5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81409) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:21:52.065077 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 801/1629] 10.222.0.27 () {58 vars in 958 bytes} [Thu Oct 13 04:21:51 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 128 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 13 04:21:52.077276 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-65b6fa0f-1dbc-4438-8765-05bac37e555f None None] Authenticating user token {{(pid=81408) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:21:52.080130 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.fernet_utils [None req-65b6fa0f-1dbc-4438-8765-05bac37e555f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81408) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:21:52.114276 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG dbcounter [-] [81408] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81408) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:21:52.167926 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 829/1630] 10.222.0.27 () {58 vars in 958 bytes} [Thu Oct 13 04:21:52 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 149 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 13 04:21:52.193532 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-65b6fa0f-1dbc-4438-8765-05bac37e555f None None] Validating token access rules against request {{(pid=81408) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:21:52.202639 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b587feb9-cde8-47c2-ba42-12df5eaf5ed5 None None] Authenticating user token {{(pid=81409) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:21:52.206522 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.fernet_utils [None req-b587feb9-cde8-47c2-ba42-12df5eaf5ed5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81409) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:21:52.291630 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3186c971-ccaa-40b9-b274-520e959dcf7a service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd8812c3974f84871bb2d761cf0016257', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dcec0431fb6849499d6aec35a234daa6', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81408) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:21:52.292687 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3186c971-ccaa-40b9-b274-520e959dcf7a service neutron] REQUEST_METHOD: `GET` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:21:52.293195 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3186c971-ccaa-40b9-b274-520e959dcf7a service neutron] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:21:52.305532 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3186c971-ccaa-40b9-b274-520e959dcf7a service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:21:52.337149 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3186c971-ccaa-40b9-b274-520e959dcf7a service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:21:52.341969 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3186c971-ccaa-40b9-b274-520e959dcf7a service neutron] RBAC: Authorization granted {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:21:52.358553 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 802/1631] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Oct 13 04:21:52 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 284 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Oct 13 04:21:52.423748 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b587feb9-cde8-47c2-ba42-12df5eaf5ed5 None None] Validating token access rules against request {{(pid=81409) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:21:52.444310 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-82759861-6e1a-42a2-a988-801095c93dc4 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd8812c3974f84871bb2d761cf0016257', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dcec0431fb6849499d6aec35a234daa6', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81409) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:21:52.445178 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82759861-6e1a-42a2-a988-801095c93dc4 service neutron] REQUEST_METHOD: `GET` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:21:52.445453 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82759861-6e1a-42a2-a988-801095c93dc4 service neutron] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:21:52.445740 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82759861-6e1a-42a2-a988-801095c93dc4 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:21:52.470627 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-82759861-6e1a-42a2-a988-801095c93dc4 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:21:52.470983 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-82759861-6e1a-42a2-a988-801095c93dc4 service neutron] RBAC: Authorization granted {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:21:52.485746 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 830/1632] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Oct 13 04:21:52 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 298 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 13 04:21:52.902489 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3186c971-ccaa-40b9-b274-520e959dcf7a service neutron] Authenticating user token {{(pid=81408) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:21:52.905386 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.fernet_utils [None req-3186c971-ccaa-40b9-b274-520e959dcf7a service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81408) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:21:52.932776 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3186c971-ccaa-40b9-b274-520e959dcf7a service neutron] Validating token access rules against request {{(pid=81408) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:21:52.944978 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9af1fec9-16fd-4253-9106-0f75bc25c212 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8ba823f313144da3b683bacef8fd62e0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81408) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:21:52.946349 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9af1fec9-16fd-4253-9106-0f75bc25c212 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:21:52.946818 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9af1fec9-16fd-4253-9106-0f75bc25c212 admin admin] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:21:52.947318 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9af1fec9-16fd-4253-9106-0f75bc25c212 admin admin] PATH_INFO: `/v3/projects/f5cff077cac34052ad41eb6d02ba1f59` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:21:52.953710 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9af1fec9-16fd-4253-9106-0f75bc25c212 admin admin] RBAC: Authorizing `identity:delete_project(project_id=f5cff077cac34052ad41eb6d02ba1f59)` {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:21:52.956985 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9af1fec9-16fd-4253-9106-0f75bc25c212 admin admin] RBAC: Authorization granted {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:21:53.027480 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.notifications [None req-9af1fec9-16fd-4253-9106-0f75bc25c212 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'f5cff077cac34052ad41eb6d02ba1f59', 'request_id': 'req-9af1fec9-16fd-4253-9106-0f75bc25c212'} {{(pid=81408) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 13 04:21:53.046334 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.notifications [None req-9af1fec9-16fd-4253-9106-0f75bc25c212 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'f5cff077cac34052ad41eb6d02ba1f59', 'request_id': 'req-9af1fec9-16fd-4253-9106-0f75bc25c212'} {{(pid=81408) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 13 04:21:53.054995 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.notifications [None req-9af1fec9-16fd-4253-9106-0f75bc25c212 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'f5cff077cac34052ad41eb6d02ba1f59', 'request_id': 'req-9af1fec9-16fd-4253-9106-0f75bc25c212'} {{(pid=81408) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 13 04:21:53.075605 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.notifications [None req-9af1fec9-16fd-4253-9106-0f75bc25c212 admin admin] The token cache is being invalidate because project f5cff077cac34052ad41eb6d02ba1f59 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81408) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 13 04:21:53.076711 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.notifications [None req-9af1fec9-16fd-4253-9106-0f75bc25c212 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81408) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 13 04:21:53.077619 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.notifications [None req-9af1fec9-16fd-4253-9106-0f75bc25c212 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81408) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 13 04:21:53.080360 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 803/1633] 10.222.0.27 () {58 vars in 1233 bytes} [Thu Oct 13 04:21:52 2022] DELETE /identity/v3/projects/f5cff077cac34052ad41eb6d02ba1f59 => generated 0 bytes in 179 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 13 04:21:53.084021 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-82759861-6e1a-42a2-a988-801095c93dc4 service neutron] Authenticating user token {{(pid=81409) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:21:53.097044 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-82759861-6e1a-42a2-a988-801095c93dc4 service neutron] Validating token access rules against request {{(pid=81409) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:21:53.187325 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-748b2375-311e-44b8-b6ec-7fc1e55ad820 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8ba823f313144da3b683bacef8fd62e0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81409) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:21:53.188454 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-748b2375-311e-44b8-b6ec-7fc1e55ad820 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:21:53.188936 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-748b2375-311e-44b8-b6ec-7fc1e55ad820 admin admin] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:21:53.189416 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-748b2375-311e-44b8-b6ec-7fc1e55ad820 admin admin] PATH_INFO: `/v3/users/1750c6bde3ae401dbb44e2db6fb4fc37` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:21:53.190838 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-748b2375-311e-44b8-b6ec-7fc1e55ad820 admin admin] RBAC: Authorizing `identity:delete_user(user_id=1750c6bde3ae401dbb44e2db6fb4fc37)` {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:21:53.193078 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-748b2375-311e-44b8-b6ec-7fc1e55ad820 admin admin] RBAC: Authorization granted {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:21:53.234447 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG dbcounter [-] [81409] Writing DB stats keystone:SELECT=94,keystone:DELETE=6 {{(pid=81409) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:21:53.246518 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9af1fec9-16fd-4253-9106-0f75bc25c212 admin admin] Authenticating user token {{(pid=81408) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:21:53.264968 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.fernet_utils [None req-9af1fec9-16fd-4253-9106-0f75bc25c212 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81408) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:21:53.281405 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9af1fec9-16fd-4253-9106-0f75bc25c212 admin admin] Validating token access rules against request {{(pid=81408) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:21:53.282331 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.notifications [None req-748b2375-311e-44b8-b6ec-7fc1e55ad820 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '1750c6bde3ae401dbb44e2db6fb4fc37'} {{(pid=81409) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 13 04:21:53.283221 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.notifications [None req-748b2375-311e-44b8-b6ec-7fc1e55ad820 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '1750c6bde3ae401dbb44e2db6fb4fc37'} {{(pid=81409) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 13 04:21:53.287548 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4e2f3544-3e6c-4961-ba84-f279e66d5bd6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8ba823f313144da3b683bacef8fd62e0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81408) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:21:53.288255 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e2f3544-3e6c-4961-ba84-f279e66d5bd6 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:21:53.288453 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e2f3544-3e6c-4961-ba84-f279e66d5bd6 admin admin] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:21:53.288723 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e2f3544-3e6c-4961-ba84-f279e66d5bd6 admin admin] PATH_INFO: `/v3/projects/95e7139479ed41fa8e7a1cf6e3e874d7` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:21:53.291290 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4e2f3544-3e6c-4961-ba84-f279e66d5bd6 admin admin] RBAC: Authorizing `identity:delete_project(project_id=95e7139479ed41fa8e7a1cf6e3e874d7)` {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:21:53.293021 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.notifications [None req-748b2375-311e-44b8-b6ec-7fc1e55ad820 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '1750c6bde3ae401dbb44e2db6fb4fc37'} {{(pid=81409) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 13 04:21:53.294095 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.notifications [None req-748b2375-311e-44b8-b6ec-7fc1e55ad820 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '1750c6bde3ae401dbb44e2db6fb4fc37'} {{(pid=81409) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 13 04:21:53.294320 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4e2f3544-3e6c-4961-ba84-f279e66d5bd6 admin admin] RBAC: Authorization granted {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:21:53.306078 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.notifications [None req-748b2375-311e-44b8-b6ec-7fc1e55ad820 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '1750c6bde3ae401dbb44e2db6fb4fc37'} {{(pid=81409) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 13 04:21:53.329303 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 831/1634] 10.222.0.27 () {58 vars in 1224 bytes} [Thu Oct 13 04:21:53 2022] DELETE /identity/v3/users/1750c6bde3ae401dbb44e2db6fb4fc37 => generated 0 bytes in 246 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 13 04:21:53.355426 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.notifications [None req-4e2f3544-3e6c-4961-ba84-f279e66d5bd6 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '95e7139479ed41fa8e7a1cf6e3e874d7', 'request_id': 'req-4e2f3544-3e6c-4961-ba84-f279e66d5bd6'} {{(pid=81408) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 13 04:21:53.360170 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG dbcounter [-] [81408] Writing DB stats keystone:SELECT=86,keystone:DELETE=12,keystone:INSERT=2 {{(pid=81408) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:21:53.365018 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.notifications [None req-4e2f3544-3e6c-4961-ba84-f279e66d5bd6 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '95e7139479ed41fa8e7a1cf6e3e874d7', 'request_id': 'req-4e2f3544-3e6c-4961-ba84-f279e66d5bd6'} {{(pid=81408) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 13 04:21:53.373611 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.notifications [None req-4e2f3544-3e6c-4961-ba84-f279e66d5bd6 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '95e7139479ed41fa8e7a1cf6e3e874d7', 'request_id': 'req-4e2f3544-3e6c-4961-ba84-f279e66d5bd6'} {{(pid=81408) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 13 04:21:53.388156 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.notifications [None req-4e2f3544-3e6c-4961-ba84-f279e66d5bd6 admin admin] The token cache is being invalidate because project 95e7139479ed41fa8e7a1cf6e3e874d7 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81408) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 13 04:21:53.388515 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.notifications [None req-4e2f3544-3e6c-4961-ba84-f279e66d5bd6 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81408) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 13 04:21:53.389283 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.notifications [None req-4e2f3544-3e6c-4961-ba84-f279e66d5bd6 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81408) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 13 04:21:53.391057 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 804/1635] 10.222.0.27 () {58 vars in 1233 bytes} [Thu Oct 13 04:21:53 2022] DELETE /identity/v3/projects/95e7139479ed41fa8e7a1cf6e3e874d7 => generated 0 bytes in 145 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 13 04:21:53.395256 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-748b2375-311e-44b8-b6ec-7fc1e55ad820 admin admin] Authenticating user token {{(pid=81409) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:21:53.397134 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.fernet_utils [None req-748b2375-311e-44b8-b6ec-7fc1e55ad820 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81409) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:21:53.491593 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-748b2375-311e-44b8-b6ec-7fc1e55ad820 admin admin] Validating token access rules against request {{(pid=81409) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:21:53.578432 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cfc750c1-19d7-4525-9d1e-f382c6e2052a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8ba823f313144da3b683bacef8fd62e0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81409) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:21:53.583114 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cfc750c1-19d7-4525-9d1e-f382c6e2052a admin admin] REQUEST_METHOD: `DELETE` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:21:53.583676 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cfc750c1-19d7-4525-9d1e-f382c6e2052a admin admin] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:21:53.584307 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cfc750c1-19d7-4525-9d1e-f382c6e2052a admin admin] PATH_INFO: `/v3/users/1e075f9fd76d46eda970c96b03c1cec2` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:21:53.586243 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cfc750c1-19d7-4525-9d1e-f382c6e2052a admin admin] RBAC: Authorizing `identity:delete_user(user_id=1e075f9fd76d46eda970c96b03c1cec2)` {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:21:53.588603 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cfc750c1-19d7-4525-9d1e-f382c6e2052a admin admin] RBAC: Authorization granted {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:21:53.647572 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.notifications [None req-cfc750c1-19d7-4525-9d1e-f382c6e2052a admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '1e075f9fd76d46eda970c96b03c1cec2'} {{(pid=81409) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 13 04:21:53.648267 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.notifications [None req-cfc750c1-19d7-4525-9d1e-f382c6e2052a admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '1e075f9fd76d46eda970c96b03c1cec2'} {{(pid=81409) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 13 04:21:53.654032 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG dbcounter [-] [81409] Writing DB stats keystone:DELETE=13,keystone:SELECT=86,keystone:INSERT=1 {{(pid=81409) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:21:53.657673 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.notifications [None req-cfc750c1-19d7-4525-9d1e-f382c6e2052a admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '1e075f9fd76d46eda970c96b03c1cec2'} {{(pid=81409) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 13 04:21:53.659228 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.notifications [None req-cfc750c1-19d7-4525-9d1e-f382c6e2052a admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '1e075f9fd76d46eda970c96b03c1cec2'} {{(pid=81409) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 13 04:21:53.667278 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.notifications [None req-cfc750c1-19d7-4525-9d1e-f382c6e2052a admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '1e075f9fd76d46eda970c96b03c1cec2'} {{(pid=81409) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 13 04:21:53.678909 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4e2f3544-3e6c-4961-ba84-f279e66d5bd6 admin admin] Authenticating user token {{(pid=81408) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:21:53.680541 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.fernet_utils [None req-4e2f3544-3e6c-4961-ba84-f279e66d5bd6 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81408) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:21:53.690972 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 832/1636] 10.222.0.27 () {58 vars in 1224 bytes} [Thu Oct 13 04:21:53 2022] DELETE /identity/v3/users/1e075f9fd76d46eda970c96b03c1cec2 => generated 0 bytes in 297 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 13 04:21:53.698460 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4e2f3544-3e6c-4961-ba84-f279e66d5bd6 admin admin] Validating token access rules against request {{(pid=81408) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:21:53.770672 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cf77818c-0087-4acf-88a1-ff4487e4797e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8ba823f313144da3b683bacef8fd62e0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81408) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:21:53.771465 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf77818c-0087-4acf-88a1-ff4487e4797e admin admin] REQUEST_METHOD: `DELETE` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:21:53.771764 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf77818c-0087-4acf-88a1-ff4487e4797e admin admin] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:21:53.771860 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf77818c-0087-4acf-88a1-ff4487e4797e admin admin] PATH_INFO: `/v3/projects/663ccd212ee6498996d9ca2ff9c77927` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:21:53.773207 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cf77818c-0087-4acf-88a1-ff4487e4797e admin admin] RBAC: Authorizing `identity:delete_project(project_id=663ccd212ee6498996d9ca2ff9c77927)` {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:21:53.775190 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cf77818c-0087-4acf-88a1-ff4487e4797e admin admin] RBAC: Authorization granted {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:21:53.827355 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.notifications [None req-cf77818c-0087-4acf-88a1-ff4487e4797e admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '663ccd212ee6498996d9ca2ff9c77927', 'request_id': 'req-cf77818c-0087-4acf-88a1-ff4487e4797e'} {{(pid=81408) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 13 04:21:53.835388 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.notifications [None req-cf77818c-0087-4acf-88a1-ff4487e4797e admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '663ccd212ee6498996d9ca2ff9c77927', 'request_id': 'req-cf77818c-0087-4acf-88a1-ff4487e4797e'} {{(pid=81408) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 13 04:21:53.841985 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.notifications [None req-cf77818c-0087-4acf-88a1-ff4487e4797e admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '663ccd212ee6498996d9ca2ff9c77927', 'request_id': 'req-cf77818c-0087-4acf-88a1-ff4487e4797e'} {{(pid=81408) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 13 04:21:53.854917 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.notifications [None req-cf77818c-0087-4acf-88a1-ff4487e4797e admin admin] The token cache is being invalidate because project 663ccd212ee6498996d9ca2ff9c77927 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81408) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 13 04:21:53.855205 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.notifications [None req-cf77818c-0087-4acf-88a1-ff4487e4797e admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81408) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 13 04:21:53.855875 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.notifications [None req-cf77818c-0087-4acf-88a1-ff4487e4797e admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81408) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 13 04:21:53.858247 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 805/1637] 10.222.0.27 () {58 vars in 1233 bytes} [Thu Oct 13 04:21:53 2022] DELETE /identity/v3/projects/663ccd212ee6498996d9ca2ff9c77927 => generated 0 bytes in 180 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 13 04:21:53.863987 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4bf76b4d-1385-4342-a64f-ecccace37a81 None None] REQUEST_METHOD: `POST` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:21:53.864515 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4bf76b4d-1385-4342-a64f-ecccace37a81 None None] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:21:53.865012 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4bf76b4d-1385-4342-a64f-ecccace37a81 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:21:53.891141 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.auth.core [None req-4bf76b4d-1385-4342-a64f-ecccace37a81 None None] MFA Rules not processed for user `0e2d979563d6421995451a0e86bf3022`. Rule list: `[]` (Enabled: `True`). {{(pid=81409) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 13 04:21:53.893296 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.fernet_utils [None req-4bf76b4d-1385-4342-a64f-ecccace37a81 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81409) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:21:54.032746 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cf77818c-0087-4acf-88a1-ff4487e4797e admin admin] Authenticating user token {{(pid=81408) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:21:54.034342 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 833/1638] 10.222.0.27 () {58 vars in 958 bytes} [Thu Oct 13 04:21:53 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 172 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 13 04:21:54.034583 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.fernet_utils [None req-cf77818c-0087-4acf-88a1-ff4487e4797e admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81408) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:21:54.037089 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ce0ec03-14c3-406c-8471-52cc7bfa7684 None None] REQUEST_METHOD: `POST` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:21:54.037376 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ce0ec03-14c3-406c-8471-52cc7bfa7684 None None] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:21:54.037690 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ce0ec03-14c3-406c-8471-52cc7bfa7684 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:21:54.052305 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cf77818c-0087-4acf-88a1-ff4487e4797e admin admin] Validating token access rules against request {{(pid=81408) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:21:54.057274 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d959e61d-2c72-4196-bc36-8d11590f922d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8ba823f313144da3b683bacef8fd62e0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81408) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:21:54.057960 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d959e61d-2c72-4196-bc36-8d11590f922d admin admin] REQUEST_METHOD: `DELETE` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:21:54.058276 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d959e61d-2c72-4196-bc36-8d11590f922d admin admin] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:21:54.058648 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d959e61d-2c72-4196-bc36-8d11590f922d admin admin] PATH_INFO: `/v3/projects/1ceae26ece6c4ad0a47610d8af5047c1` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:21:54.060428 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d959e61d-2c72-4196-bc36-8d11590f922d admin admin] RBAC: Authorizing `identity:delete_project(project_id=1ceae26ece6c4ad0a47610d8af5047c1)` {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:21:54.063672 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d959e61d-2c72-4196-bc36-8d11590f922d admin admin] RBAC: Authorization granted {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:21:54.064255 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.auth.core [None req-9ce0ec03-14c3-406c-8471-52cc7bfa7684 None None] MFA Rules not processed for user `0e2d979563d6421995451a0e86bf3022`. Rule list: `[]` (Enabled: `True`). {{(pid=81409) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 13 04:21:54.065429 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.fernet_utils [None req-9ce0ec03-14c3-406c-8471-52cc7bfa7684 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81409) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:21:54.076340 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 834/1639] 10.222.0.27 () {58 vars in 958 bytes} [Thu Oct 13 04:21:54 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 40 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 13 04:21:54.080566 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9ce0ec03-14c3-406c-8471-52cc7bfa7684 None None] Authenticating user token {{(pid=81409) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:21:54.095181 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9ce0ec03-14c3-406c-8471-52cc7bfa7684 None None] Validating token access rules against request {{(pid=81409) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:21:54.100663 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-196b00c3-86db-46ba-b460-b0c995eebcf2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8ba823f313144da3b683bacef8fd62e0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81409) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:21:54.101275 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-196b00c3-86db-46ba-b460-b0c995eebcf2 admin admin] REQUEST_METHOD: `GET` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:21:54.101574 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-196b00c3-86db-46ba-b460-b0c995eebcf2 admin admin] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:21:54.101843 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-196b00c3-86db-46ba-b460-b0c995eebcf2 admin admin] PATH_INFO: `/v3/domains` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:21:54.103088 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-196b00c3-86db-46ba-b460-b0c995eebcf2 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81409) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 13 04:21:54.103400 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-196b00c3-86db-46ba-b460-b0c995eebcf2 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:21:54.105004 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-196b00c3-86db-46ba-b460-b0c995eebcf2 admin admin] RBAC: Authorization granted {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:21:54.120352 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 835/1640] 10.222.0.27 () {58 vars in 1153 bytes} [Thu Oct 13 04:21:54 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 40 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 13 04:21:54.125300 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-196b00c3-86db-46ba-b460-b0c995eebcf2 admin admin] Authenticating user token {{(pid=81409) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:21:54.133644 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.notifications [None req-d959e61d-2c72-4196-bc36-8d11590f922d admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '1ceae26ece6c4ad0a47610d8af5047c1', 'request_id': 'req-d959e61d-2c72-4196-bc36-8d11590f922d'} {{(pid=81408) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 13 04:21:54.135512 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-196b00c3-86db-46ba-b460-b0c995eebcf2 admin admin] Validating token access rules against request {{(pid=81409) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:21:54.144505 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.notifications [None req-d959e61d-2c72-4196-bc36-8d11590f922d admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '1ceae26ece6c4ad0a47610d8af5047c1', 'request_id': 'req-d959e61d-2c72-4196-bc36-8d11590f922d'} {{(pid=81408) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 13 04:21:54.153408 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.notifications [None req-d959e61d-2c72-4196-bc36-8d11590f922d admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '1ceae26ece6c4ad0a47610d8af5047c1', 'request_id': 'req-d959e61d-2c72-4196-bc36-8d11590f922d'} {{(pid=81408) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 13 04:21:54.160092 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG dbcounter [-] [81408] Writing DB stats keystone:SELECT=82,keystone:DELETE=16,keystone:INSERT=2 {{(pid=81408) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:21:54.170023 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.notifications [None req-d959e61d-2c72-4196-bc36-8d11590f922d admin admin] The token cache is being invalidate because project 1ceae26ece6c4ad0a47610d8af5047c1 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81408) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 13 04:21:54.170352 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.notifications [None req-d959e61d-2c72-4196-bc36-8d11590f922d admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81408) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 13 04:21:54.171001 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.notifications [None req-d959e61d-2c72-4196-bc36-8d11590f922d admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81408) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 13 04:21:54.173723 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 806/1641] 10.222.0.27 () {58 vars in 1233 bytes} [Thu Oct 13 04:21:54 2022] DELETE /identity/v3/projects/1ceae26ece6c4ad0a47610d8af5047c1 => generated 0 bytes in 141 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 13 04:21:54.177556 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d959e61d-2c72-4196-bc36-8d11590f922d admin admin] Authenticating user token {{(pid=81408) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:21:54.179110 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.fernet_utils [None req-d959e61d-2c72-4196-bc36-8d11590f922d admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81408) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:21:54.201272 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e9753587-b17b-4aca-831c-de88ad980167 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8ba823f313144da3b683bacef8fd62e0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81409) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:21:54.202752 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9753587-b17b-4aca-831c-de88ad980167 admin admin] REQUEST_METHOD: `POST` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:21:54.202752 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9753587-b17b-4aca-831c-de88ad980167 admin admin] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:21:54.202752 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9753587-b17b-4aca-831c-de88ad980167 admin admin] PATH_INFO: `/v3/projects` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:21:54.203114 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e9753587-b17b-4aca-831c-de88ad980167 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:21:54.205255 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e9753587-b17b-4aca-831c-de88ad980167 admin admin] RBAC: Authorization granted {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:21:54.237683 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 836/1642] 10.222.0.27 () {60 vars in 1153 bytes} [Thu Oct 13 04:21:54 2022] POST /identity/v3/projects => generated 367 bytes in 113 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 13 04:21:54.242184 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e9753587-b17b-4aca-831c-de88ad980167 admin admin] Authenticating user token {{(pid=81409) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:21:54.244022 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.fernet_utils [None req-e9753587-b17b-4aca-831c-de88ad980167 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81409) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:21:54.255465 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d959e61d-2c72-4196-bc36-8d11590f922d admin admin] Validating token access rules against request {{(pid=81408) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:21:54.292551 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG dbcounter [-] [81409] Writing DB stats keystone:SELECT=95,keystone:INSERT=2,keystone:DELETE=3 {{(pid=81409) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:21:54.328237 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e9753587-b17b-4aca-831c-de88ad980167 admin admin] Validating token access rules against request {{(pid=81409) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:21:54.333027 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5e96a126-8c27-450b-9dfc-d5c3511ebe80 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8ba823f313144da3b683bacef8fd62e0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81409) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:21:54.333999 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e96a126-8c27-450b-9dfc-d5c3511ebe80 admin admin] REQUEST_METHOD: `POST` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:21:54.333999 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e96a126-8c27-450b-9dfc-d5c3511ebe80 admin admin] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:21:54.334311 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e96a126-8c27-450b-9dfc-d5c3511ebe80 admin admin] PATH_INFO: `/v3/users` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:21:54.334895 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5e96a126-8c27-450b-9dfc-d5c3511ebe80 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:21:54.336803 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5e96a126-8c27-450b-9dfc-d5c3511ebe80 admin admin] RBAC: Authorization granted {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:21:54.344108 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-31475085-bfe2-42f8-a623-93ee8336f4cb admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8ba823f313144da3b683bacef8fd62e0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81408) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:21:54.347323 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31475085-bfe2-42f8-a623-93ee8336f4cb admin admin] REQUEST_METHOD: `DELETE` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:21:54.348550 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31475085-bfe2-42f8-a623-93ee8336f4cb admin admin] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:21:54.349329 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31475085-bfe2-42f8-a623-93ee8336f4cb admin admin] PATH_INFO: `/v3/users/987d5e2972cf4038a3ba585a4d4b6f07` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:21:54.351337 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-31475085-bfe2-42f8-a623-93ee8336f4cb admin admin] RBAC: Authorizing `identity:delete_user(user_id=987d5e2972cf4038a3ba585a4d4b6f07)` {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:21:54.353686 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-31475085-bfe2-42f8-a623-93ee8336f4cb admin admin] RBAC: Authorization granted {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:21:54.361609 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 837/1643] 10.222.0.27 () {60 vars in 1144 bytes} [Thu Oct 13 04:21:54 2022] POST /identity/v3/users => generated 284 bytes in 120 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 13 04:21:54.365477 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5e96a126-8c27-450b-9dfc-d5c3511ebe80 admin admin] Authenticating user token {{(pid=81409) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:21:54.378510 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5e96a126-8c27-450b-9dfc-d5c3511ebe80 admin admin] Validating token access rules against request {{(pid=81409) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:21:54.385386 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-30a54798-2d30-476d-b55b-ddd45af625a7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8ba823f313144da3b683bacef8fd62e0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81409) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:21:54.386095 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30a54798-2d30-476d-b55b-ddd45af625a7 admin admin] REQUEST_METHOD: `GET` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:21:54.386471 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30a54798-2d30-476d-b55b-ddd45af625a7 admin admin] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:21:54.386765 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30a54798-2d30-476d-b55b-ddd45af625a7 admin admin] PATH_INFO: `/v3/roles` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:21:54.387911 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-30a54798-2d30-476d-b55b-ddd45af625a7 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:21:54.389607 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-30a54798-2d30-476d-b55b-ddd45af625a7 admin admin] RBAC: Authorization granted {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:21:54.401591 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 838/1644] 10.222.0.27 () {58 vars in 1122 bytes} [Thu Oct 13 04:21:54 2022] GET /identity/v3/roles => generated 1420 bytes in 37 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Oct 13 04:21:54.406862 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-30a54798-2d30-476d-b55b-ddd45af625a7 admin admin] Authenticating user token {{(pid=81409) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:21:54.409906 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.notifications [None req-31475085-bfe2-42f8-a623-93ee8336f4cb admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '987d5e2972cf4038a3ba585a4d4b6f07'} {{(pid=81408) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 13 04:21:54.417207 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.notifications [None req-31475085-bfe2-42f8-a623-93ee8336f4cb admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '987d5e2972cf4038a3ba585a4d4b6f07'} {{(pid=81408) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 13 04:21:54.419889 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.notifications [None req-31475085-bfe2-42f8-a623-93ee8336f4cb admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '987d5e2972cf4038a3ba585a4d4b6f07'} {{(pid=81408) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 13 04:21:54.420772 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.notifications [None req-31475085-bfe2-42f8-a623-93ee8336f4cb admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '987d5e2972cf4038a3ba585a4d4b6f07'} {{(pid=81408) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 13 04:21:54.422070 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-30a54798-2d30-476d-b55b-ddd45af625a7 admin admin] Validating token access rules against request {{(pid=81409) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:21:54.427026 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a23b1ef7-7989-42a5-b94e-bbe3eff09924 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8ba823f313144da3b683bacef8fd62e0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81409) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:21:54.427781 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a23b1ef7-7989-42a5-b94e-bbe3eff09924 admin admin] REQUEST_METHOD: `GET` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:21:54.428152 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a23b1ef7-7989-42a5-b94e-bbe3eff09924 admin admin] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:21:54.428475 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a23b1ef7-7989-42a5-b94e-bbe3eff09924 admin admin] PATH_INFO: `/v3/roles` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:21:54.429182 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a23b1ef7-7989-42a5-b94e-bbe3eff09924 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:21:54.431261 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a23b1ef7-7989-42a5-b94e-bbe3eff09924 admin admin] RBAC: Authorization granted {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:21:54.431520 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.notifications [None req-31475085-bfe2-42f8-a623-93ee8336f4cb admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '987d5e2972cf4038a3ba585a4d4b6f07'} {{(pid=81408) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 13 04:21:54.448969 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 839/1645] 10.222.0.27 () {58 vars in 1122 bytes} [Thu Oct 13 04:21:54 2022] GET /identity/v3/roles => generated 1420 bytes in 40 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Oct 13 04:21:54.450067 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a23b1ef7-7989-42a5-b94e-bbe3eff09924 admin admin] Authenticating user token {{(pid=81409) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:21:54.452108 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.fernet_utils [None req-a23b1ef7-7989-42a5-b94e-bbe3eff09924 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81409) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:21:54.457960 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 807/1646] 10.222.0.27 () {58 vars in 1224 bytes} [Thu Oct 13 04:21:54 2022] DELETE /identity/v3/users/987d5e2972cf4038a3ba585a4d4b6f07 => generated 0 bytes in 281 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 13 04:21:54.466717 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a23b1ef7-7989-42a5-b94e-bbe3eff09924 admin admin] Validating token access rules against request {{(pid=81409) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:21:54.557939 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-477070f2-4a32-450b-9247-3585f2349aea admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8ba823f313144da3b683bacef8fd62e0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81409) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:21:54.558681 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-477070f2-4a32-450b-9247-3585f2349aea admin admin] REQUEST_METHOD: `PUT` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:21:54.558963 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-477070f2-4a32-450b-9247-3585f2349aea admin admin] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:21:54.559527 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-477070f2-4a32-450b-9247-3585f2349aea admin admin] PATH_INFO: `/v3/projects/4aac93da506f49528e48ef1c1d2afbfd/users/9cd41672e51247f487f557b1bdebf6d1/roles/cc56c4ae13a7424ea771c15da4b153c1` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:21:54.593339 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-477070f2-4a32-450b-9247-3585f2349aea admin admin] RBAC: Authorizing `identity:create_grant(project_id=4aac93da506f49528e48ef1c1d2afbfd, user_id=9cd41672e51247f487f557b1bdebf6d1, role_id=cc56c4ae13a7424ea771c15da4b153c1)` {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:21:54.596382 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-477070f2-4a32-450b-9247-3585f2349aea admin admin] RBAC: Authorization granted {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:21:54.643349 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 840/1647] 10.222.0.27 () {60 vars in 1483 bytes} [Thu Oct 13 04:21:54 2022] PUT /identity/v3/projects/4aac93da506f49528e48ef1c1d2afbfd/users/9cd41672e51247f487f557b1bdebf6d1/roles/cc56c4ae13a7424ea771c15da4b153c1 => generated 0 bytes in 190 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 13 04:21:54.650481 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0ad6762-3f95-485c-8ada-bdf603d1e63b None None] REQUEST_METHOD: `POST` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:21:54.650777 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0ad6762-3f95-485c-8ada-bdf603d1e63b None None] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:21:54.651098 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0ad6762-3f95-485c-8ada-bdf603d1e63b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:21:54.714594 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.auth.core [None req-e0ad6762-3f95-485c-8ada-bdf603d1e63b None None] MFA Rules not processed for user `9cd41672e51247f487f557b1bdebf6d1`. Rule list: `[]` (Enabled: `True`). {{(pid=81408) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 13 04:21:54.716539 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.fernet_utils [None req-e0ad6762-3f95-485c-8ada-bdf603d1e63b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81408) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:21:54.738965 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG dbcounter [-] [81408] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81408) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:21:54.760239 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-477070f2-4a32-450b-9247-3585f2349aea admin admin] Authenticating user token {{(pid=81409) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:21:54.762056 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.fernet_utils [None req-477070f2-4a32-450b-9247-3585f2349aea admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81409) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:21:54.838641 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 808/1648] 10.222.0.27 () {58 vars in 958 bytes} [Thu Oct 13 04:21:54 2022] POST /identity/v3/auth/tokens => generated 3244 bytes in 189 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 13 04:21:54.847889 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e0ad6762-3f95-485c-8ada-bdf603d1e63b None None] Authenticating user token {{(pid=81408) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:21:54.849807 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.fernet_utils [None req-e0ad6762-3f95-485c-8ada-bdf603d1e63b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81408) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:21:54.862284 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-477070f2-4a32-450b-9247-3585f2349aea admin admin] Validating token access rules against request {{(pid=81409) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:21:54.868478 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-044d3ae8-afff-4990-8a68-4d716c09ff63 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8ba823f313144da3b683bacef8fd62e0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81409) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:21:54.869129 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-044d3ae8-afff-4990-8a68-4d716c09ff63 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:21:54.869408 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-044d3ae8-afff-4990-8a68-4d716c09ff63 admin admin] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:21:54.869683 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-044d3ae8-afff-4990-8a68-4d716c09ff63 admin admin] PATH_INFO: `/v3/users/c7d32e81e9ad4563a4d6b7f3ef8456df` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:21:54.870890 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-044d3ae8-afff-4990-8a68-4d716c09ff63 admin admin] RBAC: Authorizing `identity:delete_user(user_id=c7d32e81e9ad4563a4d6b7f3ef8456df)` {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:21:54.874229 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-044d3ae8-afff-4990-8a68-4d716c09ff63 admin admin] RBAC: Authorization granted {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:21:54.928422 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG dbcounter [-] [81409] Writing DB stats keystone:SELECT=91,keystone:INSERT=4,keystone:DELETE=5 {{(pid=81409) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:21:54.957334 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e0ad6762-3f95-485c-8ada-bdf603d1e63b None None] Validating token access rules against request {{(pid=81408) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:21:54.961666 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a2ad1802-b013-4acb-b0d6-4fde38d7eb1b service glance] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a6700f49b86543f6a5b92cad8faaba42', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dcec0431fb6849499d6aec35a234daa6', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81408) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:21:54.962898 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2ad1802-b013-4acb-b0d6-4fde38d7eb1b service glance] REQUEST_METHOD: `GET` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:21:54.963313 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2ad1802-b013-4acb-b0d6-4fde38d7eb1b service glance] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:21:54.963687 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2ad1802-b013-4acb-b0d6-4fde38d7eb1b service glance] PATH_INFO: `/v3/auth/tokens` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:21:54.984587 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.notifications [None req-044d3ae8-afff-4990-8a68-4d716c09ff63 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'c7d32e81e9ad4563a4d6b7f3ef8456df'} {{(pid=81409) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 13 04:21:54.986649 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.notifications [None req-044d3ae8-afff-4990-8a68-4d716c09ff63 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'c7d32e81e9ad4563a4d6b7f3ef8456df'} {{(pid=81409) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 13 04:21:54.990755 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a2ad1802-b013-4acb-b0d6-4fde38d7eb1b service glance] RBAC: Authorizing `identity:validate_token()` {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:21:54.994334 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a2ad1802-b013-4acb-b0d6-4fde38d7eb1b service glance] RBAC: Authorization granted {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:21:54.996950 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.notifications [None req-044d3ae8-afff-4990-8a68-4d716c09ff63 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'c7d32e81e9ad4563a4d6b7f3ef8456df'} {{(pid=81409) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 13 04:21:54.997986 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.notifications [None req-044d3ae8-afff-4990-8a68-4d716c09ff63 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'c7d32e81e9ad4563a4d6b7f3ef8456df'} {{(pid=81409) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 13 04:21:55.007248 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.notifications [None req-044d3ae8-afff-4990-8a68-4d716c09ff63 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'c7d32e81e9ad4563a4d6b7f3ef8456df'} {{(pid=81409) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 13 04:21:55.025614 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 809/1649] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Oct 13 04:21:54 2022] GET /identity/v3/auth/tokens => generated 3244 bytes in 178 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 13 04:21:55.027171 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a2ad1802-b013-4acb-b0d6-4fde38d7eb1b service glance] Authenticating user token {{(pid=81408) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:21:55.030835 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.fernet_utils [None req-a2ad1802-b013-4acb-b0d6-4fde38d7eb1b service glance] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81408) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:21:55.049072 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a2ad1802-b013-4acb-b0d6-4fde38d7eb1b service glance] Validating token access rules against request {{(pid=81408) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:21:55.053689 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 841/1650] 10.222.0.27 () {58 vars in 1224 bytes} [Thu Oct 13 04:21:54 2022] DELETE /identity/v3/users/c7d32e81e9ad4563a4d6b7f3ef8456df => generated 0 bytes in 294 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 13 04:21:55.058145 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0accb5fa-dfe6-4353-b960-7b9cf8f5ac55 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8ba823f313144da3b683bacef8fd62e0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81408) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:21:55.059719 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0accb5fa-dfe6-4353-b960-7b9cf8f5ac55 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:21:55.060331 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0accb5fa-dfe6-4353-b960-7b9cf8f5ac55 admin admin] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:21:55.060844 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0accb5fa-dfe6-4353-b960-7b9cf8f5ac55 admin admin] PATH_INFO: `/v3/projects/610bb38e1f5e4bd2ad2afe210df75757` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:21:55.062225 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0accb5fa-dfe6-4353-b960-7b9cf8f5ac55 admin admin] RBAC: Authorizing `identity:delete_project(project_id=610bb38e1f5e4bd2ad2afe210df75757)` {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:21:55.064363 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0accb5fa-dfe6-4353-b960-7b9cf8f5ac55 admin admin] RBAC: Authorization granted {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:21:55.067929 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e125bf5a-d418-4292-810a-db804a00f3f2 None None] REQUEST_METHOD: `POST` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:21:55.068421 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e125bf5a-d418-4292-810a-db804a00f3f2 None None] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:21:55.068874 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e125bf5a-d418-4292-810a-db804a00f3f2 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:21:55.097609 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.auth.core [None req-e125bf5a-d418-4292-810a-db804a00f3f2 None None] MFA Rules not processed for user `0e2d979563d6421995451a0e86bf3022`. Rule list: `[]` (Enabled: `True`). {{(pid=81409) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 13 04:21:55.098986 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.fernet_utils [None req-e125bf5a-d418-4292-810a-db804a00f3f2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81409) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:21:55.132550 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.notifications [None req-0accb5fa-dfe6-4353-b960-7b9cf8f5ac55 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '610bb38e1f5e4bd2ad2afe210df75757', 'request_id': 'req-0accb5fa-dfe6-4353-b960-7b9cf8f5ac55'} {{(pid=81408) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 13 04:21:55.148277 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.notifications [None req-0accb5fa-dfe6-4353-b960-7b9cf8f5ac55 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '610bb38e1f5e4bd2ad2afe210df75757', 'request_id': 'req-0accb5fa-dfe6-4353-b960-7b9cf8f5ac55'} {{(pid=81408) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 13 04:21:55.161832 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.notifications [None req-0accb5fa-dfe6-4353-b960-7b9cf8f5ac55 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '610bb38e1f5e4bd2ad2afe210df75757', 'request_id': 'req-0accb5fa-dfe6-4353-b960-7b9cf8f5ac55'} {{(pid=81408) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 13 04:21:55.176596 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.notifications [None req-0accb5fa-dfe6-4353-b960-7b9cf8f5ac55 admin admin] The token cache is being invalidate because project 610bb38e1f5e4bd2ad2afe210df75757 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81408) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 13 04:21:55.176732 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 842/1651] 10.222.0.27 () {58 vars in 958 bytes} [Thu Oct 13 04:21:55 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 116 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 13 04:21:55.176814 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.notifications [None req-0accb5fa-dfe6-4353-b960-7b9cf8f5ac55 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81408) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 13 04:21:55.176814 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.notifications [None req-0accb5fa-dfe6-4353-b960-7b9cf8f5ac55 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81408) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 13 04:21:55.178413 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 810/1652] 10.222.0.27 () {58 vars in 1233 bytes} [Thu Oct 13 04:21:55 2022] DELETE /identity/v3/projects/610bb38e1f5e4bd2ad2afe210df75757 => generated 0 bytes in 152 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 13 04:21:55.185737 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a437fbb-921f-4a5d-a612-cf90f9592eaa None None] REQUEST_METHOD: `POST` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:21:55.186094 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a437fbb-921f-4a5d-a612-cf90f9592eaa None None] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:21:55.186467 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a437fbb-921f-4a5d-a612-cf90f9592eaa None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:21:55.190748 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0accb5fa-dfe6-4353-b960-7b9cf8f5ac55 admin admin] Authenticating user token {{(pid=81408) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:21:55.192332 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.fernet_utils [None req-0accb5fa-dfe6-4353-b960-7b9cf8f5ac55 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81408) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:21:55.209526 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.auth.core [None req-2a437fbb-921f-4a5d-a612-cf90f9592eaa None None] MFA Rules not processed for user `0e2d979563d6421995451a0e86bf3022`. Rule list: `[]` (Enabled: `True`). {{(pid=81409) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 13 04:21:55.210920 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.fernet_utils [None req-2a437fbb-921f-4a5d-a612-cf90f9592eaa None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81409) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:21:55.220924 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG dbcounter [-] [81408] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=81408) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:21:55.283342 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0accb5fa-dfe6-4353-b960-7b9cf8f5ac55 admin admin] Validating token access rules against request {{(pid=81408) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:21:55.307364 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-11034aea-778d-48a8-bdb9-c4b4ac2577aa service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd8812c3974f84871bb2d761cf0016257', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dcec0431fb6849499d6aec35a234daa6', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81408) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:21:55.308250 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11034aea-778d-48a8-bdb9-c4b4ac2577aa service neutron] REQUEST_METHOD: `GET` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:21:55.309951 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11034aea-778d-48a8-bdb9-c4b4ac2577aa service neutron] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:21:55.310642 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11034aea-778d-48a8-bdb9-c4b4ac2577aa service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:21:55.313934 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.fernet_utils [None req-11034aea-778d-48a8-bdb9-c4b4ac2577aa service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81408) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:21:55.406815 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-11034aea-778d-48a8-bdb9-c4b4ac2577aa service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:21:55.408802 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-11034aea-778d-48a8-bdb9-c4b4ac2577aa service neutron] RBAC: Authorization granted {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:21:55.421892 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 811/1653] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Oct 13 04:21:55 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 232 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 13 04:21:55.434170 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 843/1654] 10.222.0.27 () {58 vars in 958 bytes} [Thu Oct 13 04:21:55 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 250 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 13 04:21:55.443977 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fdb3962f-eab0-4ccb-9885-9652a53759e8 None None] REQUEST_METHOD: `POST` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:21:55.444370 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fdb3962f-eab0-4ccb-9885-9652a53759e8 None None] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:21:55.444635 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fdb3962f-eab0-4ccb-9885-9652a53759e8 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:21:55.476188 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.auth.core [None req-fdb3962f-eab0-4ccb-9885-9652a53759e8 None None] MFA Rules not processed for user `0e2d979563d6421995451a0e86bf3022`. Rule list: `[]` (Enabled: `True`). {{(pid=81408) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 13 04:21:55.477837 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.fernet_utils [None req-fdb3962f-eab0-4ccb-9885-9652a53759e8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81408) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:21:55.489248 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 812/1655] 10.222.0.27 () {58 vars in 958 bytes} [Thu Oct 13 04:21:55 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 47 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 13 04:21:55.494236 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2a437fbb-921f-4a5d-a612-cf90f9592eaa None None] Authenticating user token {{(pid=81409) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:21:55.508314 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2a437fbb-921f-4a5d-a612-cf90f9592eaa None None] Validating token access rules against request {{(pid=81409) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:21:55.514013 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dbea9d4a-290c-4443-8cc6-4b87210857f2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8ba823f313144da3b683bacef8fd62e0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81409) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:21:55.514666 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dbea9d4a-290c-4443-8cc6-4b87210857f2 admin admin] REQUEST_METHOD: `GET` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:21:55.514938 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dbea9d4a-290c-4443-8cc6-4b87210857f2 admin admin] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:21:55.515214 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dbea9d4a-290c-4443-8cc6-4b87210857f2 admin admin] PATH_INFO: `/v3/domains` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:21:55.516003 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dbea9d4a-290c-4443-8cc6-4b87210857f2 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81409) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 13 04:21:55.516310 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dbea9d4a-290c-4443-8cc6-4b87210857f2 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:21:55.518075 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dbea9d4a-290c-4443-8cc6-4b87210857f2 admin admin] RBAC: Authorization granted {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:21:55.538940 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 844/1656] 10.222.0.27 () {58 vars in 1153 bytes} [Thu Oct 13 04:21:55 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 45 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 13 04:21:55.544705 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fdb3962f-eab0-4ccb-9885-9652a53759e8 None None] Authenticating user token {{(pid=81408) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:21:55.556631 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fdb3962f-eab0-4ccb-9885-9652a53759e8 None None] Validating token access rules against request {{(pid=81408) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:21:55.564455 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-667c90c7-4178-4eda-ae95-4dac9d643f14 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8ba823f313144da3b683bacef8fd62e0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81408) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:21:55.565134 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-667c90c7-4178-4eda-ae95-4dac9d643f14 admin admin] REQUEST_METHOD: `POST` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:21:55.565414 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-667c90c7-4178-4eda-ae95-4dac9d643f14 admin admin] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:21:55.565738 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-667c90c7-4178-4eda-ae95-4dac9d643f14 admin admin] PATH_INFO: `/v3/projects` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:21:55.566720 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-667c90c7-4178-4eda-ae95-4dac9d643f14 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:21:55.568473 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-667c90c7-4178-4eda-ae95-4dac9d643f14 admin admin] RBAC: Authorization granted {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:21:55.618277 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 813/1657] 10.222.0.27 () {60 vars in 1153 bytes} [Thu Oct 13 04:21:55 2022] POST /identity/v3/projects => generated 375 bytes in 73 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 13 04:21:55.620170 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dbea9d4a-290c-4443-8cc6-4b87210857f2 admin admin] Authenticating user token {{(pid=81409) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:21:55.639537 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dbea9d4a-290c-4443-8cc6-4b87210857f2 admin admin] Validating token access rules against request {{(pid=81409) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:21:55.678977 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG dbcounter [-] [81409] Writing DB stats keystone:SELECT=94,keystone:DELETE=5,keystone:INSERT=1 {{(pid=81409) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:21:55.729305 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e531909b-cc68-4d5f-916c-8c5f0617469b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8ba823f313144da3b683bacef8fd62e0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81409) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:21:55.729946 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e531909b-cc68-4d5f-916c-8c5f0617469b admin admin] REQUEST_METHOD: `POST` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:21:55.730298 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e531909b-cc68-4d5f-916c-8c5f0617469b admin admin] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:21:55.730536 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e531909b-cc68-4d5f-916c-8c5f0617469b admin admin] PATH_INFO: `/v3/users` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:21:55.731830 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e531909b-cc68-4d5f-916c-8c5f0617469b admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:21:55.734128 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e531909b-cc68-4d5f-916c-8c5f0617469b admin admin] RBAC: Authorization granted {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:21:55.766153 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 845/1658] 10.222.0.27 () {60 vars in 1144 bytes} [Thu Oct 13 04:21:55 2022] POST /identity/v3/users => generated 288 bytes in 147 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 13 04:21:55.770423 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-667c90c7-4178-4eda-ae95-4dac9d643f14 admin admin] Authenticating user token {{(pid=81408) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:21:55.791493 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-667c90c7-4178-4eda-ae95-4dac9d643f14 admin admin] Validating token access rules against request {{(pid=81408) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:21:55.804971 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ee680cdc-529b-4563-9d9e-bdafe8e76dcd admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8ba823f313144da3b683bacef8fd62e0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81408) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:21:55.805603 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee680cdc-529b-4563-9d9e-bdafe8e76dcd admin admin] REQUEST_METHOD: `GET` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:21:55.805878 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee680cdc-529b-4563-9d9e-bdafe8e76dcd admin admin] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:21:55.811351 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee680cdc-529b-4563-9d9e-bdafe8e76dcd admin admin] PATH_INFO: `/v3/roles` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:21:55.813581 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ee680cdc-529b-4563-9d9e-bdafe8e76dcd admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:21:55.814237 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ee680cdc-529b-4563-9d9e-bdafe8e76dcd admin admin] RBAC: Authorization granted {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:21:55.826365 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 814/1659] 10.222.0.27 () {58 vars in 1122 bytes} [Thu Oct 13 04:21:55 2022] GET /identity/v3/roles => generated 1420 bytes in 57 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 13 04:21:55.830767 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e531909b-cc68-4d5f-916c-8c5f0617469b admin admin] Authenticating user token {{(pid=81409) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:21:55.849668 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e531909b-cc68-4d5f-916c-8c5f0617469b admin admin] Validating token access rules against request {{(pid=81409) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:21:55.854926 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8c6a9538-7725-45bf-b349-c6e2d795615b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8ba823f313144da3b683bacef8fd62e0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81409) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:21:55.855663 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c6a9538-7725-45bf-b349-c6e2d795615b admin admin] REQUEST_METHOD: `GET` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:21:55.855967 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c6a9538-7725-45bf-b349-c6e2d795615b admin admin] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:21:55.856736 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c6a9538-7725-45bf-b349-c6e2d795615b admin admin] PATH_INFO: `/v3/roles` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:21:55.857705 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8c6a9538-7725-45bf-b349-c6e2d795615b admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:21:55.859342 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8c6a9538-7725-45bf-b349-c6e2d795615b admin admin] RBAC: Authorization granted {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:21:55.862064 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ee680cdc-529b-4563-9d9e-bdafe8e76dcd admin admin] Authenticating user token {{(pid=81408) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:21:55.864283 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.fernet_utils [None req-ee680cdc-529b-4563-9d9e-bdafe8e76dcd admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81408) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:21:55.873983 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 846/1660] 10.222.0.27 () {58 vars in 1122 bytes} [Thu Oct 13 04:21:55 2022] GET /identity/v3/roles => generated 1420 bytes in 40 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 13 04:21:55.875041 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8c6a9538-7725-45bf-b349-c6e2d795615b admin admin] Authenticating user token {{(pid=81409) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:21:55.887967 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8c6a9538-7725-45bf-b349-c6e2d795615b admin admin] Validating token access rules against request {{(pid=81409) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:21:55.894974 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ee680cdc-529b-4563-9d9e-bdafe8e76dcd admin admin] Validating token access rules against request {{(pid=81408) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:21:55.900961 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0029e0ca-4757-46aa-994d-afc8985c0360 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8ba823f313144da3b683bacef8fd62e0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81409) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:21:55.902471 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0029e0ca-4757-46aa-994d-afc8985c0360 admin admin] REQUEST_METHOD: `PUT` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:21:55.902597 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0029e0ca-4757-46aa-994d-afc8985c0360 admin admin] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:21:55.903610 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0029e0ca-4757-46aa-994d-afc8985c0360 admin admin] PATH_INFO: `/v3/projects/19dae11d071942fd8e62c91a1c7716ae/users/6d6b64e609474033ae6b1cce827e277e/roles/cc56c4ae13a7424ea771c15da4b153c1` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:21:55.921618 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5251450a-3a11-4e95-bc43-0a32917a5dde admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8ba823f313144da3b683bacef8fd62e0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81408) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:21:55.923322 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5251450a-3a11-4e95-bc43-0a32917a5dde admin admin] REQUEST_METHOD: `DELETE` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:21:55.923600 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5251450a-3a11-4e95-bc43-0a32917a5dde admin admin] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:21:55.923927 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5251450a-3a11-4e95-bc43-0a32917a5dde admin admin] PATH_INFO: `/v3/projects/8d029efee04a45549aa744c9c7e09acf` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:21:55.926342 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5251450a-3a11-4e95-bc43-0a32917a5dde admin admin] RBAC: Authorizing `identity:delete_project(project_id=8d029efee04a45549aa744c9c7e09acf)` {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:21:55.934664 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5251450a-3a11-4e95-bc43-0a32917a5dde admin admin] RBAC: Authorization granted {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:21:55.951984 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0029e0ca-4757-46aa-994d-afc8985c0360 admin admin] RBAC: Authorizing `identity:create_grant(project_id=19dae11d071942fd8e62c91a1c7716ae, user_id=6d6b64e609474033ae6b1cce827e277e, role_id=cc56c4ae13a7424ea771c15da4b153c1)` {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:21:55.954876 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0029e0ca-4757-46aa-994d-afc8985c0360 admin admin] RBAC: Authorization granted {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:21:55.975622 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 847/1661] 10.222.0.27 () {60 vars in 1483 bytes} [Thu Oct 13 04:21:55 2022] PUT /identity/v3/projects/19dae11d071942fd8e62c91a1c7716ae/users/6d6b64e609474033ae6b1cce827e277e/roles/cc56c4ae13a7424ea771c15da4b153c1 => generated 0 bytes in 99 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 13 04:21:55.987119 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9c9e901-3063-43a8-aa88-3049b16db69b None None] REQUEST_METHOD: `POST` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:21:55.987419 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9c9e901-3063-43a8-aa88-3049b16db69b None None] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:21:55.987808 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9c9e901-3063-43a8-aa88-3049b16db69b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:21:56.035815 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.auth.core [None req-f9c9e901-3063-43a8-aa88-3049b16db69b None None] MFA Rules not processed for user `6d6b64e609474033ae6b1cce827e277e`. Rule list: `[]` (Enabled: `True`). {{(pid=81409) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 13 04:21:56.038126 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.fernet_utils [None req-f9c9e901-3063-43a8-aa88-3049b16db69b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81409) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:21:56.059406 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.notifications [None req-5251450a-3a11-4e95-bc43-0a32917a5dde admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '8d029efee04a45549aa744c9c7e09acf', 'request_id': 'req-5251450a-3a11-4e95-bc43-0a32917a5dde'} {{(pid=81408) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 13 04:21:56.072431 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.notifications [None req-5251450a-3a11-4e95-bc43-0a32917a5dde admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '8d029efee04a45549aa744c9c7e09acf', 'request_id': 'req-5251450a-3a11-4e95-bc43-0a32917a5dde'} {{(pid=81408) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 13 04:21:56.087663 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.notifications [None req-5251450a-3a11-4e95-bc43-0a32917a5dde admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '8d029efee04a45549aa744c9c7e09acf', 'request_id': 'req-5251450a-3a11-4e95-bc43-0a32917a5dde'} {{(pid=81408) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 13 04:21:56.103718 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.notifications [None req-5251450a-3a11-4e95-bc43-0a32917a5dde admin admin] The token cache is being invalidate because project 8d029efee04a45549aa744c9c7e09acf was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81408) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 13 04:21:56.104071 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.notifications [None req-5251450a-3a11-4e95-bc43-0a32917a5dde admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81408) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 13 04:21:56.105568 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.notifications [None req-5251450a-3a11-4e95-bc43-0a32917a5dde admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81408) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 13 04:21:56.112435 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 815/1662] 10.222.0.27 () {58 vars in 1233 bytes} [Thu Oct 13 04:21:55 2022] DELETE /identity/v3/projects/8d029efee04a45549aa744c9c7e09acf => generated 0 bytes in 248 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 13 04:21:56.114223 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5251450a-3a11-4e95-bc43-0a32917a5dde admin admin] Authenticating user token {{(pid=81408) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:21:56.119329 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.fernet_utils [None req-5251450a-3a11-4e95-bc43-0a32917a5dde admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81408) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:21:56.132531 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 848/1663] 10.222.0.27 () {58 vars in 958 bytes} [Thu Oct 13 04:21:55 2022] POST /identity/v3/auth/tokens => generated 3252 bytes in 147 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 13 04:21:56.133625 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f9c9e901-3063-43a8-aa88-3049b16db69b None None] Authenticating user token {{(pid=81409) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:21:56.147653 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.fernet_utils [None req-f9c9e901-3063-43a8-aa88-3049b16db69b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81409) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:21:56.153672 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG dbcounter [-] [81408] Writing DB stats keystone:SELECT=91,keystone:INSERT=2,keystone:DELETE=7 {{(pid=81408) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:21:56.251776 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5251450a-3a11-4e95-bc43-0a32917a5dde admin admin] Validating token access rules against request {{(pid=81408) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:21:56.260018 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a0a2e922-855d-4804-8466-c1ec2881f1bc admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8ba823f313144da3b683bacef8fd62e0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81408) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:21:56.261440 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0a2e922-855d-4804-8466-c1ec2881f1bc admin admin] REQUEST_METHOD: `DELETE` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:21:56.261728 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0a2e922-855d-4804-8466-c1ec2881f1bc admin admin] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:21:56.262005 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0a2e922-855d-4804-8466-c1ec2881f1bc admin admin] PATH_INFO: `/v3/users/dd72d097cd8642b48de6e3bc2753d2f9` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:21:56.264621 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a0a2e922-855d-4804-8466-c1ec2881f1bc admin admin] RBAC: Authorizing `identity:delete_user(user_id=dd72d097cd8642b48de6e3bc2753d2f9)` {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:21:56.268428 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a0a2e922-855d-4804-8466-c1ec2881f1bc admin admin] RBAC: Authorization granted {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:21:56.346622 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f9c9e901-3063-43a8-aa88-3049b16db69b None None] Validating token access rules against request {{(pid=81409) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:21:56.363181 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3f091299-9c5e-43f6-8a2a-ccf5bfc8bc2b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8ba823f313144da3b683bacef8fd62e0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81409) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:21:56.363566 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f091299-9c5e-43f6-8a2a-ccf5bfc8bc2b admin admin] REQUEST_METHOD: `POST` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:21:56.363848 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f091299-9c5e-43f6-8a2a-ccf5bfc8bc2b admin admin] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:21:56.364430 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f091299-9c5e-43f6-8a2a-ccf5bfc8bc2b admin admin] PATH_INFO: `/v3/projects` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:21:56.365300 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3f091299-9c5e-43f6-8a2a-ccf5bfc8bc2b admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:21:56.367565 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3f091299-9c5e-43f6-8a2a-ccf5bfc8bc2b admin admin] RBAC: Authorization granted {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:21:56.377694 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.notifications [None req-a0a2e922-855d-4804-8466-c1ec2881f1bc admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'dd72d097cd8642b48de6e3bc2753d2f9'} {{(pid=81408) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 13 04:21:56.407495 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.notifications [None req-a0a2e922-855d-4804-8466-c1ec2881f1bc admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'dd72d097cd8642b48de6e3bc2753d2f9'} {{(pid=81408) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 13 04:21:56.407495 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.notifications [None req-a0a2e922-855d-4804-8466-c1ec2881f1bc admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'dd72d097cd8642b48de6e3bc2753d2f9'} {{(pid=81408) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 13 04:21:56.407495 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.notifications [None req-a0a2e922-855d-4804-8466-c1ec2881f1bc admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'dd72d097cd8642b48de6e3bc2753d2f9'} {{(pid=81408) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 13 04:21:56.425538 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.notifications [None req-a0a2e922-855d-4804-8466-c1ec2881f1bc admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'dd72d097cd8642b48de6e3bc2753d2f9'} {{(pid=81408) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 13 04:21:56.455735 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 816/1664] 10.222.0.27 () {58 vars in 1224 bytes} [Thu Oct 13 04:21:56 2022] DELETE /identity/v3/users/dd72d097cd8642b48de6e3bc2753d2f9 => generated 0 bytes in 339 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 13 04:21:56.467995 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 849/1665] 10.222.0.27 () {60 vars in 1153 bytes} [Thu Oct 13 04:21:56 2022] POST /identity/v3/projects => generated 373 bytes in 335 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 13 04:21:56.472469 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a0a2e922-855d-4804-8466-c1ec2881f1bc admin admin] Authenticating user token {{(pid=81408) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:21:56.474084 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.fernet_utils [None req-a0a2e922-855d-4804-8466-c1ec2881f1bc admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81408) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:21:56.554711 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a0a2e922-855d-4804-8466-c1ec2881f1bc admin admin] Validating token access rules against request {{(pid=81408) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:21:56.604724 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG dbcounter [-] [81408] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81408) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:21:56.649389 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6b807200-a15c-4617-b5d3-8d4b82d11395 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8ba823f313144da3b683bacef8fd62e0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81408) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:21:56.652004 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b807200-a15c-4617-b5d3-8d4b82d11395 admin admin] REQUEST_METHOD: `POST` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:21:56.652288 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b807200-a15c-4617-b5d3-8d4b82d11395 admin admin] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:21:56.652911 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b807200-a15c-4617-b5d3-8d4b82d11395 admin admin] PATH_INFO: `/v3/users` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:21:56.653822 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6b807200-a15c-4617-b5d3-8d4b82d11395 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:21:56.660590 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6b807200-a15c-4617-b5d3-8d4b82d11395 admin admin] RBAC: Authorization granted {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:21:56.706406 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 817/1666] 10.222.0.27 () {60 vars in 1144 bytes} [Thu Oct 13 04:21:56 2022] POST /identity/v3/users => generated 293 bytes in 235 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 13 04:21:56.710289 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3f091299-9c5e-43f6-8a2a-ccf5bfc8bc2b admin admin] Authenticating user token {{(pid=81409) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:21:56.717238 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG dbcounter [-] [81409] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=81409) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:21:56.726528 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3f091299-9c5e-43f6-8a2a-ccf5bfc8bc2b admin admin] Validating token access rules against request {{(pid=81409) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:21:56.737552 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-18ce91c4-47f6-4e79-a50d-33c981170cda admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8ba823f313144da3b683bacef8fd62e0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81409) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:21:56.738514 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18ce91c4-47f6-4e79-a50d-33c981170cda admin admin] REQUEST_METHOD: `GET` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:21:56.739091 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18ce91c4-47f6-4e79-a50d-33c981170cda admin admin] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:21:56.739899 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18ce91c4-47f6-4e79-a50d-33c981170cda admin admin] PATH_INFO: `/v3/roles` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:21:56.740575 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-18ce91c4-47f6-4e79-a50d-33c981170cda admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:21:56.748347 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-18ce91c4-47f6-4e79-a50d-33c981170cda admin admin] RBAC: Authorization granted {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:21:56.781312 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 850/1667] 10.222.0.27 () {58 vars in 1122 bytes} [Thu Oct 13 04:21:56 2022] GET /identity/v3/roles => generated 1420 bytes in 68 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 13 04:21:56.783122 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6b807200-a15c-4617-b5d3-8d4b82d11395 admin admin] Authenticating user token {{(pid=81408) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:21:56.801666 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6b807200-a15c-4617-b5d3-8d4b82d11395 admin admin] Validating token access rules against request {{(pid=81408) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:21:56.813943 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e02c034e-24fb-4ee9-abba-4e0f1094fb57 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8ba823f313144da3b683bacef8fd62e0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81408) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:21:56.815733 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e02c034e-24fb-4ee9-abba-4e0f1094fb57 admin admin] REQUEST_METHOD: `PUT` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:21:56.816406 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e02c034e-24fb-4ee9-abba-4e0f1094fb57 admin admin] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:21:56.816888 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e02c034e-24fb-4ee9-abba-4e0f1094fb57 admin admin] PATH_INFO: `/v3/projects/06441ef140474903a394a59c0c7695f2/users/9ed11ff4bed24e0696d6b8e7c0076d55/roles/ef55fbd94e1040be83695754075a7ed5` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:21:56.862183 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e02c034e-24fb-4ee9-abba-4e0f1094fb57 admin admin] RBAC: Authorizing `identity:create_grant(project_id=06441ef140474903a394a59c0c7695f2, user_id=9ed11ff4bed24e0696d6b8e7c0076d55, role_id=ef55fbd94e1040be83695754075a7ed5)` {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:21:56.870571 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e02c034e-24fb-4ee9-abba-4e0f1094fb57 admin admin] RBAC: Authorization granted {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:21:56.900621 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 818/1668] 10.222.0.27 () {60 vars in 1483 bytes} [Thu Oct 13 04:21:56 2022] PUT /identity/v3/projects/06441ef140474903a394a59c0c7695f2/users/9ed11ff4bed24e0696d6b8e7c0076d55/roles/ef55fbd94e1040be83695754075a7ed5 => generated 0 bytes in 115 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 13 04:21:56.903159 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-18ce91c4-47f6-4e79-a50d-33c981170cda admin admin] Authenticating user token {{(pid=81409) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:21:56.917988 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-18ce91c4-47f6-4e79-a50d-33c981170cda admin admin] Validating token access rules against request {{(pid=81409) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:21:56.927619 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e02c034e-24fb-4ee9-abba-4e0f1094fb57 admin admin] Authenticating user token {{(pid=81408) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:21:56.929830 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.fernet_utils [None req-e02c034e-24fb-4ee9-abba-4e0f1094fb57 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81408) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:21:57.092229 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e02c034e-24fb-4ee9-abba-4e0f1094fb57 admin admin] Validating token access rules against request {{(pid=81408) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:21:57.098985 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-516ef71c-470c-40bf-a724-b0de06f85a85 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8ba823f313144da3b683bacef8fd62e0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81409) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:21:57.099629 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-516ef71c-470c-40bf-a724-b0de06f85a85 admin admin] REQUEST_METHOD: `GET` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:21:57.099995 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-516ef71c-470c-40bf-a724-b0de06f85a85 admin admin] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:21:57.100231 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-516ef71c-470c-40bf-a724-b0de06f85a85 admin admin] PATH_INFO: `/v3/roles` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:21:57.100897 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-516ef71c-470c-40bf-a724-b0de06f85a85 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:21:57.101162 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1413649c-7c3e-406f-90a9-3fab4cde7ff7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8ba823f313144da3b683bacef8fd62e0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81408) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:21:57.101872 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1413649c-7c3e-406f-90a9-3fab4cde7ff7 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:21:57.102152 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1413649c-7c3e-406f-90a9-3fab4cde7ff7 admin admin] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:21:57.102446 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1413649c-7c3e-406f-90a9-3fab4cde7ff7 admin admin] PATH_INFO: `/v3/projects/28438e126c82422882635ce40802a418` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:21:57.102621 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-516ef71c-470c-40bf-a724-b0de06f85a85 admin admin] RBAC: Authorization granted {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:21:57.103645 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1413649c-7c3e-406f-90a9-3fab4cde7ff7 admin admin] RBAC: Authorizing `identity:delete_project(project_id=28438e126c82422882635ce40802a418)` {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:21:57.105600 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1413649c-7c3e-406f-90a9-3fab4cde7ff7 admin admin] RBAC: Authorization granted {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:21:57.128477 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 851/1669] 10.222.0.27 () {58 vars in 1122 bytes} [Thu Oct 13 04:21:56 2022] GET /identity/v3/roles => generated 1420 bytes in 224 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 13 04:21:57.130387 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-516ef71c-470c-40bf-a724-b0de06f85a85 admin admin] Authenticating user token {{(pid=81409) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:21:57.146533 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-516ef71c-470c-40bf-a724-b0de06f85a85 admin admin] Validating token access rules against request {{(pid=81409) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:21:57.160289 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ef76e963-7cb2-4183-810c-41d01f32f95f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8ba823f313144da3b683bacef8fd62e0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81409) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:21:57.161129 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef76e963-7cb2-4183-810c-41d01f32f95f admin admin] REQUEST_METHOD: `PUT` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:21:57.161485 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef76e963-7cb2-4183-810c-41d01f32f95f admin admin] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:21:57.161830 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef76e963-7cb2-4183-810c-41d01f32f95f admin admin] PATH_INFO: `/v3/projects/06441ef140474903a394a59c0c7695f2/users/9ed11ff4bed24e0696d6b8e7c0076d55/roles/ef55fbd94e1040be83695754075a7ed5` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:21:57.165887 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ef76e963-7cb2-4183-810c-41d01f32f95f admin admin] RBAC: Authorizing `identity:create_grant(project_id=06441ef140474903a394a59c0c7695f2, user_id=9ed11ff4bed24e0696d6b8e7c0076d55, role_id=ef55fbd94e1040be83695754075a7ed5)` {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:21:57.171550 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ef76e963-7cb2-4183-810c-41d01f32f95f admin admin] RBAC: Authorization granted {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:21:57.174273 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.notifications [None req-1413649c-7c3e-406f-90a9-3fab4cde7ff7 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '28438e126c82422882635ce40802a418', 'request_id': 'req-1413649c-7c3e-406f-90a9-3fab4cde7ff7'} {{(pid=81408) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 13 04:21:57.184010 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.notifications [None req-1413649c-7c3e-406f-90a9-3fab4cde7ff7 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '28438e126c82422882635ce40802a418', 'request_id': 'req-1413649c-7c3e-406f-90a9-3fab4cde7ff7'} {{(pid=81408) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 13 04:21:57.193772 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 852/1670] 10.222.0.27 () {60 vars in 1483 bytes} [Thu Oct 13 04:21:57 2022] PUT /identity/v3/projects/06441ef140474903a394a59c0c7695f2/users/9ed11ff4bed24e0696d6b8e7c0076d55/roles/ef55fbd94e1040be83695754075a7ed5 => generated 0 bytes in 64 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 13 04:21:57.205621 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0147b249-b41b-429c-8e27-8f06f8fcb4a2 None None] REQUEST_METHOD: `POST` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:21:57.205797 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.notifications [None req-1413649c-7c3e-406f-90a9-3fab4cde7ff7 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '28438e126c82422882635ce40802a418', 'request_id': 'req-1413649c-7c3e-406f-90a9-3fab4cde7ff7'} {{(pid=81408) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 13 04:21:57.205891 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0147b249-b41b-429c-8e27-8f06f8fcb4a2 None None] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:21:57.206112 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0147b249-b41b-429c-8e27-8f06f8fcb4a2 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:21:57.218423 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.notifications [None req-1413649c-7c3e-406f-90a9-3fab4cde7ff7 admin admin] The token cache is being invalidate because project 28438e126c82422882635ce40802a418 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81408) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 13 04:21:57.218806 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.notifications [None req-1413649c-7c3e-406f-90a9-3fab4cde7ff7 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81408) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 13 04:21:57.219539 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.notifications [None req-1413649c-7c3e-406f-90a9-3fab4cde7ff7 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81408) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 13 04:21:57.221078 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 819/1671] 10.222.0.27 () {58 vars in 1233 bytes} [Thu Oct 13 04:21:56 2022] DELETE /identity/v3/projects/28438e126c82422882635ce40802a418 => generated 0 bytes in 294 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 13 04:21:57.231739 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42af1f4d-fca9-4286-b1f2-f4757b0b7c26 None None] REQUEST_METHOD: `POST` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:21:57.231992 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42af1f4d-fca9-4286-b1f2-f4757b0b7c26 None None] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:21:57.232387 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42af1f4d-fca9-4286-b1f2-f4757b0b7c26 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:21:57.254460 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.auth.core [None req-0147b249-b41b-429c-8e27-8f06f8fcb4a2 None None] MFA Rules not processed for user `9ed11ff4bed24e0696d6b8e7c0076d55`. Rule list: `[]` (Enabled: `True`). {{(pid=81409) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 13 04:21:57.255764 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.fernet_utils [None req-0147b249-b41b-429c-8e27-8f06f8fcb4a2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81409) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:21:57.256171 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.auth.core [None req-42af1f4d-fca9-4286-b1f2-f4757b0b7c26 None None] MFA Rules not processed for user `0e2d979563d6421995451a0e86bf3022`. Rule list: `[]` (Enabled: `True`). {{(pid=81408) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 13 04:21:57.257601 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.fernet_utils [None req-42af1f4d-fca9-4286-b1f2-f4757b0b7c26 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81408) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:21:57.293109 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG dbcounter [-] [81408] Writing DB stats keystone:SELECT=88,keystone:INSERT=5,keystone:DELETE=7 {{(pid=81408) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:21:57.345294 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 820/1672] 10.222.0.27 () {58 vars in 958 bytes} [Thu Oct 13 04:21:57 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 120 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 13 04:21:57.346952 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83192070-ae77-4cf0-949d-c528538d34b5 None None] REQUEST_METHOD: `POST` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:21:57.347237 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83192070-ae77-4cf0-949d-c528538d34b5 None None] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:21:57.347522 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83192070-ae77-4cf0-949d-c528538d34b5 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:21:57.374633 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.auth.core [None req-83192070-ae77-4cf0-949d-c528538d34b5 None None] MFA Rules not processed for user `0e2d979563d6421995451a0e86bf3022`. Rule list: `[]` (Enabled: `True`). {{(pid=81408) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 13 04:21:57.375871 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.fernet_utils [None req-83192070-ae77-4cf0-949d-c528538d34b5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81408) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:21:57.384276 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 821/1673] 10.222.0.27 () {58 vars in 958 bytes} [Thu Oct 13 04:21:57 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 38 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 13 04:21:57.388847 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-83192070-ae77-4cf0-949d-c528538d34b5 None None] Authenticating user token {{(pid=81408) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:21:57.395062 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 853/1674] 10.222.0.27 () {58 vars in 958 bytes} [Thu Oct 13 04:21:57 2022] POST /identity/v3/auth/tokens => generated 3317 bytes in 190 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 13 04:21:57.403496 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0147b249-b41b-429c-8e27-8f06f8fcb4a2 None None] Authenticating user token {{(pid=81409) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:21:57.406133 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.fernet_utils [None req-0147b249-b41b-429c-8e27-8f06f8fcb4a2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81409) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:21:57.410930 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-83192070-ae77-4cf0-949d-c528538d34b5 None None] Validating token access rules against request {{(pid=81408) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:21:57.435067 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6c04f88c-3ba6-4896-bd74-e2426a6b796f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8ba823f313144da3b683bacef8fd62e0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81408) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:21:57.435735 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c04f88c-3ba6-4896-bd74-e2426a6b796f admin admin] REQUEST_METHOD: `GET` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:21:57.436017 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c04f88c-3ba6-4896-bd74-e2426a6b796f admin admin] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:21:57.436324 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c04f88c-3ba6-4896-bd74-e2426a6b796f admin admin] PATH_INFO: `/v3/domains` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:21:57.436987 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6c04f88c-3ba6-4896-bd74-e2426a6b796f admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81408) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 13 04:21:57.437492 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6c04f88c-3ba6-4896-bd74-e2426a6b796f admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:21:57.439234 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6c04f88c-3ba6-4896-bd74-e2426a6b796f admin admin] RBAC: Authorization granted {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:21:57.481876 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 822/1675] 10.222.0.27 () {58 vars in 1153 bytes} [Thu Oct 13 04:21:57 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 93 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 13 04:21:57.484053 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6c04f88c-3ba6-4896-bd74-e2426a6b796f admin admin] Authenticating user token {{(pid=81408) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:21:57.495466 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6c04f88c-3ba6-4896-bd74-e2426a6b796f admin admin] Validating token access rules against request {{(pid=81408) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:21:57.507099 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c2335db4-a177-47a8-a57f-8fceb79ac5e8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8ba823f313144da3b683bacef8fd62e0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81408) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:21:57.511351 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2335db4-a177-47a8-a57f-8fceb79ac5e8 admin admin] REQUEST_METHOD: `POST` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:21:57.512029 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2335db4-a177-47a8-a57f-8fceb79ac5e8 admin admin] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:21:57.513241 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2335db4-a177-47a8-a57f-8fceb79ac5e8 admin admin] PATH_INFO: `/v3/projects` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:21:57.514272 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c2335db4-a177-47a8-a57f-8fceb79ac5e8 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:21:57.517981 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c2335db4-a177-47a8-a57f-8fceb79ac5e8 admin admin] RBAC: Authorization granted {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:21:57.553543 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0147b249-b41b-429c-8e27-8f06f8fcb4a2 None None] Validating token access rules against request {{(pid=81409) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:21:57.554790 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 823/1676] 10.222.0.27 () {60 vars in 1153 bytes} [Thu Oct 13 04:21:57 2022] POST /identity/v3/projects => generated 379 bytes in 72 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 13 04:21:57.560564 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c2335db4-a177-47a8-a57f-8fceb79ac5e8 admin admin] Authenticating user token {{(pid=81408) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:21:57.573111 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG dbcounter [-] [81409] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81409) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:21:57.651813 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-69c6c0f8-69b7-47d1-85f0-5ac08da21314 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6141b8a6c3d54e179d6879bb019f4d12', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dcec0431fb6849499d6aec35a234daa6', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81409) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:21:57.653297 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69c6c0f8-69b7-47d1-85f0-5ac08da21314 service cinder] REQUEST_METHOD: `GET` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:21:57.653672 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69c6c0f8-69b7-47d1-85f0-5ac08da21314 service cinder] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:21:57.654057 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69c6c0f8-69b7-47d1-85f0-5ac08da21314 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:21:57.657471 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.fernet_utils [None req-69c6c0f8-69b7-47d1-85f0-5ac08da21314 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81409) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:21:57.730107 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c2335db4-a177-47a8-a57f-8fceb79ac5e8 admin admin] Validating token access rules against request {{(pid=81408) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:21:57.737898 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c7e01531-ffb7-4b17-b90c-d29e1739f348 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8ba823f313144da3b683bacef8fd62e0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81408) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:21:57.739578 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7e01531-ffb7-4b17-b90c-d29e1739f348 admin admin] REQUEST_METHOD: `POST` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:21:57.739870 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7e01531-ffb7-4b17-b90c-d29e1739f348 admin admin] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:21:57.740150 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7e01531-ffb7-4b17-b90c-d29e1739f348 admin admin] PATH_INFO: `/v3/users` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:21:57.741083 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c7e01531-ffb7-4b17-b90c-d29e1739f348 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:21:57.744193 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c7e01531-ffb7-4b17-b90c-d29e1739f348 admin admin] RBAC: Authorization granted {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:21:57.755466 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-69c6c0f8-69b7-47d1-85f0-5ac08da21314 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:21:57.759746 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-69c6c0f8-69b7-47d1-85f0-5ac08da21314 service cinder] RBAC: Authorization granted {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:21:57.793198 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 854/1677] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Oct 13 04:21:57 2022] GET /identity/v3/auth/tokens => generated 3252 bytes in 389 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 13 04:21:57.795360 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-69c6c0f8-69b7-47d1-85f0-5ac08da21314 service cinder] Authenticating user token {{(pid=81409) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:21:57.796994 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 824/1678] 10.222.0.27 () {60 vars in 1144 bytes} [Thu Oct 13 04:21:57 2022] POST /identity/v3/users => generated 290 bytes in 238 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 13 04:21:57.811431 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-69c6c0f8-69b7-47d1-85f0-5ac08da21314 service cinder] Validating token access rules against request {{(pid=81409) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:21:57.822027 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-329f1a67-d1ac-44e6-82bf-5af52a3fe435 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8ba823f313144da3b683bacef8fd62e0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81409) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:21:57.822792 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-329f1a67-d1ac-44e6-82bf-5af52a3fe435 admin admin] REQUEST_METHOD: `GET` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:21:57.823751 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-329f1a67-d1ac-44e6-82bf-5af52a3fe435 admin admin] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:21:57.824083 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-329f1a67-d1ac-44e6-82bf-5af52a3fe435 admin admin] PATH_INFO: `/v3/roles` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:21:57.829035 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-329f1a67-d1ac-44e6-82bf-5af52a3fe435 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:21:57.832199 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-329f1a67-d1ac-44e6-82bf-5af52a3fe435 admin admin] RBAC: Authorization granted {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:21:57.867190 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 855/1679] 10.222.0.27 () {58 vars in 1122 bytes} [Thu Oct 13 04:21:57 2022] GET /identity/v3/roles => generated 1420 bytes in 72 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 13 04:21:57.872171 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c7e01531-ffb7-4b17-b90c-d29e1739f348 admin admin] Authenticating user token {{(pid=81408) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:21:57.894722 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c7e01531-ffb7-4b17-b90c-d29e1739f348 admin admin] Validating token access rules against request {{(pid=81408) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:21:57.913675 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e95e1f54-cc2b-451b-a1de-c751381bf798 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8ba823f313144da3b683bacef8fd62e0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81408) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:21:57.915124 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e95e1f54-cc2b-451b-a1de-c751381bf798 admin admin] REQUEST_METHOD: `GET` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:21:57.915422 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e95e1f54-cc2b-451b-a1de-c751381bf798 admin admin] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:21:57.915693 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e95e1f54-cc2b-451b-a1de-c751381bf798 admin admin] PATH_INFO: `/v3/roles` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:21:57.916351 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e95e1f54-cc2b-451b-a1de-c751381bf798 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:21:57.918024 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e95e1f54-cc2b-451b-a1de-c751381bf798 admin admin] RBAC: Authorization granted {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:21:57.936652 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 825/1680] 10.222.0.27 () {58 vars in 1122 bytes} [Thu Oct 13 04:21:57 2022] GET /identity/v3/roles => generated 1420 bytes in 66 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 13 04:21:57.941099 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-329f1a67-d1ac-44e6-82bf-5af52a3fe435 admin admin] Authenticating user token {{(pid=81409) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:21:57.958729 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-329f1a67-d1ac-44e6-82bf-5af52a3fe435 admin admin] Validating token access rules against request {{(pid=81409) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:21:57.966537 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0be6f7fe-738a-4e32-9cc0-de458e5c9389 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8ba823f313144da3b683bacef8fd62e0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81409) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:21:57.967685 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0be6f7fe-738a-4e32-9cc0-de458e5c9389 admin admin] REQUEST_METHOD: `PUT` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:21:57.967974 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0be6f7fe-738a-4e32-9cc0-de458e5c9389 admin admin] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:21:57.968250 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0be6f7fe-738a-4e32-9cc0-de458e5c9389 admin admin] PATH_INFO: `/v3/projects/e6c0a0159bf94b63b29e6cf320832781/users/42b59eb051a54fbb8104ca09ca1fe1a8/roles/cc56c4ae13a7424ea771c15da4b153c1` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:21:57.988832 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0be6f7fe-738a-4e32-9cc0-de458e5c9389 admin admin] RBAC: Authorizing `identity:create_grant(project_id=e6c0a0159bf94b63b29e6cf320832781, user_id=42b59eb051a54fbb8104ca09ca1fe1a8, role_id=cc56c4ae13a7424ea771c15da4b153c1)` {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:21:57.991918 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0be6f7fe-738a-4e32-9cc0-de458e5c9389 admin admin] RBAC: Authorization granted {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:21:58.009384 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 856/1681] 10.222.0.27 () {60 vars in 1483 bytes} [Thu Oct 13 04:21:57 2022] PUT /identity/v3/projects/e6c0a0159bf94b63b29e6cf320832781/users/42b59eb051a54fbb8104ca09ca1fe1a8/roles/cc56c4ae13a7424ea771c15da4b153c1 => generated 0 bytes in 69 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 13 04:21:58.022374 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1d77d38-5120-4a98-a11c-4f72b555939a None None] REQUEST_METHOD: `POST` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:21:58.022713 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1d77d38-5120-4a98-a11c-4f72b555939a None None] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:21:58.022991 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1d77d38-5120-4a98-a11c-4f72b555939a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:21:58.067494 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.auth.core [None req-d1d77d38-5120-4a98-a11c-4f72b555939a None None] MFA Rules not processed for user `42b59eb051a54fbb8104ca09ca1fe1a8`. Rule list: `[]` (Enabled: `True`). {{(pid=81408) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 13 04:21:58.070102 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.fernet_utils [None req-d1d77d38-5120-4a98-a11c-4f72b555939a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81408) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:21:58.236766 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG dbcounter [-] [81408] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=81408) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:21:58.247494 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 826/1682] 10.222.0.27 () {58 vars in 958 bytes} [Thu Oct 13 04:21:58 2022] POST /identity/v3/auth/tokens => generated 3256 bytes in 226 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 13 04:21:58.251635 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0be6f7fe-738a-4e32-9cc0-de458e5c9389 admin admin] Authenticating user token {{(pid=81409) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:21:58.370553 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0be6f7fe-738a-4e32-9cc0-de458e5c9389 admin admin] Validating token access rules against request {{(pid=81409) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:21:58.376099 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f8a0e2e9-4161-4850-8e06-68f100383e7c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8ba823f313144da3b683bacef8fd62e0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81409) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:21:58.376708 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8a0e2e9-4161-4850-8e06-68f100383e7c admin admin] REQUEST_METHOD: `POST` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:21:58.377004 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8a0e2e9-4161-4850-8e06-68f100383e7c admin admin] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:21:58.377267 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8a0e2e9-4161-4850-8e06-68f100383e7c admin admin] PATH_INFO: `/v3/projects` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:21:58.377892 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f8a0e2e9-4161-4850-8e06-68f100383e7c admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:21:58.379813 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f8a0e2e9-4161-4850-8e06-68f100383e7c admin admin] RBAC: Authorization granted {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:21:58.407233 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 857/1683] 10.222.0.27 () {60 vars in 1153 bytes} [Thu Oct 13 04:21:58 2022] POST /identity/v3/projects => generated 379 bytes in 160 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 13 04:21:58.413073 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d1d77d38-5120-4a98-a11c-4f72b555939a None None] Authenticating user token {{(pid=81408) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:21:58.424878 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d1d77d38-5120-4a98-a11c-4f72b555939a None None] Validating token access rules against request {{(pid=81408) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:21:58.490812 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1db17f14-bdac-4a9b-a265-802ee609fd4d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8ba823f313144da3b683bacef8fd62e0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81408) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:21:58.491341 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1db17f14-bdac-4a9b-a265-802ee609fd4d admin admin] REQUEST_METHOD: `POST` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:21:58.491638 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1db17f14-bdac-4a9b-a265-802ee609fd4d admin admin] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:21:58.492062 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1db17f14-bdac-4a9b-a265-802ee609fd4d admin admin] PATH_INFO: `/v3/users` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:21:58.492731 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1db17f14-bdac-4a9b-a265-802ee609fd4d admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:21:58.494813 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1db17f14-bdac-4a9b-a265-802ee609fd4d admin admin] RBAC: Authorization granted {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:21:58.517084 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 827/1684] 10.222.0.27 () {60 vars in 1144 bytes} [Thu Oct 13 04:21:58 2022] POST /identity/v3/users => generated 296 bytes in 105 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 13 04:21:58.521183 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f8a0e2e9-4161-4850-8e06-68f100383e7c admin admin] Authenticating user token {{(pid=81409) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:21:58.531670 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f8a0e2e9-4161-4850-8e06-68f100383e7c admin admin] Validating token access rules against request {{(pid=81409) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:21:58.536207 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c0f0dd67-ae2b-4dbc-8aa7-22909fe468cf admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8ba823f313144da3b683bacef8fd62e0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81409) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:21:58.536812 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0f0dd67-ae2b-4dbc-8aa7-22909fe468cf admin admin] REQUEST_METHOD: `GET` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:21:58.537085 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0f0dd67-ae2b-4dbc-8aa7-22909fe468cf admin admin] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:21:58.537352 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0f0dd67-ae2b-4dbc-8aa7-22909fe468cf admin admin] PATH_INFO: `/v3/roles` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:21:58.537986 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c0f0dd67-ae2b-4dbc-8aa7-22909fe468cf admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:21:58.539613 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c0f0dd67-ae2b-4dbc-8aa7-22909fe468cf admin admin] RBAC: Authorization granted {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:21:58.551425 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 858/1685] 10.222.0.27 () {58 vars in 1122 bytes} [Thu Oct 13 04:21:58 2022] GET /identity/v3/roles => generated 1420 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 13 04:21:58.556182 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1db17f14-bdac-4a9b-a265-802ee609fd4d admin admin] Authenticating user token {{(pid=81408) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:21:58.567436 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1db17f14-bdac-4a9b-a265-802ee609fd4d admin admin] Validating token access rules against request {{(pid=81408) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:21:58.572935 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-98f94c7d-fa77-4641-9191-585f4d7784d9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8ba823f313144da3b683bacef8fd62e0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81408) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:21:58.573614 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98f94c7d-fa77-4641-9191-585f4d7784d9 admin admin] REQUEST_METHOD: `PUT` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:21:58.573889 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98f94c7d-fa77-4641-9191-585f4d7784d9 admin admin] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:21:58.574167 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98f94c7d-fa77-4641-9191-585f4d7784d9 admin admin] PATH_INFO: `/v3/projects/eba3ba669eda49388a393dae4e68b331/users/59f91b539a2a45cbbc682c85d5eb0219/roles/ef55fbd94e1040be83695754075a7ed5` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:21:58.591219 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-98f94c7d-fa77-4641-9191-585f4d7784d9 admin admin] RBAC: Authorizing `identity:create_grant(project_id=eba3ba669eda49388a393dae4e68b331, user_id=59f91b539a2a45cbbc682c85d5eb0219, role_id=ef55fbd94e1040be83695754075a7ed5)` {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:21:58.593915 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-98f94c7d-fa77-4641-9191-585f4d7784d9 admin admin] RBAC: Authorization granted {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:21:58.607561 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 828/1686] 10.222.0.27 () {60 vars in 1483 bytes} [Thu Oct 13 04:21:58 2022] PUT /identity/v3/projects/eba3ba669eda49388a393dae4e68b331/users/59f91b539a2a45cbbc682c85d5eb0219/roles/ef55fbd94e1040be83695754075a7ed5 => generated 0 bytes in 52 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 13 04:21:58.612110 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c0f0dd67-ae2b-4dbc-8aa7-22909fe468cf admin admin] Authenticating user token {{(pid=81409) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:21:58.623316 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c0f0dd67-ae2b-4dbc-8aa7-22909fe468cf admin admin] Validating token access rules against request {{(pid=81409) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:21:58.628061 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG dbcounter [-] [81409] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=81409) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:21:58.681636 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fa3184b1-4cb6-4a96-a9a9-c37dab4cbb1d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8ba823f313144da3b683bacef8fd62e0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81409) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:21:58.682285 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa3184b1-4cb6-4a96-a9a9-c37dab4cbb1d admin admin] REQUEST_METHOD: `GET` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:21:58.682561 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa3184b1-4cb6-4a96-a9a9-c37dab4cbb1d admin admin] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:21:58.682831 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa3184b1-4cb6-4a96-a9a9-c37dab4cbb1d admin admin] PATH_INFO: `/v3/roles` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:21:58.683471 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fa3184b1-4cb6-4a96-a9a9-c37dab4cbb1d admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:21:58.685694 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fa3184b1-4cb6-4a96-a9a9-c37dab4cbb1d admin admin] RBAC: Authorization granted {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:21:58.695651 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 859/1687] 10.222.0.27 () {58 vars in 1122 bytes} [Thu Oct 13 04:21:58 2022] GET /identity/v3/roles => generated 1420 bytes in 84 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 13 04:21:58.699958 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-98f94c7d-fa77-4641-9191-585f4d7784d9 admin admin] Authenticating user token {{(pid=81408) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:21:58.711539 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-98f94c7d-fa77-4641-9191-585f4d7784d9 admin admin] Validating token access rules against request {{(pid=81408) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:21:58.720848 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bf388302-0fb3-47cc-a94f-0301190f41bb admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8ba823f313144da3b683bacef8fd62e0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81408) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:21:58.721531 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf388302-0fb3-47cc-a94f-0301190f41bb admin admin] REQUEST_METHOD: `PUT` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:21:58.721971 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf388302-0fb3-47cc-a94f-0301190f41bb admin admin] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:21:58.722281 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf388302-0fb3-47cc-a94f-0301190f41bb admin admin] PATH_INFO: `/v3/projects/eba3ba669eda49388a393dae4e68b331/users/59f91b539a2a45cbbc682c85d5eb0219/roles/ef55fbd94e1040be83695754075a7ed5` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:21:58.724234 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bf388302-0fb3-47cc-a94f-0301190f41bb admin admin] RBAC: Authorizing `identity:create_grant(project_id=eba3ba669eda49388a393dae4e68b331, user_id=59f91b539a2a45cbbc682c85d5eb0219, role_id=ef55fbd94e1040be83695754075a7ed5)` {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:21:58.727095 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bf388302-0fb3-47cc-a94f-0301190f41bb admin admin] RBAC: Authorization granted {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:21:58.739058 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 829/1688] 10.222.0.27 () {60 vars in 1483 bytes} [Thu Oct 13 04:21:58 2022] PUT /identity/v3/projects/eba3ba669eda49388a393dae4e68b331/users/59f91b539a2a45cbbc682c85d5eb0219/roles/ef55fbd94e1040be83695754075a7ed5 => generated 0 bytes in 40 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 13 04:21:58.750008 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e18c1bf3-9881-42d5-a4e8-d4a65f498470 None None] REQUEST_METHOD: `POST` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:21:58.750440 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e18c1bf3-9881-42d5-a4e8-d4a65f498470 None None] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:21:58.750768 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e18c1bf3-9881-42d5-a4e8-d4a65f498470 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:21:58.778894 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.auth.core [None req-e18c1bf3-9881-42d5-a4e8-d4a65f498470 None None] MFA Rules not processed for user `59f91b539a2a45cbbc682c85d5eb0219`. Rule list: `[]` (Enabled: `True`). {{(pid=81409) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 13 04:21:58.780646 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.fernet_utils [None req-e18c1bf3-9881-42d5-a4e8-d4a65f498470 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81409) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:21:58.861104 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 860/1689] 10.222.0.27 () {58 vars in 958 bytes} [Thu Oct 13 04:21:58 2022] POST /identity/v3/auth/tokens => generated 3323 bytes in 112 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 13 04:21:58.875461 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bf388302-0fb3-47cc-a94f-0301190f41bb admin admin] Authenticating user token {{(pid=81408) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:21:58.952900 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bf388302-0fb3-47cc-a94f-0301190f41bb admin admin] Validating token access rules against request {{(pid=81408) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:21:58.957036 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e04ba973-5a3b-435c-b736-6217d5f73cb1 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6141b8a6c3d54e179d6879bb019f4d12', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dcec0431fb6849499d6aec35a234daa6', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81408) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:21:58.957636 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e04ba973-5a3b-435c-b736-6217d5f73cb1 service cinder] REQUEST_METHOD: `GET` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:21:58.957946 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e04ba973-5a3b-435c-b736-6217d5f73cb1 service cinder] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:21:58.958189 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e04ba973-5a3b-435c-b736-6217d5f73cb1 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:21:58.968827 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e04ba973-5a3b-435c-b736-6217d5f73cb1 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:21:58.970290 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e04ba973-5a3b-435c-b736-6217d5f73cb1 service cinder] RBAC: Authorization granted {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:21:58.985498 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 830/1690] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Oct 13 04:21:58 2022] GET /identity/v3/auth/tokens => generated 3323 bytes in 111 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 13 04:21:59.547584 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e18c1bf3-9881-42d5-a4e8-d4a65f498470 None None] Authenticating user token {{(pid=81409) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:21:59.660190 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e18c1bf3-9881-42d5-a4e8-d4a65f498470 None None] Validating token access rules against request {{(pid=81409) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:21:59.666033 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7e9cdc76-0b30-4e94-9e6f-a357d40490c9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8ba823f313144da3b683bacef8fd62e0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81409) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:21:59.666729 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e9cdc76-0b30-4e94-9e6f-a357d40490c9 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:21:59.667010 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e9cdc76-0b30-4e94-9e6f-a357d40490c9 admin admin] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:21:59.667280 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e9cdc76-0b30-4e94-9e6f-a357d40490c9 admin admin] PATH_INFO: `/v3/users/42b59eb051a54fbb8104ca09ca1fe1a8` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:21:59.668432 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7e9cdc76-0b30-4e94-9e6f-a357d40490c9 admin admin] RBAC: Authorizing `identity:delete_user(user_id=42b59eb051a54fbb8104ca09ca1fe1a8)` {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:21:59.670399 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7e9cdc76-0b30-4e94-9e6f-a357d40490c9 admin admin] RBAC: Authorization granted {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:21:59.707387 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG dbcounter [-] [81409] Writing DB stats keystone:SELECT=95,keystone:DELETE=5 {{(pid=81409) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:21:59.720952 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.notifications [None req-7e9cdc76-0b30-4e94-9e6f-a357d40490c9 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '42b59eb051a54fbb8104ca09ca1fe1a8'} {{(pid=81409) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 13 04:21:59.721602 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.notifications [None req-7e9cdc76-0b30-4e94-9e6f-a357d40490c9 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '42b59eb051a54fbb8104ca09ca1fe1a8'} {{(pid=81409) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 13 04:21:59.728415 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.notifications [None req-7e9cdc76-0b30-4e94-9e6f-a357d40490c9 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '42b59eb051a54fbb8104ca09ca1fe1a8'} {{(pid=81409) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 13 04:21:59.729046 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.notifications [None req-7e9cdc76-0b30-4e94-9e6f-a357d40490c9 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '42b59eb051a54fbb8104ca09ca1fe1a8'} {{(pid=81409) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 13 04:21:59.736091 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.notifications [None req-7e9cdc76-0b30-4e94-9e6f-a357d40490c9 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '42b59eb051a54fbb8104ca09ca1fe1a8'} {{(pid=81409) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 13 04:21:59.757139 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 861/1691] 10.222.0.27 () {58 vars in 1224 bytes} [Thu Oct 13 04:21:59 2022] DELETE /identity/v3/users/42b59eb051a54fbb8104ca09ca1fe1a8 => generated 0 bytes in 208 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 13 04:21:59.758720 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c49b2cda-b7f8-4d07-bc93-9c195cdf443f None None] REQUEST_METHOD: `POST` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:21:59.759215 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c49b2cda-b7f8-4d07-bc93-9c195cdf443f None None] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:21:59.759332 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c49b2cda-b7f8-4d07-bc93-9c195cdf443f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:21:59.779801 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.auth.core [None req-c49b2cda-b7f8-4d07-bc93-9c195cdf443f None None] MFA Rules not processed for user `0e2d979563d6421995451a0e86bf3022`. Rule list: `[]` (Enabled: `True`). {{(pid=81408) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 13 04:21:59.781013 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.fernet_utils [None req-c49b2cda-b7f8-4d07-bc93-9c195cdf443f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81408) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:21:59.833721 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG dbcounter [-] [81408] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=81408) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:21:59.846339 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 831/1692] 10.222.0.27 () {58 vars in 958 bytes} [Thu Oct 13 04:21:59 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 89 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 13 04:21:59.856407 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7e9cdc76-0b30-4e94-9e6f-a357d40490c9 admin admin] Authenticating user token {{(pid=81409) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:21:59.857786 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.fernet_utils [None req-7e9cdc76-0b30-4e94-9e6f-a357d40490c9 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81409) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:21:59.918887 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7e9cdc76-0b30-4e94-9e6f-a357d40490c9 admin admin] Validating token access rules against request {{(pid=81409) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:21:59.922851 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-423c9b59-e011-438f-a747-d56075d2ec9d service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd8812c3974f84871bb2d761cf0016257', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dcec0431fb6849499d6aec35a234daa6', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81409) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:21:59.923465 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-423c9b59-e011-438f-a747-d56075d2ec9d service neutron] REQUEST_METHOD: `GET` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:21:59.923738 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-423c9b59-e011-438f-a747-d56075d2ec9d service neutron] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:21:59.924005 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-423c9b59-e011-438f-a747-d56075d2ec9d service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:21:59.934725 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-423c9b59-e011-438f-a747-d56075d2ec9d service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:21:59.936423 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-423c9b59-e011-438f-a747-d56075d2ec9d service neutron] RBAC: Authorization granted {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:21:59.947680 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 862/1693] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Oct 13 04:21:59 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 92 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 13 04:22:00.253284 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c49b2cda-b7f8-4d07-bc93-9c195cdf443f None None] Authenticating user token {{(pid=81408) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:22:00.254866 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.fernet_utils [None req-c49b2cda-b7f8-4d07-bc93-9c195cdf443f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81408) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:22:00.268458 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c49b2cda-b7f8-4d07-bc93-9c195cdf443f None None] Validating token access rules against request {{(pid=81408) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:22:00.275637 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8bd32fae-c027-46aa-85c6-a5dfda12ca62 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8ba823f313144da3b683bacef8fd62e0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81408) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:22:00.276448 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8bd32fae-c027-46aa-85c6-a5dfda12ca62 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:22:00.276756 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8bd32fae-c027-46aa-85c6-a5dfda12ca62 admin admin] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:22:00.276950 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8bd32fae-c027-46aa-85c6-a5dfda12ca62 admin admin] PATH_INFO: `/v3/projects/e6c0a0159bf94b63b29e6cf320832781` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:22:00.278721 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8bd32fae-c027-46aa-85c6-a5dfda12ca62 admin admin] RBAC: Authorizing `identity:delete_project(project_id=e6c0a0159bf94b63b29e6cf320832781)` {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:22:00.280720 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8bd32fae-c027-46aa-85c6-a5dfda12ca62 admin admin] RBAC: Authorization granted {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:22:00.338037 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.notifications [None req-8bd32fae-c027-46aa-85c6-a5dfda12ca62 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'e6c0a0159bf94b63b29e6cf320832781', 'request_id': 'req-8bd32fae-c027-46aa-85c6-a5dfda12ca62'} {{(pid=81408) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 13 04:22:00.347182 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.notifications [None req-8bd32fae-c027-46aa-85c6-a5dfda12ca62 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'e6c0a0159bf94b63b29e6cf320832781', 'request_id': 'req-8bd32fae-c027-46aa-85c6-a5dfda12ca62'} {{(pid=81408) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 13 04:22:00.353987 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.notifications [None req-8bd32fae-c027-46aa-85c6-a5dfda12ca62 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'e6c0a0159bf94b63b29e6cf320832781', 'request_id': 'req-8bd32fae-c027-46aa-85c6-a5dfda12ca62'} {{(pid=81408) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 13 04:22:00.366095 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.notifications [None req-8bd32fae-c027-46aa-85c6-a5dfda12ca62 admin admin] The token cache is being invalidate because project e6c0a0159bf94b63b29e6cf320832781 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81408) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 13 04:22:00.366425 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.notifications [None req-8bd32fae-c027-46aa-85c6-a5dfda12ca62 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81408) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 13 04:22:00.367072 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.notifications [None req-8bd32fae-c027-46aa-85c6-a5dfda12ca62 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81408) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 13 04:22:00.368793 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 832/1694] 10.222.0.27 () {58 vars in 1233 bytes} [Thu Oct 13 04:22:00 2022] DELETE /identity/v3/projects/e6c0a0159bf94b63b29e6cf320832781 => generated 0 bytes in 122 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 13 04:22:00.372719 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-423c9b59-e011-438f-a747-d56075d2ec9d service neutron] Authenticating user token {{(pid=81409) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:22:00.382889 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-423c9b59-e011-438f-a747-d56075d2ec9d service neutron] Validating token access rules against request {{(pid=81409) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:22:00.448937 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-15d04784-eb5c-4f84-b4ca-cf517b81b712 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8ba823f313144da3b683bacef8fd62e0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81409) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:22:00.450367 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15d04784-eb5c-4f84-b4ca-cf517b81b712 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:22:00.450739 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15d04784-eb5c-4f84-b4ca-cf517b81b712 admin admin] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:22:00.451021 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15d04784-eb5c-4f84-b4ca-cf517b81b712 admin admin] PATH_INFO: `/v3/users/59f91b539a2a45cbbc682c85d5eb0219` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:22:00.452166 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-15d04784-eb5c-4f84-b4ca-cf517b81b712 admin admin] RBAC: Authorizing `identity:delete_user(user_id=59f91b539a2a45cbbc682c85d5eb0219)` {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:22:00.454097 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-15d04784-eb5c-4f84-b4ca-cf517b81b712 admin admin] RBAC: Authorization granted {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:22:00.505980 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.notifications [None req-15d04784-eb5c-4f84-b4ca-cf517b81b712 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '59f91b539a2a45cbbc682c85d5eb0219'} {{(pid=81409) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 13 04:22:00.506748 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.notifications [None req-15d04784-eb5c-4f84-b4ca-cf517b81b712 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '59f91b539a2a45cbbc682c85d5eb0219'} {{(pid=81409) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 13 04:22:00.512521 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG dbcounter [-] [81409] Writing DB stats keystone:SELECT=87,keystone:DELETE=12,keystone:INSERT=1 {{(pid=81409) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:22:00.515064 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.notifications [None req-15d04784-eb5c-4f84-b4ca-cf517b81b712 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '59f91b539a2a45cbbc682c85d5eb0219'} {{(pid=81409) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 13 04:22:00.516048 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.notifications [None req-15d04784-eb5c-4f84-b4ca-cf517b81b712 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '59f91b539a2a45cbbc682c85d5eb0219'} {{(pid=81409) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 13 04:22:00.525556 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.notifications [None req-15d04784-eb5c-4f84-b4ca-cf517b81b712 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '59f91b539a2a45cbbc682c85d5eb0219'} {{(pid=81409) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 13 04:22:00.545160 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 863/1695] 10.222.0.27 () {58 vars in 1224 bytes} [Thu Oct 13 04:22:00 2022] DELETE /identity/v3/users/59f91b539a2a45cbbc682c85d5eb0219 => generated 0 bytes in 173 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 13 04:22:00.671080 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8bd32fae-c027-46aa-85c6-a5dfda12ca62 admin admin] Authenticating user token {{(pid=81408) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:22:00.672556 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.fernet_utils [None req-8bd32fae-c027-46aa-85c6-a5dfda12ca62 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81408) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:22:00.690561 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8bd32fae-c027-46aa-85c6-a5dfda12ca62 admin admin] Validating token access rules against request {{(pid=81408) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:22:00.773527 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8a828d50-c700-4eba-9647-55b181c71b70 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8ba823f313144da3b683bacef8fd62e0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81408) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:22:00.774361 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a828d50-c700-4eba-9647-55b181c71b70 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:22:00.774754 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a828d50-c700-4eba-9647-55b181c71b70 admin admin] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:22:00.774995 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a828d50-c700-4eba-9647-55b181c71b70 admin admin] PATH_INFO: `/v3/users/6d6b64e609474033ae6b1cce827e277e` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:22:00.776243 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8a828d50-c700-4eba-9647-55b181c71b70 admin admin] RBAC: Authorizing `identity:delete_user(user_id=6d6b64e609474033ae6b1cce827e277e)` {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:22:00.780569 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8a828d50-c700-4eba-9647-55b181c71b70 admin admin] RBAC: Authorization granted {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:22:00.844757 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.notifications [None req-8a828d50-c700-4eba-9647-55b181c71b70 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '6d6b64e609474033ae6b1cce827e277e'} {{(pid=81408) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 13 04:22:00.851833 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.notifications [None req-8a828d50-c700-4eba-9647-55b181c71b70 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '6d6b64e609474033ae6b1cce827e277e'} {{(pid=81408) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 13 04:22:00.852580 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.notifications [None req-8a828d50-c700-4eba-9647-55b181c71b70 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '6d6b64e609474033ae6b1cce827e277e'} {{(pid=81408) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 13 04:22:00.853242 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.notifications [None req-8a828d50-c700-4eba-9647-55b181c71b70 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '6d6b64e609474033ae6b1cce827e277e'} {{(pid=81408) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 13 04:22:00.861876 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.notifications [None req-8a828d50-c700-4eba-9647-55b181c71b70 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '6d6b64e609474033ae6b1cce827e277e'} {{(pid=81408) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 13 04:22:00.880923 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG dbcounter [-] [81408] Writing DB stats keystone:SELECT=81,keystone:DELETE=17,keystone:INSERT=2 {{(pid=81408) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:22:00.886327 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 833/1696] 10.222.0.27 () {58 vars in 1224 bytes} [Thu Oct 13 04:22:00 2022] DELETE /identity/v3/users/6d6b64e609474033ae6b1cce827e277e => generated 0 bytes in 216 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 13 04:22:00.891066 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-107df024-70e0-47a9-9e04-72444a0ce864 None None] REQUEST_METHOD: `POST` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:22:00.891923 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-107df024-70e0-47a9-9e04-72444a0ce864 None None] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:22:00.892401 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-107df024-70e0-47a9-9e04-72444a0ce864 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:22:00.912707 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.auth.core [None req-107df024-70e0-47a9-9e04-72444a0ce864 None None] MFA Rules not processed for user `0e2d979563d6421995451a0e86bf3022`. Rule list: `[]` (Enabled: `True`). {{(pid=81409) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 13 04:22:00.914084 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.fernet_utils [None req-107df024-70e0-47a9-9e04-72444a0ce864 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81409) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:22:00.955850 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8a828d50-c700-4eba-9647-55b181c71b70 admin admin] Authenticating user token {{(pid=81408) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:22:00.957463 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.fernet_utils [None req-8a828d50-c700-4eba-9647-55b181c71b70 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81408) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:22:00.988887 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 864/1697] 10.222.0.27 () {58 vars in 958 bytes} [Thu Oct 13 04:22:00 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 99 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 13 04:22:00.999395 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-107df024-70e0-47a9-9e04-72444a0ce864 None None] Authenticating user token {{(pid=81409) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:22:01.000852 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.fernet_utils [None req-107df024-70e0-47a9-9e04-72444a0ce864 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81409) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:22:01.032380 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8a828d50-c700-4eba-9647-55b181c71b70 admin admin] Validating token access rules against request {{(pid=81408) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:22:01.037913 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4babc94e-e25c-49f8-aa5c-55077ab293ab admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8ba823f313144da3b683bacef8fd62e0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81408) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:22:01.038591 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4babc94e-e25c-49f8-aa5c-55077ab293ab admin admin] REQUEST_METHOD: `DELETE` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:22:01.038872 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4babc94e-e25c-49f8-aa5c-55077ab293ab admin admin] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:22:01.039146 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4babc94e-e25c-49f8-aa5c-55077ab293ab admin admin] PATH_INFO: `/v3/projects/eba3ba669eda49388a393dae4e68b331` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:22:01.040282 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4babc94e-e25c-49f8-aa5c-55077ab293ab admin admin] RBAC: Authorizing `identity:delete_project(project_id=eba3ba669eda49388a393dae4e68b331)` {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:22:01.042584 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4babc94e-e25c-49f8-aa5c-55077ab293ab admin admin] RBAC: Authorization granted {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:22:01.064151 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-107df024-70e0-47a9-9e04-72444a0ce864 None None] Validating token access rules against request {{(pid=81409) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:22:01.068118 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7a2bd130-d87a-4e83-b393-f8d939aa0862 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd8812c3974f84871bb2d761cf0016257', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dcec0431fb6849499d6aec35a234daa6', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81409) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:22:01.068553 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a2bd130-d87a-4e83-b393-f8d939aa0862 service neutron] REQUEST_METHOD: `GET` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:22:01.068824 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a2bd130-d87a-4e83-b393-f8d939aa0862 service neutron] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:22:01.069091 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a2bd130-d87a-4e83-b393-f8d939aa0862 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:22:01.081841 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7a2bd130-d87a-4e83-b393-f8d939aa0862 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:22:01.083385 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7a2bd130-d87a-4e83-b393-f8d939aa0862 service neutron] RBAC: Authorization granted {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:22:01.094805 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 865/1698] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Oct 13 04:22:00 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 96 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Oct 13 04:22:01.122775 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.notifications [None req-4babc94e-e25c-49f8-aa5c-55077ab293ab admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'eba3ba669eda49388a393dae4e68b331', 'request_id': 'req-4babc94e-e25c-49f8-aa5c-55077ab293ab'} {{(pid=81408) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 13 04:22:01.131880 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.notifications [None req-4babc94e-e25c-49f8-aa5c-55077ab293ab admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'eba3ba669eda49388a393dae4e68b331', 'request_id': 'req-4babc94e-e25c-49f8-aa5c-55077ab293ab'} {{(pid=81408) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 13 04:22:01.139846 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.notifications [None req-4babc94e-e25c-49f8-aa5c-55077ab293ab admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'eba3ba669eda49388a393dae4e68b331', 'request_id': 'req-4babc94e-e25c-49f8-aa5c-55077ab293ab'} {{(pid=81408) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 13 04:22:01.151832 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.notifications [None req-4babc94e-e25c-49f8-aa5c-55077ab293ab admin admin] The token cache is being invalidate because project eba3ba669eda49388a393dae4e68b331 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81408) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 13 04:22:01.152202 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.notifications [None req-4babc94e-e25c-49f8-aa5c-55077ab293ab admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81408) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 13 04:22:01.153029 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.notifications [None req-4babc94e-e25c-49f8-aa5c-55077ab293ab admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81408) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 13 04:22:01.154969 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 834/1699] 10.222.0.27 () {58 vars in 1233 bytes} [Thu Oct 13 04:22:00 2022] DELETE /identity/v3/projects/eba3ba669eda49388a393dae4e68b331 => generated 0 bytes in 200 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 13 04:22:01.159860 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da03bcbb-c22b-410a-b8ac-e0d42ea1b119 None None] REQUEST_METHOD: `POST` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:22:01.160144 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da03bcbb-c22b-410a-b8ac-e0d42ea1b119 None None] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:22:01.160407 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da03bcbb-c22b-410a-b8ac-e0d42ea1b119 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:22:01.181809 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.auth.core [None req-da03bcbb-c22b-410a-b8ac-e0d42ea1b119 None None] MFA Rules not processed for user `0e2d979563d6421995451a0e86bf3022`. Rule list: `[]` (Enabled: `True`). {{(pid=81409) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 13 04:22:01.183441 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.fernet_utils [None req-da03bcbb-c22b-410a-b8ac-e0d42ea1b119 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81409) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:22:01.251481 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 866/1700] 10.222.0.27 () {58 vars in 958 bytes} [Thu Oct 13 04:22:01 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 93 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 13 04:22:01.260545 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9b0c56f-d3e4-4af4-95d5-5e209d5fe01d None None] REQUEST_METHOD: `POST` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:22:01.261324 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9b0c56f-d3e4-4af4-95d5-5e209d5fe01d None None] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:22:01.261521 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9b0c56f-d3e4-4af4-95d5-5e209d5fe01d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:22:01.287402 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.auth.core [None req-a9b0c56f-d3e4-4af4-95d5-5e209d5fe01d None None] MFA Rules not processed for user `0e2d979563d6421995451a0e86bf3022`. Rule list: `[]` (Enabled: `True`). {{(pid=81408) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 13 04:22:01.288631 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.fernet_utils [None req-a9b0c56f-d3e4-4af4-95d5-5e209d5fe01d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81408) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:22:01.298703 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 835/1701] 10.222.0.27 () {58 vars in 958 bytes} [Thu Oct 13 04:22:01 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 39 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 13 04:22:01.303151 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-da03bcbb-c22b-410a-b8ac-e0d42ea1b119 None None] Authenticating user token {{(pid=81409) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:22:01.313724 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-da03bcbb-c22b-410a-b8ac-e0d42ea1b119 None None] Validating token access rules against request {{(pid=81409) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:22:01.318448 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2838618e-6589-4c1a-946c-e298f76b6cb8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8ba823f313144da3b683bacef8fd62e0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81409) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:22:01.319056 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2838618e-6589-4c1a-946c-e298f76b6cb8 admin admin] REQUEST_METHOD: `GET` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:22:01.319337 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2838618e-6589-4c1a-946c-e298f76b6cb8 admin admin] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:22:01.319610 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2838618e-6589-4c1a-946c-e298f76b6cb8 admin admin] PATH_INFO: `/v3/domains` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:22:01.320317 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2838618e-6589-4c1a-946c-e298f76b6cb8 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81409) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 13 04:22:01.320932 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2838618e-6589-4c1a-946c-e298f76b6cb8 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:22:01.322230 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2838618e-6589-4c1a-946c-e298f76b6cb8 admin admin] RBAC: Authorization granted {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:22:01.335442 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 867/1702] 10.222.0.27 () {58 vars in 1153 bytes} [Thu Oct 13 04:22:01 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 33 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 13 04:22:01.339715 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a9b0c56f-d3e4-4af4-95d5-5e209d5fe01d None None] Authenticating user token {{(pid=81408) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:22:01.349273 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a9b0c56f-d3e4-4af4-95d5-5e209d5fe01d None None] Validating token access rules against request {{(pid=81408) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:22:01.354436 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4681e1d3-e542-4993-91ba-01d6be4a064d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8ba823f313144da3b683bacef8fd62e0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81408) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:22:01.355106 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4681e1d3-e542-4993-91ba-01d6be4a064d admin admin] REQUEST_METHOD: `POST` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:22:01.355452 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4681e1d3-e542-4993-91ba-01d6be4a064d admin admin] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:22:01.356003 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4681e1d3-e542-4993-91ba-01d6be4a064d admin admin] PATH_INFO: `/v3/projects` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:22:01.356952 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4681e1d3-e542-4993-91ba-01d6be4a064d admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:22:01.358910 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4681e1d3-e542-4993-91ba-01d6be4a064d admin admin] RBAC: Authorization granted {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:22:01.397775 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 836/1703] 10.222.0.27 () {60 vars in 1153 bytes} [Thu Oct 13 04:22:01 2022] POST /identity/v3/projects => generated 363 bytes in 53 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 13 04:22:01.399293 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4681e1d3-e542-4993-91ba-01d6be4a064d admin admin] Authenticating user token {{(pid=81408) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:22:01.399516 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2838618e-6589-4c1a-946c-e298f76b6cb8 admin admin] Authenticating user token {{(pid=81409) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:22:01.401712 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.fernet_utils [None req-2838618e-6589-4c1a-946c-e298f76b6cb8 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81409) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:22:01.417746 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG dbcounter [-] [81409] Writing DB stats keystone:SELECT=96,keystone:INSERT=1,keystone:DELETE=3 {{(pid=81409) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:22:01.425851 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2838618e-6589-4c1a-946c-e298f76b6cb8 admin admin] Validating token access rules against request {{(pid=81409) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:22:01.488834 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4681e1d3-e542-4993-91ba-01d6be4a064d admin admin] Validating token access rules against request {{(pid=81408) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:22:01.493612 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b606f314-1cbe-4e88-932b-830adb35fce7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8ba823f313144da3b683bacef8fd62e0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81408) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:22:01.494247 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b606f314-1cbe-4e88-932b-830adb35fce7 admin admin] REQUEST_METHOD: `POST` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:22:01.494528 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b606f314-1cbe-4e88-932b-830adb35fce7 admin admin] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:22:01.494800 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b606f314-1cbe-4e88-932b-830adb35fce7 admin admin] PATH_INFO: `/v3/users` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:22:01.495435 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b606f314-1cbe-4e88-932b-830adb35fce7 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:22:01.497554 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-087c4715-9513-40db-8ae2-4b50196660e2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8ba823f313144da3b683bacef8fd62e0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81409) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:22:01.497973 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b606f314-1cbe-4e88-932b-830adb35fce7 admin admin] RBAC: Authorization granted {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:22:01.498182 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-087c4715-9513-40db-8ae2-4b50196660e2 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:22:01.498488 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-087c4715-9513-40db-8ae2-4b50196660e2 admin admin] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:22:01.498759 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-087c4715-9513-40db-8ae2-4b50196660e2 admin admin] PATH_INFO: `/v3/projects/19dae11d071942fd8e62c91a1c7716ae` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:22:01.499876 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-087c4715-9513-40db-8ae2-4b50196660e2 admin admin] RBAC: Authorizing `identity:delete_project(project_id=19dae11d071942fd8e62c91a1c7716ae)` {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:22:01.501763 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-087c4715-9513-40db-8ae2-4b50196660e2 admin admin] RBAC: Authorization granted {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:22:01.506315 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG dbcounter [-] [81408] Writing DB stats keystone:SELECT=91,keystone:DELETE=7,keystone:INSERT=2 {{(pid=81408) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:22:01.521854 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 837/1704] 10.222.0.27 () {60 vars in 1144 bytes} [Thu Oct 13 04:22:01 2022] POST /identity/v3/users => generated 282 bytes in 123 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 13 04:22:01.525775 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b606f314-1cbe-4e88-932b-830adb35fce7 admin admin] Authenticating user token {{(pid=81408) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:22:01.541764 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b606f314-1cbe-4e88-932b-830adb35fce7 admin admin] Validating token access rules against request {{(pid=81408) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:22:01.556102 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.notifications [None req-087c4715-9513-40db-8ae2-4b50196660e2 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '19dae11d071942fd8e62c91a1c7716ae', 'request_id': 'req-087c4715-9513-40db-8ae2-4b50196660e2'} {{(pid=81409) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 13 04:22:01.564451 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.notifications [None req-087c4715-9513-40db-8ae2-4b50196660e2 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '19dae11d071942fd8e62c91a1c7716ae', 'request_id': 'req-087c4715-9513-40db-8ae2-4b50196660e2'} {{(pid=81409) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 13 04:22:01.571660 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.notifications [None req-087c4715-9513-40db-8ae2-4b50196660e2 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '19dae11d071942fd8e62c91a1c7716ae', 'request_id': 'req-087c4715-9513-40db-8ae2-4b50196660e2'} {{(pid=81409) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 13 04:22:01.582298 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.notifications [None req-087c4715-9513-40db-8ae2-4b50196660e2 admin admin] The token cache is being invalidate because project 19dae11d071942fd8e62c91a1c7716ae was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81409) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 13 04:22:01.582799 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.notifications [None req-087c4715-9513-40db-8ae2-4b50196660e2 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81409) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 13 04:22:01.583601 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.notifications [None req-087c4715-9513-40db-8ae2-4b50196660e2 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81409) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 13 04:22:01.585299 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 868/1705] 10.222.0.27 () {58 vars in 1233 bytes} [Thu Oct 13 04:22:01 2022] DELETE /identity/v3/projects/19dae11d071942fd8e62c91a1c7716ae => generated 0 bytes in 187 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 13 04:22:01.589168 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-087c4715-9513-40db-8ae2-4b50196660e2 admin admin] Authenticating user token {{(pid=81409) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:22:01.591220 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.fernet_utils [None req-087c4715-9513-40db-8ae2-4b50196660e2 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81409) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:22:01.601505 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a8eea7b3-c3a7-40bd-8392-4c78ebf58abe admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8ba823f313144da3b683bacef8fd62e0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81408) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:22:01.602263 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8eea7b3-c3a7-40bd-8392-4c78ebf58abe admin admin] REQUEST_METHOD: `GET` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:22:01.602545 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8eea7b3-c3a7-40bd-8392-4c78ebf58abe admin admin] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:22:01.602845 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8eea7b3-c3a7-40bd-8392-4c78ebf58abe admin admin] PATH_INFO: `/v3/roles` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:22:01.603521 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a8eea7b3-c3a7-40bd-8392-4c78ebf58abe admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:22:01.605187 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a8eea7b3-c3a7-40bd-8392-4c78ebf58abe admin admin] RBAC: Authorization granted {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:22:01.619470 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 838/1706] 10.222.0.27 () {58 vars in 1122 bytes} [Thu Oct 13 04:22:01 2022] GET /identity/v3/roles => generated 1420 bytes in 91 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Oct 13 04:22:01.622086 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a8eea7b3-c3a7-40bd-8392-4c78ebf58abe admin admin] Authenticating user token {{(pid=81408) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:22:01.624723 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.fernet_utils [None req-a8eea7b3-c3a7-40bd-8392-4c78ebf58abe admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81408) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:22:01.643451 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a8eea7b3-c3a7-40bd-8392-4c78ebf58abe admin admin] Validating token access rules against request {{(pid=81408) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:22:01.648641 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6c5d404d-7b92-4f13-8bba-aee1a93b0b25 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8ba823f313144da3b683bacef8fd62e0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81408) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:22:01.649572 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c5d404d-7b92-4f13-8bba-aee1a93b0b25 admin admin] REQUEST_METHOD: `GET` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:22:01.649572 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c5d404d-7b92-4f13-8bba-aee1a93b0b25 admin admin] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:22:01.649784 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c5d404d-7b92-4f13-8bba-aee1a93b0b25 admin admin] PATH_INFO: `/v3/roles` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:22:01.650529 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6c5d404d-7b92-4f13-8bba-aee1a93b0b25 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:22:01.652124 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6c5d404d-7b92-4f13-8bba-aee1a93b0b25 admin admin] RBAC: Authorization granted {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:22:01.664204 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 839/1707] 10.222.0.27 () {58 vars in 1122 bytes} [Thu Oct 13 04:22:01 2022] GET /identity/v3/roles => generated 1420 bytes in 44 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 13 04:22:01.667777 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-087c4715-9513-40db-8ae2-4b50196660e2 admin admin] Validating token access rules against request {{(pid=81409) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:22:01.668876 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6c5d404d-7b92-4f13-8bba-aee1a93b0b25 admin admin] Authenticating user token {{(pid=81408) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:22:01.678919 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6c5d404d-7b92-4f13-8bba-aee1a93b0b25 admin admin] Validating token access rules against request {{(pid=81408) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:22:01.679094 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8856d457-5f7f-4d29-a8f2-8ec2b546e2ee admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8ba823f313144da3b683bacef8fd62e0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81409) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:22:01.680057 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8856d457-5f7f-4d29-a8f2-8ec2b546e2ee admin admin] REQUEST_METHOD: `DELETE` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:22:01.680351 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8856d457-5f7f-4d29-a8f2-8ec2b546e2ee admin admin] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:22:01.680687 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8856d457-5f7f-4d29-a8f2-8ec2b546e2ee admin admin] PATH_INFO: `/v3/users/9ed11ff4bed24e0696d6b8e7c0076d55` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:22:01.681758 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8856d457-5f7f-4d29-a8f2-8ec2b546e2ee admin admin] RBAC: Authorizing `identity:delete_user(user_id=9ed11ff4bed24e0696d6b8e7c0076d55)` {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:22:01.683674 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e88f87e3-425b-496a-a292-a84cc8c898ea admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8ba823f313144da3b683bacef8fd62e0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81408) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:22:01.685247 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e88f87e3-425b-496a-a292-a84cc8c898ea admin admin] REQUEST_METHOD: `PUT` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:22:01.685359 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8856d457-5f7f-4d29-a8f2-8ec2b546e2ee admin admin] RBAC: Authorization granted {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:22:01.685628 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e88f87e3-425b-496a-a292-a84cc8c898ea admin admin] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:22:01.685904 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e88f87e3-425b-496a-a292-a84cc8c898ea admin admin] PATH_INFO: `/v3/projects/188def90cec84de6b0527111babb5370/users/ca7301586f634c90a52759f788ef0b46/roles/cc56c4ae13a7424ea771c15da4b153c1` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:22:01.701286 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e88f87e3-425b-496a-a292-a84cc8c898ea admin admin] RBAC: Authorizing `identity:create_grant(project_id=188def90cec84de6b0527111babb5370, user_id=ca7301586f634c90a52759f788ef0b46, role_id=cc56c4ae13a7424ea771c15da4b153c1)` {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:22:01.704267 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e88f87e3-425b-496a-a292-a84cc8c898ea admin admin] RBAC: Authorization granted {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:22:01.725635 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 840/1708] 10.222.0.27 () {60 vars in 1483 bytes} [Thu Oct 13 04:22:01 2022] PUT /identity/v3/projects/188def90cec84de6b0527111babb5370/users/ca7301586f634c90a52759f788ef0b46/roles/cc56c4ae13a7424ea771c15da4b153c1 => generated 0 bytes in 58 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 13 04:22:01.727219 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG dbcounter [-] [81409] Writing DB stats keystone:SELECT=87,keystone:DELETE=12,keystone:INSERT=1 {{(pid=81409) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:22:01.738015 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.notifications [None req-8856d457-5f7f-4d29-a8f2-8ec2b546e2ee admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '9ed11ff4bed24e0696d6b8e7c0076d55'} {{(pid=81409) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 13 04:22:01.738168 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f4cc83f1-2181-49ea-a079-57d24313b056 None None] REQUEST_METHOD: `POST` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:22:01.738483 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f4cc83f1-2181-49ea-a079-57d24313b056 None None] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:22:01.738741 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.notifications [None req-8856d457-5f7f-4d29-a8f2-8ec2b546e2ee admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '9ed11ff4bed24e0696d6b8e7c0076d55'} {{(pid=81409) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 13 04:22:01.738894 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f4cc83f1-2181-49ea-a079-57d24313b056 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:22:01.746181 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.notifications [None req-8856d457-5f7f-4d29-a8f2-8ec2b546e2ee admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '9ed11ff4bed24e0696d6b8e7c0076d55'} {{(pid=81409) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 13 04:22:01.748917 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.notifications [None req-8856d457-5f7f-4d29-a8f2-8ec2b546e2ee admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '9ed11ff4bed24e0696d6b8e7c0076d55'} {{(pid=81409) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 13 04:22:01.757692 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.notifications [None req-8856d457-5f7f-4d29-a8f2-8ec2b546e2ee admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '9ed11ff4bed24e0696d6b8e7c0076d55'} {{(pid=81409) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 13 04:22:01.770130 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.auth.core [None req-f4cc83f1-2181-49ea-a079-57d24313b056 None None] MFA Rules not processed for user `ca7301586f634c90a52759f788ef0b46`. Rule list: `[]` (Enabled: `True`). {{(pid=81408) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 13 04:22:01.783616 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.fernet_utils [None req-f4cc83f1-2181-49ea-a079-57d24313b056 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81408) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:22:01.784414 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 869/1709] 10.222.0.27 () {58 vars in 1224 bytes} [Thu Oct 13 04:22:01 2022] DELETE /identity/v3/users/9ed11ff4bed24e0696d6b8e7c0076d55 => generated 0 bytes in 189 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 13 04:22:01.862399 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 841/1710] 10.222.0.27 () {58 vars in 958 bytes} [Thu Oct 13 04:22:01 2022] POST /identity/v3/auth/tokens => generated 3240 bytes in 126 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 13 04:22:01.875174 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8856d457-5f7f-4d29-a8f2-8ec2b546e2ee admin admin] Authenticating user token {{(pid=81409) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:22:01.877130 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.fernet_utils [None req-8856d457-5f7f-4d29-a8f2-8ec2b546e2ee admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81409) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:22:01.947650 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8856d457-5f7f-4d29-a8f2-8ec2b546e2ee admin admin] Validating token access rules against request {{(pid=81409) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:22:01.952016 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-156cbf75-8ec1-4139-82b0-63e83db4e4d6 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6141b8a6c3d54e179d6879bb019f4d12', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dcec0431fb6849499d6aec35a234daa6', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81409) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:22:01.952724 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-156cbf75-8ec1-4139-82b0-63e83db4e4d6 service cinder] REQUEST_METHOD: `GET` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:22:01.953031 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-156cbf75-8ec1-4139-82b0-63e83db4e4d6 service cinder] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:22:01.953921 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-156cbf75-8ec1-4139-82b0-63e83db4e4d6 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:22:01.970734 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-156cbf75-8ec1-4139-82b0-63e83db4e4d6 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:22:01.972417 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-156cbf75-8ec1-4139-82b0-63e83db4e4d6 service cinder] RBAC: Authorization granted {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:22:01.983076 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 870/1711] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Oct 13 04:22:01 2022] GET /identity/v3/auth/tokens => generated 3240 bytes in 108 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 13 04:22:02.095259 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f4cc83f1-2181-49ea-a079-57d24313b056 None None] Authenticating user token {{(pid=81408) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:22:02.096790 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.fernet_utils [None req-f4cc83f1-2181-49ea-a079-57d24313b056 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81408) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:22:02.148529 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG dbcounter [-] [81408] Writing DB stats keystone:INSERT=4,keystone:SELECT=96 {{(pid=81408) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:22:02.180793 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f4cc83f1-2181-49ea-a079-57d24313b056 None None] Validating token access rules against request {{(pid=81408) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:22:02.185468 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4ef0316e-eb5a-4bec-a5fc-55a25ce44a80 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8ba823f313144da3b683bacef8fd62e0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81408) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:22:02.186104 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ef0316e-eb5a-4bec-a5fc-55a25ce44a80 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:22:02.186417 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ef0316e-eb5a-4bec-a5fc-55a25ce44a80 admin admin] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:22:02.186688 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ef0316e-eb5a-4bec-a5fc-55a25ce44a80 admin admin] PATH_INFO: `/v3/projects/06441ef140474903a394a59c0c7695f2` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:22:02.188024 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4ef0316e-eb5a-4bec-a5fc-55a25ce44a80 admin admin] RBAC: Authorizing `identity:delete_project(project_id=06441ef140474903a394a59c0c7695f2)` {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:22:02.190333 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4ef0316e-eb5a-4bec-a5fc-55a25ce44a80 admin admin] RBAC: Authorization granted {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:22:02.242866 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.notifications [None req-4ef0316e-eb5a-4bec-a5fc-55a25ce44a80 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '06441ef140474903a394a59c0c7695f2', 'request_id': 'req-4ef0316e-eb5a-4bec-a5fc-55a25ce44a80'} {{(pid=81408) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 13 04:22:02.251669 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.notifications [None req-4ef0316e-eb5a-4bec-a5fc-55a25ce44a80 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '06441ef140474903a394a59c0c7695f2', 'request_id': 'req-4ef0316e-eb5a-4bec-a5fc-55a25ce44a80'} {{(pid=81408) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 13 04:22:02.259683 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.notifications [None req-4ef0316e-eb5a-4bec-a5fc-55a25ce44a80 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '06441ef140474903a394a59c0c7695f2', 'request_id': 'req-4ef0316e-eb5a-4bec-a5fc-55a25ce44a80'} {{(pid=81408) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 13 04:22:02.280450 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.notifications [None req-4ef0316e-eb5a-4bec-a5fc-55a25ce44a80 admin admin] The token cache is being invalidate because project 06441ef140474903a394a59c0c7695f2 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81408) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 13 04:22:02.281146 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.notifications [None req-4ef0316e-eb5a-4bec-a5fc-55a25ce44a80 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81408) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 13 04:22:02.282316 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.notifications [None req-4ef0316e-eb5a-4bec-a5fc-55a25ce44a80 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81408) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 13 04:22:02.284314 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 842/1712] 10.222.0.27 () {58 vars in 1233 bytes} [Thu Oct 13 04:22:02 2022] DELETE /identity/v3/projects/06441ef140474903a394a59c0c7695f2 => generated 0 bytes in 189 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 13 04:22:02.289083 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f7bc39e-4bc0-4f11-b926-07c70eb3b6eb None None] REQUEST_METHOD: `POST` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:22:02.289359 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f7bc39e-4bc0-4f11-b926-07c70eb3b6eb None None] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:22:02.289630 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f7bc39e-4bc0-4f11-b926-07c70eb3b6eb None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:22:02.327851 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.auth.core [None req-4f7bc39e-4bc0-4f11-b926-07c70eb3b6eb None None] MFA Rules not processed for user `0e2d979563d6421995451a0e86bf3022`. Rule list: `[]` (Enabled: `True`). {{(pid=81409) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 13 04:22:02.330172 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.fernet_utils [None req-4f7bc39e-4bc0-4f11-b926-07c70eb3b6eb None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81409) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:22:02.410597 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 871/1713] 10.222.0.27 () {58 vars in 958 bytes} [Thu Oct 13 04:22:02 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 122 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 13 04:22:02.418841 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8253a243-95da-491f-b2bd-eecd1ae8e3f6 None None] REQUEST_METHOD: `POST` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:22:02.419311 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8253a243-95da-491f-b2bd-eecd1ae8e3f6 None None] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:22:02.419569 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8253a243-95da-491f-b2bd-eecd1ae8e3f6 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:22:02.439843 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.auth.core [None req-8253a243-95da-491f-b2bd-eecd1ae8e3f6 None None] MFA Rules not processed for user `0e2d979563d6421995451a0e86bf3022`. Rule list: `[]` (Enabled: `True`). {{(pid=81408) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 13 04:22:02.441190 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.fernet_utils [None req-8253a243-95da-491f-b2bd-eecd1ae8e3f6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81408) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:22:02.449835 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 843/1714] 10.222.0.27 () {58 vars in 958 bytes} [Thu Oct 13 04:22:02 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 32 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 13 04:22:02.454075 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4f7bc39e-4bc0-4f11-b926-07c70eb3b6eb None None] Authenticating user token {{(pid=81409) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:22:02.463813 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4f7bc39e-4bc0-4f11-b926-07c70eb3b6eb None None] Validating token access rules against request {{(pid=81409) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:22:02.469332 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-68d5c645-2465-4023-8af9-d48e3a355dac admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8ba823f313144da3b683bacef8fd62e0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81409) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:22:02.469941 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68d5c645-2465-4023-8af9-d48e3a355dac admin admin] REQUEST_METHOD: `GET` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:22:02.470224 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68d5c645-2465-4023-8af9-d48e3a355dac admin admin] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:22:02.470522 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68d5c645-2465-4023-8af9-d48e3a355dac admin admin] PATH_INFO: `/v3/domains` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:22:02.471167 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-68d5c645-2465-4023-8af9-d48e3a355dac admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81409) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 13 04:22:02.471463 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-68d5c645-2465-4023-8af9-d48e3a355dac admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:22:02.473048 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-68d5c645-2465-4023-8af9-d48e3a355dac admin admin] RBAC: Authorization granted {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:22:02.484940 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 872/1715] 10.222.0.27 () {58 vars in 1153 bytes} [Thu Oct 13 04:22:02 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 13 04:22:02.489261 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8253a243-95da-491f-b2bd-eecd1ae8e3f6 None None] Authenticating user token {{(pid=81408) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:22:02.498433 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8253a243-95da-491f-b2bd-eecd1ae8e3f6 None None] Validating token access rules against request {{(pid=81408) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:22:02.503028 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c37e1828-5a50-4141-8745-34eb5c02e4be admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8ba823f313144da3b683bacef8fd62e0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81408) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:22:02.503618 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c37e1828-5a50-4141-8745-34eb5c02e4be admin admin] REQUEST_METHOD: `POST` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:22:02.503896 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c37e1828-5a50-4141-8745-34eb5c02e4be admin admin] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:22:02.504161 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c37e1828-5a50-4141-8745-34eb5c02e4be admin admin] PATH_INFO: `/v3/projects` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:22:02.504794 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c37e1828-5a50-4141-8745-34eb5c02e4be admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:22:02.506721 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c37e1828-5a50-4141-8745-34eb5c02e4be admin admin] RBAC: Authorization granted {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:22:02.535014 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 844/1716] 10.222.0.27 () {60 vars in 1153 bytes} [Thu Oct 13 04:22:02 2022] POST /identity/v3/projects => generated 385 bytes in 46 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 13 04:22:02.539135 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-68d5c645-2465-4023-8af9-d48e3a355dac admin admin] Authenticating user token {{(pid=81409) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:22:02.549067 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-68d5c645-2465-4023-8af9-d48e3a355dac admin admin] Validating token access rules against request {{(pid=81409) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:22:02.591473 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG dbcounter [-] [81409] Writing DB stats keystone:SELECT=94,keystone:DELETE=5,keystone:INSERT=1 {{(pid=81409) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:22:02.610734 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cfa7da37-fe06-4a5d-aede-c150a83c511e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8ba823f313144da3b683bacef8fd62e0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81409) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:22:02.611268 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cfa7da37-fe06-4a5d-aede-c150a83c511e admin admin] REQUEST_METHOD: `POST` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:22:02.611540 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cfa7da37-fe06-4a5d-aede-c150a83c511e admin admin] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:22:02.611808 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cfa7da37-fe06-4a5d-aede-c150a83c511e admin admin] PATH_INFO: `/v3/users` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:22:02.612434 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cfa7da37-fe06-4a5d-aede-c150a83c511e admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:22:02.614416 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cfa7da37-fe06-4a5d-aede-c150a83c511e admin admin] RBAC: Authorization granted {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:22:02.635497 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 873/1717] 10.222.0.27 () {60 vars in 1144 bytes} [Thu Oct 13 04:22:02 2022] POST /identity/v3/users => generated 293 bytes in 97 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 13 04:22:02.640529 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c37e1828-5a50-4141-8745-34eb5c02e4be admin admin] Authenticating user token {{(pid=81408) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:22:02.650735 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c37e1828-5a50-4141-8745-34eb5c02e4be admin admin] Validating token access rules against request {{(pid=81408) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:22:02.655451 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3fe0a7fa-3874-49e1-97ef-c16337742410 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8ba823f313144da3b683bacef8fd62e0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81408) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:22:02.656045 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3fe0a7fa-3874-49e1-97ef-c16337742410 admin admin] REQUEST_METHOD: `GET` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:22:02.656325 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3fe0a7fa-3874-49e1-97ef-c16337742410 admin admin] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:22:02.656596 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3fe0a7fa-3874-49e1-97ef-c16337742410 admin admin] PATH_INFO: `/v3/roles` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:22:02.657237 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3fe0a7fa-3874-49e1-97ef-c16337742410 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:22:02.658912 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3fe0a7fa-3874-49e1-97ef-c16337742410 admin admin] RBAC: Authorization granted {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:22:02.672348 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 845/1718] 10.222.0.27 () {58 vars in 1122 bytes} [Thu Oct 13 04:22:02 2022] GET /identity/v3/roles => generated 1420 bytes in 29 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 13 04:22:02.673428 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cfa7da37-fe06-4a5d-aede-c150a83c511e admin admin] Authenticating user token {{(pid=81409) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:22:02.684888 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cfa7da37-fe06-4a5d-aede-c150a83c511e admin admin] Validating token access rules against request {{(pid=81409) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:22:02.689683 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8a1d0e59-fd65-4acf-85aa-79f64dfde5f7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8ba823f313144da3b683bacef8fd62e0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81409) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:22:02.690317 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a1d0e59-fd65-4acf-85aa-79f64dfde5f7 admin admin] REQUEST_METHOD: `GET` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:22:02.690590 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a1d0e59-fd65-4acf-85aa-79f64dfde5f7 admin admin] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:22:02.690887 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a1d0e59-fd65-4acf-85aa-79f64dfde5f7 admin admin] PATH_INFO: `/v3/roles` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:22:02.691519 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8a1d0e59-fd65-4acf-85aa-79f64dfde5f7 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:22:02.693119 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8a1d0e59-fd65-4acf-85aa-79f64dfde5f7 admin admin] RBAC: Authorization granted {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:22:02.704216 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 874/1719] 10.222.0.27 () {58 vars in 1122 bytes} [Thu Oct 13 04:22:02 2022] GET /identity/v3/roles => generated 1420 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Oct 13 04:22:02.708402 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3fe0a7fa-3874-49e1-97ef-c16337742410 admin admin] Authenticating user token {{(pid=81408) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:22:02.718414 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3fe0a7fa-3874-49e1-97ef-c16337742410 admin admin] Validating token access rules against request {{(pid=81408) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:22:02.722934 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6cf7a7cd-11f3-448f-b825-30c471abfdcb admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8ba823f313144da3b683bacef8fd62e0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81408) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:22:02.723574 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6cf7a7cd-11f3-448f-b825-30c471abfdcb admin admin] REQUEST_METHOD: `PUT` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:22:02.723849 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6cf7a7cd-11f3-448f-b825-30c471abfdcb admin admin] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:22:02.724121 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6cf7a7cd-11f3-448f-b825-30c471abfdcb admin admin] PATH_INFO: `/v3/projects/e70893eca76a4c329d4b96e94f9aea74/users/5c842b046969424894edfde6946c23d6/roles/cc56c4ae13a7424ea771c15da4b153c1` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:22:02.738610 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6cf7a7cd-11f3-448f-b825-30c471abfdcb admin admin] RBAC: Authorizing `identity:create_grant(project_id=e70893eca76a4c329d4b96e94f9aea74, user_id=5c842b046969424894edfde6946c23d6, role_id=cc56c4ae13a7424ea771c15da4b153c1)` {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:22:02.741246 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6cf7a7cd-11f3-448f-b825-30c471abfdcb admin admin] RBAC: Authorization granted {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:22:02.755473 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 846/1720] 10.222.0.27 () {60 vars in 1483 bytes} [Thu Oct 13 04:22:02 2022] PUT /identity/v3/projects/e70893eca76a4c329d4b96e94f9aea74/users/5c842b046969424894edfde6946c23d6/roles/cc56c4ae13a7424ea771c15da4b153c1 => generated 0 bytes in 48 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 13 04:22:02.766754 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1ab31bd-2d44-43a9-aadc-53071f2552f4 None None] REQUEST_METHOD: `POST` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:22:02.767040 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1ab31bd-2d44-43a9-aadc-53071f2552f4 None None] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:22:02.767312 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1ab31bd-2d44-43a9-aadc-53071f2552f4 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:22:02.795776 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.auth.core [None req-a1ab31bd-2d44-43a9-aadc-53071f2552f4 None None] MFA Rules not processed for user `5c842b046969424894edfde6946c23d6`. Rule list: `[]` (Enabled: `True`). {{(pid=81409) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 13 04:22:02.796994 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.fernet_utils [None req-a1ab31bd-2d44-43a9-aadc-53071f2552f4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81409) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:22:02.861220 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 875/1721] 10.222.0.27 () {58 vars in 958 bytes} [Thu Oct 13 04:22:02 2022] POST /identity/v3/auth/tokens => generated 3262 bytes in 95 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 13 04:22:02.865857 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6cf7a7cd-11f3-448f-b825-30c471abfdcb admin admin] Authenticating user token {{(pid=81408) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:22:02.931838 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6cf7a7cd-11f3-448f-b825-30c471abfdcb admin admin] Validating token access rules against request {{(pid=81408) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:22:02.937193 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5de45122-20cb-4d07-9ff3-88e5a49ee85f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8ba823f313144da3b683bacef8fd62e0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81408) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:22:02.937792 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5de45122-20cb-4d07-9ff3-88e5a49ee85f admin admin] REQUEST_METHOD: `POST` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:22:02.938071 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5de45122-20cb-4d07-9ff3-88e5a49ee85f admin admin] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:22:02.938382 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5de45122-20cb-4d07-9ff3-88e5a49ee85f admin admin] PATH_INFO: `/v3/projects` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:22:02.939024 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5de45122-20cb-4d07-9ff3-88e5a49ee85f admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:22:02.940975 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5de45122-20cb-4d07-9ff3-88e5a49ee85f admin admin] RBAC: Authorization granted {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:22:02.951748 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG dbcounter [-] [81408] Writing DB stats keystone:SELECT=90,keystone:DELETE=7,keystone:INSERT=3 {{(pid=81408) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:22:02.969720 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 847/1722] 10.222.0.27 () {60 vars in 1153 bytes} [Thu Oct 13 04:22:02 2022] POST /identity/v3/projects => generated 385 bytes in 104 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 13 04:22:02.973752 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a1ab31bd-2d44-43a9-aadc-53071f2552f4 None None] Authenticating user token {{(pid=81409) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:22:02.985500 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a1ab31bd-2d44-43a9-aadc-53071f2552f4 None None] Validating token access rules against request {{(pid=81409) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:22:03.043407 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-52cbe428-74a3-468b-a4e5-0bd5323997e1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8ba823f313144da3b683bacef8fd62e0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81409) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:22:03.043999 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52cbe428-74a3-468b-a4e5-0bd5323997e1 admin admin] REQUEST_METHOD: `POST` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:22:03.044278 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52cbe428-74a3-468b-a4e5-0bd5323997e1 admin admin] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:22:03.044548 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52cbe428-74a3-468b-a4e5-0bd5323997e1 admin admin] PATH_INFO: `/v3/users` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:22:03.045185 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-52cbe428-74a3-468b-a4e5-0bd5323997e1 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:22:03.047123 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-52cbe428-74a3-468b-a4e5-0bd5323997e1 admin admin] RBAC: Authorization granted {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:22:03.072291 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 876/1723] 10.222.0.27 () {60 vars in 1144 bytes} [Thu Oct 13 04:22:02 2022] POST /identity/v3/users => generated 293 bytes in 96 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 13 04:22:03.075107 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5de45122-20cb-4d07-9ff3-88e5a49ee85f admin admin] Authenticating user token {{(pid=81408) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:22:03.085996 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5de45122-20cb-4d07-9ff3-88e5a49ee85f admin admin] Validating token access rules against request {{(pid=81408) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:22:03.092026 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-92df0fd5-5463-4f5d-b6c6-dd9e868d8dec admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8ba823f313144da3b683bacef8fd62e0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81408) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:22:03.092678 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92df0fd5-5463-4f5d-b6c6-dd9e868d8dec admin admin] REQUEST_METHOD: `GET` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:22:03.092949 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92df0fd5-5463-4f5d-b6c6-dd9e868d8dec admin admin] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:22:03.093227 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92df0fd5-5463-4f5d-b6c6-dd9e868d8dec admin admin] PATH_INFO: `/v3/roles` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:22:03.093853 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-92df0fd5-5463-4f5d-b6c6-dd9e868d8dec admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:22:03.095477 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-92df0fd5-5463-4f5d-b6c6-dd9e868d8dec admin admin] RBAC: Authorization granted {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:22:03.106838 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 848/1724] 10.222.0.27 () {58 vars in 1122 bytes} [Thu Oct 13 04:22:03 2022] GET /identity/v3/roles => generated 1420 bytes in 33 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 13 04:22:03.111011 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-52cbe428-74a3-468b-a4e5-0bd5323997e1 admin admin] Authenticating user token {{(pid=81409) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:22:03.132419 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-52cbe428-74a3-468b-a4e5-0bd5323997e1 admin admin] Validating token access rules against request {{(pid=81409) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:22:03.137705 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-35e80a1d-f1d5-42ba-b645-726d40bd21af admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8ba823f313144da3b683bacef8fd62e0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81409) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:22:03.138366 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35e80a1d-f1d5-42ba-b645-726d40bd21af admin admin] REQUEST_METHOD: `GET` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:22:03.138644 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35e80a1d-f1d5-42ba-b645-726d40bd21af admin admin] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:22:03.138919 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35e80a1d-f1d5-42ba-b645-726d40bd21af admin admin] PATH_INFO: `/v3/roles` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:22:03.139587 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-35e80a1d-f1d5-42ba-b645-726d40bd21af admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:22:03.141153 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-35e80a1d-f1d5-42ba-b645-726d40bd21af admin admin] RBAC: Authorization granted {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:22:03.151353 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 877/1725] 10.222.0.27 () {58 vars in 1122 bytes} [Thu Oct 13 04:22:03 2022] GET /identity/v3/roles => generated 1420 bytes in 41 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 13 04:22:03.156237 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-92df0fd5-5463-4f5d-b6c6-dd9e868d8dec admin admin] Authenticating user token {{(pid=81408) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:22:03.170023 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-92df0fd5-5463-4f5d-b6c6-dd9e868d8dec admin admin] Validating token access rules against request {{(pid=81408) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:22:03.175501 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c25d983c-c1b2-4149-a7c5-a289f03f1b90 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8ba823f313144da3b683bacef8fd62e0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81408) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:22:03.176323 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c25d983c-c1b2-4149-a7c5-a289f03f1b90 admin admin] REQUEST_METHOD: `PUT` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:22:03.176700 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c25d983c-c1b2-4149-a7c5-a289f03f1b90 admin admin] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:22:03.177067 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c25d983c-c1b2-4149-a7c5-a289f03f1b90 admin admin] PATH_INFO: `/v3/projects/3a1b5dae4dd34f449427067c79f8c21b/users/580b04d7a005447eac1034f05bec4f74/roles/cc56c4ae13a7424ea771c15da4b153c1` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:22:03.200643 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c25d983c-c1b2-4149-a7c5-a289f03f1b90 admin admin] RBAC: Authorizing `identity:create_grant(project_id=3a1b5dae4dd34f449427067c79f8c21b, user_id=580b04d7a005447eac1034f05bec4f74, role_id=cc56c4ae13a7424ea771c15da4b153c1)` {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:22:03.202241 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c25d983c-c1b2-4149-a7c5-a289f03f1b90 admin admin] RBAC: Authorization granted {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:22:03.217642 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 849/1726] 10.222.0.27 () {60 vars in 1483 bytes} [Thu Oct 13 04:22:03 2022] PUT /identity/v3/projects/3a1b5dae4dd34f449427067c79f8c21b/users/580b04d7a005447eac1034f05bec4f74/roles/cc56c4ae13a7424ea771c15da4b153c1 => generated 0 bytes in 62 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 13 04:22:03.229504 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa913452-e09b-4530-b91f-4178ac209005 None None] REQUEST_METHOD: `POST` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:22:03.229787 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa913452-e09b-4530-b91f-4178ac209005 None None] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:22:03.230054 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa913452-e09b-4530-b91f-4178ac209005 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:22:03.262989 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.auth.core [None req-fa913452-e09b-4530-b91f-4178ac209005 None None] MFA Rules not processed for user `580b04d7a005447eac1034f05bec4f74`. Rule list: `[]` (Enabled: `True`). {{(pid=81409) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 13 04:22:03.264293 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.fernet_utils [None req-fa913452-e09b-4530-b91f-4178ac209005 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81409) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:22:03.281498 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG dbcounter [-] [81409] Writing DB stats keystone:SELECT=94,keystone:INSERT=6 {{(pid=81409) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:22:03.329481 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 878/1727] 10.222.0.27 () {58 vars in 958 bytes} [Thu Oct 13 04:22:03 2022] POST /identity/v3/auth/tokens => generated 3262 bytes in 101 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 13 04:22:03.333974 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c25d983c-c1b2-4149-a7c5-a289f03f1b90 admin admin] Authenticating user token {{(pid=81408) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:22:03.410425 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c25d983c-c1b2-4149-a7c5-a289f03f1b90 admin admin] Validating token access rules against request {{(pid=81408) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:22:03.416347 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b1c84355-de1f-4fae-bbd2-9a2fa95733ae admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8ba823f313144da3b683bacef8fd62e0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81408) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:22:03.417251 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1c84355-de1f-4fae-bbd2-9a2fa95733ae admin admin] REQUEST_METHOD: `POST` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:22:03.417880 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1c84355-de1f-4fae-bbd2-9a2fa95733ae admin admin] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:22:03.418962 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1c84355-de1f-4fae-bbd2-9a2fa95733ae admin admin] PATH_INFO: `/v3/projects` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:22:03.419734 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b1c84355-de1f-4fae-bbd2-9a2fa95733ae admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:22:03.421974 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b1c84355-de1f-4fae-bbd2-9a2fa95733ae admin admin] RBAC: Authorization granted {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:22:03.450252 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 850/1728] 10.222.0.27 () {60 vars in 1153 bytes} [Thu Oct 13 04:22:03 2022] POST /identity/v3/projects => generated 383 bytes in 117 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 13 04:22:03.455164 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fa913452-e09b-4530-b91f-4178ac209005 None None] Authenticating user token {{(pid=81409) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:22:03.474575 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fa913452-e09b-4530-b91f-4178ac209005 None None] Validating token access rules against request {{(pid=81409) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:22:03.550082 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6b650209-d3b2-4a36-87ea-62e25b1ac299 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8ba823f313144da3b683bacef8fd62e0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81409) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:22:03.550832 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b650209-d3b2-4a36-87ea-62e25b1ac299 admin admin] REQUEST_METHOD: `POST` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:22:03.551111 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b650209-d3b2-4a36-87ea-62e25b1ac299 admin admin] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:22:03.551380 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b650209-d3b2-4a36-87ea-62e25b1ac299 admin admin] PATH_INFO: `/v3/users` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:22:03.552009 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6b650209-d3b2-4a36-87ea-62e25b1ac299 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:22:03.553944 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6b650209-d3b2-4a36-87ea-62e25b1ac299 admin admin] RBAC: Authorization granted {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:22:03.580690 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 879/1729] 10.222.0.27 () {60 vars in 1144 bytes} [Thu Oct 13 04:22:03 2022] POST /identity/v3/users => generated 298 bytes in 127 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 13 04:22:03.584515 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b1c84355-de1f-4fae-bbd2-9a2fa95733ae admin admin] Authenticating user token {{(pid=81408) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:22:03.597235 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b1c84355-de1f-4fae-bbd2-9a2fa95733ae admin admin] Validating token access rules against request {{(pid=81408) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:22:03.605258 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ed678e16-1e39-4bc7-9eab-af34ce373640 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8ba823f313144da3b683bacef8fd62e0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81408) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:22:03.605258 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed678e16-1e39-4bc7-9eab-af34ce373640 admin admin] REQUEST_METHOD: `GET` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:22:03.605258 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed678e16-1e39-4bc7-9eab-af34ce373640 admin admin] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:22:03.605258 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed678e16-1e39-4bc7-9eab-af34ce373640 admin admin] PATH_INFO: `/v3/roles` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:22:03.606512 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ed678e16-1e39-4bc7-9eab-af34ce373640 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:22:03.606512 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ed678e16-1e39-4bc7-9eab-af34ce373640 admin admin] RBAC: Authorization granted {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:22:03.617185 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 851/1730] 10.222.0.27 () {58 vars in 1122 bytes} [Thu Oct 13 04:22:03 2022] GET /identity/v3/roles => generated 1420 bytes in 33 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 13 04:22:03.621269 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6b650209-d3b2-4a36-87ea-62e25b1ac299 admin admin] Authenticating user token {{(pid=81409) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:22:03.631212 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6b650209-d3b2-4a36-87ea-62e25b1ac299 admin admin] Validating token access rules against request {{(pid=81409) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:22:03.636000 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-05d68585-c470-419c-9369-5eb483d78f79 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8ba823f313144da3b683bacef8fd62e0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81409) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:22:03.636589 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05d68585-c470-419c-9369-5eb483d78f79 admin admin] REQUEST_METHOD: `PUT` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:22:03.636897 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05d68585-c470-419c-9369-5eb483d78f79 admin admin] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:22:03.637161 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05d68585-c470-419c-9369-5eb483d78f79 admin admin] PATH_INFO: `/v3/projects/1a161c93104e43a8bfdbb6e97a3ebf1f/users/3600ed6e624246f49b6ee53b9de89ddc/roles/ef55fbd94e1040be83695754075a7ed5` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:22:03.653353 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-05d68585-c470-419c-9369-5eb483d78f79 admin admin] RBAC: Authorizing `identity:create_grant(project_id=1a161c93104e43a8bfdbb6e97a3ebf1f, user_id=3600ed6e624246f49b6ee53b9de89ddc, role_id=ef55fbd94e1040be83695754075a7ed5)` {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:22:03.657032 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-05d68585-c470-419c-9369-5eb483d78f79 admin admin] RBAC: Authorization granted {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:22:03.672820 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 880/1731] 10.222.0.27 () {60 vars in 1483 bytes} [Thu Oct 13 04:22:03 2022] PUT /identity/v3/projects/1a161c93104e43a8bfdbb6e97a3ebf1f/users/3600ed6e624246f49b6ee53b9de89ddc/roles/ef55fbd94e1040be83695754075a7ed5 => generated 0 bytes in 52 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 13 04:22:03.676872 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ed678e16-1e39-4bc7-9eab-af34ce373640 admin admin] Authenticating user token {{(pid=81408) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:22:03.689366 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ed678e16-1e39-4bc7-9eab-af34ce373640 admin admin] Validating token access rules against request {{(pid=81408) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:22:03.753343 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-415e74eb-0d6e-4906-a8c7-ece7239a0b3c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8ba823f313144da3b683bacef8fd62e0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81408) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:22:03.753950 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-415e74eb-0d6e-4906-a8c7-ece7239a0b3c admin admin] REQUEST_METHOD: `GET` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:22:03.754251 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-415e74eb-0d6e-4906-a8c7-ece7239a0b3c admin admin] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:22:03.754526 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-415e74eb-0d6e-4906-a8c7-ece7239a0b3c admin admin] PATH_INFO: `/v3/roles` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:22:03.755158 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-415e74eb-0d6e-4906-a8c7-ece7239a0b3c admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:22:03.757082 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-415e74eb-0d6e-4906-a8c7-ece7239a0b3c admin admin] RBAC: Authorization granted {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:22:03.767501 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 852/1732] 10.222.0.27 () {58 vars in 1122 bytes} [Thu Oct 13 04:22:03 2022] GET /identity/v3/roles => generated 1420 bytes in 91 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 13 04:22:03.771638 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-05d68585-c470-419c-9369-5eb483d78f79 admin admin] Authenticating user token {{(pid=81409) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:22:03.786452 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-05d68585-c470-419c-9369-5eb483d78f79 admin admin] Validating token access rules against request {{(pid=81409) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:22:03.791706 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d46feb3a-74b9-4cf0-bc33-c9558f2a37fe admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8ba823f313144da3b683bacef8fd62e0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81409) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:22:03.793024 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d46feb3a-74b9-4cf0-bc33-c9558f2a37fe admin admin] REQUEST_METHOD: `PUT` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:22:03.793448 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d46feb3a-74b9-4cf0-bc33-c9558f2a37fe admin admin] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:22:03.793733 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d46feb3a-74b9-4cf0-bc33-c9558f2a37fe admin admin] PATH_INFO: `/v3/projects/1a161c93104e43a8bfdbb6e97a3ebf1f/users/3600ed6e624246f49b6ee53b9de89ddc/roles/ef55fbd94e1040be83695754075a7ed5` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:22:03.796310 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d46feb3a-74b9-4cf0-bc33-c9558f2a37fe admin admin] RBAC: Authorizing `identity:create_grant(project_id=1a161c93104e43a8bfdbb6e97a3ebf1f, user_id=3600ed6e624246f49b6ee53b9de89ddc, role_id=ef55fbd94e1040be83695754075a7ed5)` {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:22:03.799035 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d46feb3a-74b9-4cf0-bc33-c9558f2a37fe admin admin] RBAC: Authorization granted {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:22:03.810764 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 881/1733] 10.222.0.27 () {60 vars in 1483 bytes} [Thu Oct 13 04:22:03 2022] PUT /identity/v3/projects/1a161c93104e43a8bfdbb6e97a3ebf1f/users/3600ed6e624246f49b6ee53b9de89ddc/roles/ef55fbd94e1040be83695754075a7ed5 => generated 0 bytes in 40 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 13 04:22:03.821202 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-409ce529-f7b0-4ed9-9118-81c9b3ee7482 None None] REQUEST_METHOD: `POST` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:22:03.821487 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-409ce529-f7b0-4ed9-9118-81c9b3ee7482 None None] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:22:03.821759 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-409ce529-f7b0-4ed9-9118-81c9b3ee7482 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:22:03.853681 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.auth.core [None req-409ce529-f7b0-4ed9-9118-81c9b3ee7482 None None] MFA Rules not processed for user `3600ed6e624246f49b6ee53b9de89ddc`. Rule list: `[]` (Enabled: `True`). {{(pid=81408) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 13 04:22:03.855151 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.fernet_utils [None req-409ce529-f7b0-4ed9-9118-81c9b3ee7482 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81408) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:22:03.870154 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG dbcounter [-] [81408] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81408) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:22:03.953009 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 853/1734] 10.222.0.27 () {58 vars in 958 bytes} [Thu Oct 13 04:22:03 2022] POST /identity/v3/auth/tokens => generated 3327 bytes in 133 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 13 04:22:03.964742 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d46feb3a-74b9-4cf0-bc33-c9558f2a37fe admin admin] Authenticating user token {{(pid=81409) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:22:03.967441 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.fernet_utils [None req-d46feb3a-74b9-4cf0-bc33-c9558f2a37fe admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81409) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:22:04.029514 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d46feb3a-74b9-4cf0-bc33-c9558f2a37fe admin admin] Validating token access rules against request {{(pid=81409) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:22:04.033266 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e202394-440c-4ae0-b311-b455c32ddef4 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6141b8a6c3d54e179d6879bb019f4d12', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dcec0431fb6849499d6aec35a234daa6', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81409) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:22:04.033871 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e202394-440c-4ae0-b311-b455c32ddef4 service cinder] REQUEST_METHOD: `GET` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:22:04.034151 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e202394-440c-4ae0-b311-b455c32ddef4 service cinder] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:22:04.034456 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e202394-440c-4ae0-b311-b455c32ddef4 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:22:04.060728 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG dbcounter [-] [81409] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=81409) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:22:04.098932 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1e202394-440c-4ae0-b311-b455c32ddef4 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:22:04.100359 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1e202394-440c-4ae0-b311-b455c32ddef4 service cinder] RBAC: Authorization granted {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:22:04.109867 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 882/1735] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Oct 13 04:22:03 2022] GET /identity/v3/auth/tokens => generated 3262 bytes in 146 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 13 04:22:05.709071 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-409ce529-f7b0-4ed9-9118-81c9b3ee7482 None None] Authenticating user token {{(pid=81408) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:22:05.722073 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-409ce529-f7b0-4ed9-9118-81c9b3ee7482 None None] Validating token access rules against request {{(pid=81408) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:22:05.726981 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f9e09c68-13d4-4d74-be75-e4d465295360 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6141b8a6c3d54e179d6879bb019f4d12', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dcec0431fb6849499d6aec35a234daa6', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81408) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:22:05.727599 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9e09c68-13d4-4d74-be75-e4d465295360 service cinder] REQUEST_METHOD: `GET` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:22:05.727874 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9e09c68-13d4-4d74-be75-e4d465295360 service cinder] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:22:05.728199 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9e09c68-13d4-4d74-be75-e4d465295360 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:22:05.796040 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f9e09c68-13d4-4d74-be75-e4d465295360 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:22:05.797480 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f9e09c68-13d4-4d74-be75-e4d465295360 service cinder] RBAC: Authorization granted {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:22:05.806524 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 854/1736] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Oct 13 04:22:05 2022] GET /identity/v3/auth/tokens => generated 3262 bytes in 98 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Oct 13 04:22:06.401897 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e202394-440c-4ae0-b311-b455c32ddef4 service cinder] Authenticating user token {{(pid=81409) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:22:06.403999 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.fernet_utils [None req-1e202394-440c-4ae0-b311-b455c32ddef4 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81409) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:22:06.415336 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e202394-440c-4ae0-b311-b455c32ddef4 service cinder] Validating token access rules against request {{(pid=81409) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:22:06.419637 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a0619ff9-c9e7-404d-af89-6d2fdff7430d service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6141b8a6c3d54e179d6879bb019f4d12', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dcec0431fb6849499d6aec35a234daa6', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81409) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:22:06.420253 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0619ff9-c9e7-404d-af89-6d2fdff7430d service cinder] REQUEST_METHOD: `GET` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:22:06.420533 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0619ff9-c9e7-404d-af89-6d2fdff7430d service cinder] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:22:06.420803 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0619ff9-c9e7-404d-af89-6d2fdff7430d service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:22:06.431756 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a0619ff9-c9e7-404d-af89-6d2fdff7430d service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:22:06.432714 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a0619ff9-c9e7-404d-af89-6d2fdff7430d service cinder] RBAC: Authorization granted {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:22:06.442723 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 883/1737] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Oct 13 04:22:06 2022] GET /identity/v3/auth/tokens => generated 3327 bytes in 41 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 13 04:22:13.605942 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f9e09c68-13d4-4d74-be75-e4d465295360 service cinder] Authenticating user token {{(pid=81408) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:22:13.674423 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f9e09c68-13d4-4d74-be75-e4d465295360 service cinder] Validating token access rules against request {{(pid=81408) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:22:13.679271 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-48c73a3f-1ae1-40c0-a43e-7e6669cb3002 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8ba823f313144da3b683bacef8fd62e0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81408) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:22:13.679897 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48c73a3f-1ae1-40c0-a43e-7e6669cb3002 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:22:13.680172 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48c73a3f-1ae1-40c0-a43e-7e6669cb3002 admin admin] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:22:13.680440 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48c73a3f-1ae1-40c0-a43e-7e6669cb3002 admin admin] PATH_INFO: `/v3/users/5c842b046969424894edfde6946c23d6` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:22:13.681567 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-48c73a3f-1ae1-40c0-a43e-7e6669cb3002 admin admin] RBAC: Authorizing `identity:delete_user(user_id=5c842b046969424894edfde6946c23d6)` {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:22:13.683532 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-48c73a3f-1ae1-40c0-a43e-7e6669cb3002 admin admin] RBAC: Authorization granted {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:22:13.733296 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.notifications [None req-48c73a3f-1ae1-40c0-a43e-7e6669cb3002 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '5c842b046969424894edfde6946c23d6'} {{(pid=81408) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 13 04:22:13.737645 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG dbcounter [-] [81408] Writing DB stats keystone:SELECT=93,keystone:DELETE=7 {{(pid=81408) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:22:13.740962 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.notifications [None req-48c73a3f-1ae1-40c0-a43e-7e6669cb3002 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '5c842b046969424894edfde6946c23d6'} {{(pid=81408) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 13 04:22:13.742261 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.notifications [None req-48c73a3f-1ae1-40c0-a43e-7e6669cb3002 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '5c842b046969424894edfde6946c23d6'} {{(pid=81408) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 13 04:22:13.742984 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.notifications [None req-48c73a3f-1ae1-40c0-a43e-7e6669cb3002 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '5c842b046969424894edfde6946c23d6'} {{(pid=81408) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 13 04:22:13.753124 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.notifications [None req-48c73a3f-1ae1-40c0-a43e-7e6669cb3002 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '5c842b046969424894edfde6946c23d6'} {{(pid=81408) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 13 04:22:13.770735 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 855/1738] 10.222.0.27 () {58 vars in 1224 bytes} [Thu Oct 13 04:22:13 2022] DELETE /identity/v3/users/5c842b046969424894edfde6946c23d6 => generated 0 bytes in 166 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 13 04:22:13.775273 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b94ef4ac-c6aa-407f-aadf-0516d3faa8a5 None None] REQUEST_METHOD: `POST` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:22:13.775811 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b94ef4ac-c6aa-407f-aadf-0516d3faa8a5 None None] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:22:13.775915 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b94ef4ac-c6aa-407f-aadf-0516d3faa8a5 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:22:13.799395 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.auth.core [None req-b94ef4ac-c6aa-407f-aadf-0516d3faa8a5 None None] MFA Rules not processed for user `0e2d979563d6421995451a0e86bf3022`. Rule list: `[]` (Enabled: `True`). {{(pid=81409) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 13 04:22:13.800635 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.fernet_utils [None req-b94ef4ac-c6aa-407f-aadf-0516d3faa8a5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81409) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:22:13.870145 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 884/1739] 10.222.0.27 () {58 vars in 958 bytes} [Thu Oct 13 04:22:13 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 96 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 13 04:22:13.880899 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-48c73a3f-1ae1-40c0-a43e-7e6669cb3002 admin admin] Authenticating user token {{(pid=81408) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:22:13.882540 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.fernet_utils [None req-48c73a3f-1ae1-40c0-a43e-7e6669cb3002 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81408) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:22:13.954380 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-48c73a3f-1ae1-40c0-a43e-7e6669cb3002 admin admin] Validating token access rules against request {{(pid=81408) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:22:13.958104 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-56a186fe-cff4-4286-bd7c-c3efda43258a service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd8812c3974f84871bb2d761cf0016257', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dcec0431fb6849499d6aec35a234daa6', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81408) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:22:13.958740 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56a186fe-cff4-4286-bd7c-c3efda43258a service neutron] REQUEST_METHOD: `GET` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:22:13.959021 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56a186fe-cff4-4286-bd7c-c3efda43258a service neutron] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:22:13.959336 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56a186fe-cff4-4286-bd7c-c3efda43258a service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:22:13.973535 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-56a186fe-cff4-4286-bd7c-c3efda43258a service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:22:13.975247 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-56a186fe-cff4-4286-bd7c-c3efda43258a service neutron] RBAC: Authorization granted {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:22:13.993522 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 856/1740] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Oct 13 04:22:13 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 113 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 13 04:22:14.307682 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b94ef4ac-c6aa-407f-aadf-0516d3faa8a5 None None] Authenticating user token {{(pid=81409) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:22:14.309116 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.fernet_utils [None req-b94ef4ac-c6aa-407f-aadf-0516d3faa8a5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81409) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:22:14.321999 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b94ef4ac-c6aa-407f-aadf-0516d3faa8a5 None None] Validating token access rules against request {{(pid=81409) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:22:14.328607 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-82bacab0-685b-4f57-a45d-12721c67a385 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8ba823f313144da3b683bacef8fd62e0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81409) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:22:14.330290 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82bacab0-685b-4f57-a45d-12721c67a385 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:22:14.330572 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82bacab0-685b-4f57-a45d-12721c67a385 admin admin] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:22:14.330838 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82bacab0-685b-4f57-a45d-12721c67a385 admin admin] PATH_INFO: `/v3/projects/e70893eca76a4c329d4b96e94f9aea74` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:22:14.331993 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-82bacab0-685b-4f57-a45d-12721c67a385 admin admin] RBAC: Authorizing `identity:delete_project(project_id=e70893eca76a4c329d4b96e94f9aea74)` {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:22:14.333982 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-82bacab0-685b-4f57-a45d-12721c67a385 admin admin] RBAC: Authorization granted {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:22:14.387095 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.notifications [None req-82bacab0-685b-4f57-a45d-12721c67a385 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'e70893eca76a4c329d4b96e94f9aea74', 'request_id': 'req-82bacab0-685b-4f57-a45d-12721c67a385'} {{(pid=81409) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 13 04:22:14.394551 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.notifications [None req-82bacab0-685b-4f57-a45d-12721c67a385 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'e70893eca76a4c329d4b96e94f9aea74', 'request_id': 'req-82bacab0-685b-4f57-a45d-12721c67a385'} {{(pid=81409) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 13 04:22:14.400981 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.notifications [None req-82bacab0-685b-4f57-a45d-12721c67a385 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'e70893eca76a4c329d4b96e94f9aea74', 'request_id': 'req-82bacab0-685b-4f57-a45d-12721c67a385'} {{(pid=81409) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 13 04:22:14.422488 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.notifications [None req-82bacab0-685b-4f57-a45d-12721c67a385 admin admin] The token cache is being invalidate because project e70893eca76a4c329d4b96e94f9aea74 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81409) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 13 04:22:14.422802 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.notifications [None req-82bacab0-685b-4f57-a45d-12721c67a385 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81409) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 13 04:22:14.423463 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.notifications [None req-82bacab0-685b-4f57-a45d-12721c67a385 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81409) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 13 04:22:14.425076 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 885/1741] 10.222.0.27 () {58 vars in 1233 bytes} [Thu Oct 13 04:22:14 2022] DELETE /identity/v3/projects/e70893eca76a4c329d4b96e94f9aea74 => generated 0 bytes in 118 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 13 04:22:14.429091 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-56a186fe-cff4-4286-bd7c-c3efda43258a service neutron] Authenticating user token {{(pid=81408) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:22:14.439387 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-56a186fe-cff4-4286-bd7c-c3efda43258a service neutron] Validating token access rules against request {{(pid=81408) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:22:14.509338 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4468ea18-e744-46db-8c0c-d3a5f6f4c996 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8ba823f313144da3b683bacef8fd62e0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81408) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:22:14.510188 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4468ea18-e744-46db-8c0c-d3a5f6f4c996 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:22:14.510631 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4468ea18-e744-46db-8c0c-d3a5f6f4c996 admin admin] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:22:14.511057 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4468ea18-e744-46db-8c0c-d3a5f6f4c996 admin admin] PATH_INFO: `/v3/users/580b04d7a005447eac1034f05bec4f74` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:22:14.512217 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4468ea18-e744-46db-8c0c-d3a5f6f4c996 admin admin] RBAC: Authorizing `identity:delete_user(user_id=580b04d7a005447eac1034f05bec4f74)` {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:22:14.514640 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4468ea18-e744-46db-8c0c-d3a5f6f4c996 admin admin] RBAC: Authorization granted {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:22:14.576557 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.notifications [None req-4468ea18-e744-46db-8c0c-d3a5f6f4c996 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '580b04d7a005447eac1034f05bec4f74'} {{(pid=81408) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 13 04:22:14.584369 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.notifications [None req-4468ea18-e744-46db-8c0c-d3a5f6f4c996 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '580b04d7a005447eac1034f05bec4f74'} {{(pid=81408) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 13 04:22:14.585066 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.notifications [None req-4468ea18-e744-46db-8c0c-d3a5f6f4c996 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '580b04d7a005447eac1034f05bec4f74'} {{(pid=81408) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 13 04:22:14.585669 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.notifications [None req-4468ea18-e744-46db-8c0c-d3a5f6f4c996 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '580b04d7a005447eac1034f05bec4f74'} {{(pid=81408) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 13 04:22:14.594129 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.notifications [None req-4468ea18-e744-46db-8c0c-d3a5f6f4c996 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '580b04d7a005447eac1034f05bec4f74'} {{(pid=81408) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 13 04:22:14.607879 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG dbcounter [-] [81408] Writing DB stats keystone:SELECT=86,keystone:INSERT=2,keystone:DELETE=12 {{(pid=81408) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:22:14.629465 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 857/1742] 10.222.0.27 () {58 vars in 1224 bytes} [Thu Oct 13 04:22:14 2022] DELETE /identity/v3/users/580b04d7a005447eac1034f05bec4f74 => generated 0 bytes in 201 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 13 04:22:14.925761 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-82bacab0-685b-4f57-a45d-12721c67a385 admin admin] Authenticating user token {{(pid=81409) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:22:14.927299 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.fernet_utils [None req-82bacab0-685b-4f57-a45d-12721c67a385 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81409) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:22:14.942555 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-82bacab0-685b-4f57-a45d-12721c67a385 admin admin] Validating token access rules against request {{(pid=81409) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:22:14.987335 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG dbcounter [-] [81409] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=81409) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:22:15.041168 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-375343b2-8d2f-4051-939a-c9028a94c210 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8ba823f313144da3b683bacef8fd62e0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81409) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:22:15.041822 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-375343b2-8d2f-4051-939a-c9028a94c210 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:22:15.042094 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-375343b2-8d2f-4051-939a-c9028a94c210 admin admin] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:22:15.042393 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-375343b2-8d2f-4051-939a-c9028a94c210 admin admin] PATH_INFO: `/v3/projects/3a1b5dae4dd34f449427067c79f8c21b` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:22:15.043524 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-375343b2-8d2f-4051-939a-c9028a94c210 admin admin] RBAC: Authorizing `identity:delete_project(project_id=3a1b5dae4dd34f449427067c79f8c21b)` {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:22:15.046369 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-375343b2-8d2f-4051-939a-c9028a94c210 admin admin] RBAC: Authorization granted {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:22:15.098498 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.notifications [None req-375343b2-8d2f-4051-939a-c9028a94c210 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '3a1b5dae4dd34f449427067c79f8c21b', 'request_id': 'req-375343b2-8d2f-4051-939a-c9028a94c210'} {{(pid=81409) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 13 04:22:15.108923 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.notifications [None req-375343b2-8d2f-4051-939a-c9028a94c210 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '3a1b5dae4dd34f449427067c79f8c21b', 'request_id': 'req-375343b2-8d2f-4051-939a-c9028a94c210'} {{(pid=81409) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 13 04:22:15.116149 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.notifications [None req-375343b2-8d2f-4051-939a-c9028a94c210 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '3a1b5dae4dd34f449427067c79f8c21b', 'request_id': 'req-375343b2-8d2f-4051-939a-c9028a94c210'} {{(pid=81409) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 13 04:22:15.128614 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.notifications [None req-375343b2-8d2f-4051-939a-c9028a94c210 admin admin] The token cache is being invalidate because project 3a1b5dae4dd34f449427067c79f8c21b was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81409) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 13 04:22:15.128919 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.notifications [None req-375343b2-8d2f-4051-939a-c9028a94c210 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81409) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 13 04:22:15.129947 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.notifications [None req-375343b2-8d2f-4051-939a-c9028a94c210 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81409) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 13 04:22:15.131736 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 886/1743] 10.222.0.27 () {58 vars in 1233 bytes} [Thu Oct 13 04:22:14 2022] DELETE /identity/v3/projects/3a1b5dae4dd34f449427067c79f8c21b => generated 0 bytes in 207 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 13 04:22:15.135767 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4468ea18-e744-46db-8c0c-d3a5f6f4c996 admin admin] Authenticating user token {{(pid=81408) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:22:15.137524 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.fernet_utils [None req-4468ea18-e744-46db-8c0c-d3a5f6f4c996 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81408) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:22:15.155022 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4468ea18-e744-46db-8c0c-d3a5f6f4c996 admin admin] Validating token access rules against request {{(pid=81408) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:22:15.244715 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8bb277e4-7615-499e-adb3-55a511c49275 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8ba823f313144da3b683bacef8fd62e0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81408) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:22:15.245357 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8bb277e4-7615-499e-adb3-55a511c49275 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:22:15.245631 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8bb277e4-7615-499e-adb3-55a511c49275 admin admin] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:22:15.245902 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8bb277e4-7615-499e-adb3-55a511c49275 admin admin] PATH_INFO: `/v3/users/3600ed6e624246f49b6ee53b9de89ddc` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:22:15.247059 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8bb277e4-7615-499e-adb3-55a511c49275 admin admin] RBAC: Authorizing `identity:delete_user(user_id=3600ed6e624246f49b6ee53b9de89ddc)` {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:22:15.250728 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8bb277e4-7615-499e-adb3-55a511c49275 admin admin] RBAC: Authorization granted {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:22:15.300777 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.notifications [None req-8bb277e4-7615-499e-adb3-55a511c49275 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '3600ed6e624246f49b6ee53b9de89ddc'} {{(pid=81408) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 13 04:22:15.307615 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.notifications [None req-8bb277e4-7615-499e-adb3-55a511c49275 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '3600ed6e624246f49b6ee53b9de89ddc'} {{(pid=81408) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 13 04:22:15.308483 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.notifications [None req-8bb277e4-7615-499e-adb3-55a511c49275 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '3600ed6e624246f49b6ee53b9de89ddc'} {{(pid=81408) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 13 04:22:15.309103 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.notifications [None req-8bb277e4-7615-499e-adb3-55a511c49275 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '3600ed6e624246f49b6ee53b9de89ddc'} {{(pid=81408) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 13 04:22:15.317568 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.notifications [None req-8bb277e4-7615-499e-adb3-55a511c49275 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '3600ed6e624246f49b6ee53b9de89ddc'} {{(pid=81408) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 13 04:22:15.339389 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 858/1744] 10.222.0.27 () {58 vars in 1224 bytes} [Thu Oct 13 04:22:15 2022] DELETE /identity/v3/users/3600ed6e624246f49b6ee53b9de89ddc => generated 0 bytes in 204 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 13 04:22:15.621223 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-375343b2-8d2f-4051-939a-c9028a94c210 admin admin] Authenticating user token {{(pid=81409) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:22:15.622883 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.fernet_utils [None req-375343b2-8d2f-4051-939a-c9028a94c210 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81409) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:22:15.637465 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-375343b2-8d2f-4051-939a-c9028a94c210 admin admin] Validating token access rules against request {{(pid=81409) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:22:15.696578 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e226de12-5bb4-441c-828f-e7f0b9cdeedd admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8ba823f313144da3b683bacef8fd62e0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81409) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:22:15.697238 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e226de12-5bb4-441c-828f-e7f0b9cdeedd admin admin] REQUEST_METHOD: `DELETE` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:22:15.697511 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e226de12-5bb4-441c-828f-e7f0b9cdeedd admin admin] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:22:15.697777 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e226de12-5bb4-441c-828f-e7f0b9cdeedd admin admin] PATH_INFO: `/v3/projects/1a161c93104e43a8bfdbb6e97a3ebf1f` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:22:15.698923 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e226de12-5bb4-441c-828f-e7f0b9cdeedd admin admin] RBAC: Authorizing `identity:delete_project(project_id=1a161c93104e43a8bfdbb6e97a3ebf1f)` {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:22:15.701421 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e226de12-5bb4-441c-828f-e7f0b9cdeedd admin admin] RBAC: Authorization granted {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:22:15.747019 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.notifications [None req-e226de12-5bb4-441c-828f-e7f0b9cdeedd admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '1a161c93104e43a8bfdbb6e97a3ebf1f', 'request_id': 'req-e226de12-5bb4-441c-828f-e7f0b9cdeedd'} {{(pid=81409) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 13 04:22:15.754104 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.notifications [None req-e226de12-5bb4-441c-828f-e7f0b9cdeedd admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '1a161c93104e43a8bfdbb6e97a3ebf1f', 'request_id': 'req-e226de12-5bb4-441c-828f-e7f0b9cdeedd'} {{(pid=81409) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 13 04:22:15.761151 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.notifications [None req-e226de12-5bb4-441c-828f-e7f0b9cdeedd admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '1a161c93104e43a8bfdbb6e97a3ebf1f', 'request_id': 'req-e226de12-5bb4-441c-828f-e7f0b9cdeedd'} {{(pid=81409) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 13 04:22:15.766726 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG dbcounter [-] [81409] Writing DB stats keystone:SELECT=84,keystone:DELETE=14,keystone:INSERT=2 {{(pid=81409) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:22:15.774660 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.notifications [None req-e226de12-5bb4-441c-828f-e7f0b9cdeedd admin admin] The token cache is being invalidate because project 1a161c93104e43a8bfdbb6e97a3ebf1f was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81409) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 13 04:22:15.774980 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.notifications [None req-e226de12-5bb4-441c-828f-e7f0b9cdeedd admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81409) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 13 04:22:15.775866 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.notifications [None req-e226de12-5bb4-441c-828f-e7f0b9cdeedd admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81409) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 13 04:22:15.777332 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 887/1745] 10.222.0.27 () {58 vars in 1233 bytes} [Thu Oct 13 04:22:15 2022] DELETE /identity/v3/projects/1a161c93104e43a8bfdbb6e97a3ebf1f => generated 0 bytes in 157 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 13 04:22:15.783353 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-296a54d1-f280-4664-9a64-5230eaf952b6 None None] REQUEST_METHOD: `POST` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:22:15.783827 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-296a54d1-f280-4664-9a64-5230eaf952b6 None None] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:22:15.784066 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-296a54d1-f280-4664-9a64-5230eaf952b6 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:22:15.803906 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.auth.core [None req-296a54d1-f280-4664-9a64-5230eaf952b6 None None] MFA Rules not processed for user `0e2d979563d6421995451a0e86bf3022`. Rule list: `[]` (Enabled: `True`). {{(pid=81408) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 13 04:22:15.805117 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.fernet_utils [None req-296a54d1-f280-4664-9a64-5230eaf952b6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81408) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:22:15.864899 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 859/1746] 10.222.0.27 () {58 vars in 958 bytes} [Thu Oct 13 04:22:15 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 83 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 13 04:22:15.872818 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b49eb728-1290-45bd-b687-b1e1b2e6b7ec None None] REQUEST_METHOD: `POST` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:22:15.873112 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b49eb728-1290-45bd-b687-b1e1b2e6b7ec None None] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:22:15.873381 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b49eb728-1290-45bd-b687-b1e1b2e6b7ec None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:22:15.895090 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.auth.core [None req-b49eb728-1290-45bd-b687-b1e1b2e6b7ec None None] MFA Rules not processed for user `0e2d979563d6421995451a0e86bf3022`. Rule list: `[]` (Enabled: `True`). {{(pid=81409) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 13 04:22:15.896301 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.fernet_utils [None req-b49eb728-1290-45bd-b687-b1e1b2e6b7ec None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81409) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:22:15.904836 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 888/1747] 10.222.0.27 () {58 vars in 958 bytes} [Thu Oct 13 04:22:15 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 33 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 13 04:22:15.910414 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-296a54d1-f280-4664-9a64-5230eaf952b6 None None] Authenticating user token {{(pid=81408) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:22:15.920244 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-296a54d1-f280-4664-9a64-5230eaf952b6 None None] Validating token access rules against request {{(pid=81408) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:22:15.924908 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6315c811-47de-4dbc-a0df-29ed8a6d7e40 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8ba823f313144da3b683bacef8fd62e0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81408) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:22:15.925510 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6315c811-47de-4dbc-a0df-29ed8a6d7e40 admin admin] REQUEST_METHOD: `GET` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:22:15.925787 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6315c811-47de-4dbc-a0df-29ed8a6d7e40 admin admin] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:22:15.926058 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6315c811-47de-4dbc-a0df-29ed8a6d7e40 admin admin] PATH_INFO: `/v3/domains` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:22:15.926742 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6315c811-47de-4dbc-a0df-29ed8a6d7e40 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81408) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 13 04:22:15.927042 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6315c811-47de-4dbc-a0df-29ed8a6d7e40 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:22:15.928651 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6315c811-47de-4dbc-a0df-29ed8a6d7e40 admin admin] RBAC: Authorization granted {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:22:15.935923 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG dbcounter [-] [81408] Writing DB stats keystone:SELECT=88,keystone:DELETE=11,keystone:INSERT=1 {{(pid=81408) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:22:15.942191 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 860/1748] 10.222.0.27 () {58 vars in 1153 bytes} [Thu Oct 13 04:22:15 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 13 04:22:15.946979 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b49eb728-1290-45bd-b687-b1e1b2e6b7ec None None] Authenticating user token {{(pid=81409) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:22:15.957161 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b49eb728-1290-45bd-b687-b1e1b2e6b7ec None None] Validating token access rules against request {{(pid=81409) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:22:15.961972 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fd5afae4-a2fd-4e20-9e8d-dc0221056d87 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8ba823f313144da3b683bacef8fd62e0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81409) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:22:15.962616 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd5afae4-a2fd-4e20-9e8d-dc0221056d87 admin admin] REQUEST_METHOD: `POST` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:22:15.962890 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd5afae4-a2fd-4e20-9e8d-dc0221056d87 admin admin] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:22:15.963154 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd5afae4-a2fd-4e20-9e8d-dc0221056d87 admin admin] PATH_INFO: `/v3/projects` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:22:15.963867 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fd5afae4-a2fd-4e20-9e8d-dc0221056d87 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:22:15.965787 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fd5afae4-a2fd-4e20-9e8d-dc0221056d87 admin admin] RBAC: Authorization granted {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:22:15.994092 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 889/1749] 10.222.0.27 () {60 vars in 1153 bytes} [Thu Oct 13 04:22:15 2022] POST /identity/v3/projects => generated 373 bytes in 48 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 13 04:22:15.998365 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6315c811-47de-4dbc-a0df-29ed8a6d7e40 admin admin] Authenticating user token {{(pid=81408) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:22:16.008428 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6315c811-47de-4dbc-a0df-29ed8a6d7e40 admin admin] Validating token access rules against request {{(pid=81408) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:22:16.068774 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0811f571-c848-4915-8184-33d2fe084a51 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8ba823f313144da3b683bacef8fd62e0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81408) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:22:16.069405 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0811f571-c848-4915-8184-33d2fe084a51 admin admin] REQUEST_METHOD: `POST` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:22:16.069685 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0811f571-c848-4915-8184-33d2fe084a51 admin admin] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:22:16.069955 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0811f571-c848-4915-8184-33d2fe084a51 admin admin] PATH_INFO: `/v3/users` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:22:16.070618 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0811f571-c848-4915-8184-33d2fe084a51 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:22:16.072524 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0811f571-c848-4915-8184-33d2fe084a51 admin admin] RBAC: Authorization granted {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:22:16.107740 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 861/1750] 10.222.0.27 () {60 vars in 1144 bytes} [Thu Oct 13 04:22:15 2022] POST /identity/v3/users => generated 287 bytes in 110 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 13 04:22:16.111795 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fd5afae4-a2fd-4e20-9e8d-dc0221056d87 admin admin] Authenticating user token {{(pid=81409) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:22:16.123682 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fd5afae4-a2fd-4e20-9e8d-dc0221056d87 admin admin] Validating token access rules against request {{(pid=81409) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:22:16.129662 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0197e630-c7c5-4f8d-8566-a668d57edff4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8ba823f313144da3b683bacef8fd62e0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81409) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:22:16.130439 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0197e630-c7c5-4f8d-8566-a668d57edff4 admin admin] REQUEST_METHOD: `GET` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:22:16.130833 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0197e630-c7c5-4f8d-8566-a668d57edff4 admin admin] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:22:16.131314 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0197e630-c7c5-4f8d-8566-a668d57edff4 admin admin] PATH_INFO: `/v3/roles` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:22:16.132051 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0197e630-c7c5-4f8d-8566-a668d57edff4 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:22:16.133931 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0197e630-c7c5-4f8d-8566-a668d57edff4 admin admin] RBAC: Authorization granted {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:22:16.145973 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 890/1751] 10.222.0.27 () {58 vars in 1122 bytes} [Thu Oct 13 04:22:16 2022] GET /identity/v3/roles => generated 1420 bytes in 35 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 13 04:22:16.151437 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0811f571-c848-4915-8184-33d2fe084a51 admin admin] Authenticating user token {{(pid=81408) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:22:16.174034 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0811f571-c848-4915-8184-33d2fe084a51 admin admin] Validating token access rules against request {{(pid=81408) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:22:16.178709 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a828c55d-23cc-45ba-9268-7df1a6bcb12a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8ba823f313144da3b683bacef8fd62e0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81408) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:22:16.179318 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a828c55d-23cc-45ba-9268-7df1a6bcb12a admin admin] REQUEST_METHOD: `GET` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:22:16.179591 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a828c55d-23cc-45ba-9268-7df1a6bcb12a admin admin] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:22:16.179860 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a828c55d-23cc-45ba-9268-7df1a6bcb12a admin admin] PATH_INFO: `/v3/roles` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:22:16.180611 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a828c55d-23cc-45ba-9268-7df1a6bcb12a admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:22:16.182251 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a828c55d-23cc-45ba-9268-7df1a6bcb12a admin admin] RBAC: Authorization granted {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:22:16.193407 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 862/1752] 10.222.0.27 () {58 vars in 1122 bytes} [Thu Oct 13 04:22:16 2022] GET /identity/v3/roles => generated 1420 bytes in 42 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 13 04:22:16.197426 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0197e630-c7c5-4f8d-8566-a668d57edff4 admin admin] Authenticating user token {{(pid=81409) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:22:16.208107 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0197e630-c7c5-4f8d-8566-a668d57edff4 admin admin] Validating token access rules against request {{(pid=81409) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:22:16.213014 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-88b40f8b-6a28-4bb6-8e76-80b24f9a03bf admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8ba823f313144da3b683bacef8fd62e0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81409) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:22:16.213665 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88b40f8b-6a28-4bb6-8e76-80b24f9a03bf admin admin] REQUEST_METHOD: `PUT` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:22:16.213942 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88b40f8b-6a28-4bb6-8e76-80b24f9a03bf admin admin] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:22:16.214234 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88b40f8b-6a28-4bb6-8e76-80b24f9a03bf admin admin] PATH_INFO: `/v3/projects/e7d1e420eec84d1f8b78333b7d814f3c/users/9ee6eac2e6da4d83bd14324669dc7269/roles/cc56c4ae13a7424ea771c15da4b153c1` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:22:16.230310 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-88b40f8b-6a28-4bb6-8e76-80b24f9a03bf admin admin] RBAC: Authorizing `identity:create_grant(project_id=e7d1e420eec84d1f8b78333b7d814f3c, user_id=9ee6eac2e6da4d83bd14324669dc7269, role_id=cc56c4ae13a7424ea771c15da4b153c1)` {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:22:16.233287 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-88b40f8b-6a28-4bb6-8e76-80b24f9a03bf admin admin] RBAC: Authorization granted {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:22:16.249718 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 891/1753] 10.222.0.27 () {60 vars in 1483 bytes} [Thu Oct 13 04:22:16 2022] PUT /identity/v3/projects/e7d1e420eec84d1f8b78333b7d814f3c/users/9ee6eac2e6da4d83bd14324669dc7269/roles/cc56c4ae13a7424ea771c15da4b153c1 => generated 0 bytes in 52 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 13 04:22:16.255333 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae1b2535-db34-40d3-a927-c8ff2ee45c0a None None] REQUEST_METHOD: `POST` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:22:16.255723 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae1b2535-db34-40d3-a927-c8ff2ee45c0a None None] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:22:16.256449 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae1b2535-db34-40d3-a927-c8ff2ee45c0a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:22:16.279533 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.auth.core [None req-ae1b2535-db34-40d3-a927-c8ff2ee45c0a None None] MFA Rules not processed for user `0e2d979563d6421995451a0e86bf3022`. Rule list: `[]` (Enabled: `True`). {{(pid=81408) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 13 04:22:16.281718 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.fernet_utils [None req-ae1b2535-db34-40d3-a927-c8ff2ee45c0a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81408) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:22:16.343451 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 863/1754] 10.222.0.27 () {58 vars in 958 bytes} [Thu Oct 13 04:22:16 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 90 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 13 04:22:16.353567 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-88b40f8b-6a28-4bb6-8e76-80b24f9a03bf admin admin] Authenticating user token {{(pid=81409) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:22:16.355434 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.fernet_utils [None req-88b40f8b-6a28-4bb6-8e76-80b24f9a03bf admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81409) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:22:16.418142 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-88b40f8b-6a28-4bb6-8e76-80b24f9a03bf admin admin] Validating token access rules against request {{(pid=81409) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:22:16.422142 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1c49d248-12ff-4030-b389-fd3f9dd0b965 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd8812c3974f84871bb2d761cf0016257', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dcec0431fb6849499d6aec35a234daa6', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81409) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:22:16.422784 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c49d248-12ff-4030-b389-fd3f9dd0b965 service neutron] REQUEST_METHOD: `GET` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:22:16.423056 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c49d248-12ff-4030-b389-fd3f9dd0b965 service neutron] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:22:16.423322 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c49d248-12ff-4030-b389-fd3f9dd0b965 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:22:16.432969 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1c49d248-12ff-4030-b389-fd3f9dd0b965 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:22:16.434678 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1c49d248-12ff-4030-b389-fd3f9dd0b965 service neutron] RBAC: Authorization granted {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:22:16.444948 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 892/1755] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Oct 13 04:22:16 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 92 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 13 04:22:17.251322 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0b50f61-d56e-4c69-be12-d52154a77f47 None None] REQUEST_METHOD: `POST` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:22:17.251913 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0b50f61-d56e-4c69-be12-d52154a77f47 None None] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:22:17.251913 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0b50f61-d56e-4c69-be12-d52154a77f47 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:22:17.282149 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.auth.core [None req-d0b50f61-d56e-4c69-be12-d52154a77f47 None None] MFA Rules not processed for user `9ee6eac2e6da4d83bd14324669dc7269`. Rule list: `[]` (Enabled: `True`). {{(pid=81408) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 13 04:22:17.283743 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.fernet_utils [None req-d0b50f61-d56e-4c69-be12-d52154a77f47 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81408) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:22:17.409962 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 864/1756] 10.222.0.27 () {58 vars in 958 bytes} [Thu Oct 13 04:22:17 2022] POST /identity/v3/auth/tokens => generated 3250 bytes in 159 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 13 04:22:17.429204 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1c49d248-12ff-4030-b389-fd3f9dd0b965 service neutron] Authenticating user token {{(pid=81409) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:22:17.431545 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.fernet_utils [None req-1c49d248-12ff-4030-b389-fd3f9dd0b965 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81409) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:22:17.521407 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1c49d248-12ff-4030-b389-fd3f9dd0b965 service neutron] Validating token access rules against request {{(pid=81409) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:22:17.525267 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b8a1ef40-3fa9-4a06-80fc-94a790b9c2d6 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6141b8a6c3d54e179d6879bb019f4d12', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dcec0431fb6849499d6aec35a234daa6', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81409) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:22:17.525874 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8a1ef40-3fa9-4a06-80fc-94a790b9c2d6 service cinder] REQUEST_METHOD: `GET` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:22:17.526145 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8a1ef40-3fa9-4a06-80fc-94a790b9c2d6 service cinder] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:22:17.526443 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8a1ef40-3fa9-4a06-80fc-94a790b9c2d6 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:22:17.535545 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b8a1ef40-3fa9-4a06-80fc-94a790b9c2d6 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:22:17.536982 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b8a1ef40-3fa9-4a06-80fc-94a790b9c2d6 service cinder] RBAC: Authorization granted {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:22:17.546885 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 893/1757] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Oct 13 04:22:17 2022] GET /identity/v3/auth/tokens => generated 3250 bytes in 118 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 13 04:22:18.984434 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5aa410da-9137-4ad0-8a21-3f680cf9d50b None None] REQUEST_METHOD: `POST` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:22:18.985822 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5aa410da-9137-4ad0-8a21-3f680cf9d50b None None] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:22:18.986343 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5aa410da-9137-4ad0-8a21-3f680cf9d50b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:22:19.013014 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG dbcounter [-] [81408] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81408) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:22:19.030935 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.auth.core [None req-5aa410da-9137-4ad0-8a21-3f680cf9d50b None None] MFA Rules not processed for user `9ee6eac2e6da4d83bd14324669dc7269`. Rule list: `[]` (Enabled: `True`). {{(pid=81408) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 13 04:22:19.032217 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.fernet_utils [None req-5aa410da-9137-4ad0-8a21-3f680cf9d50b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81408) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:22:19.046882 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 865/1758] 10.222.0.27 () {58 vars in 958 bytes} [Thu Oct 13 04:22:18 2022] POST /identity/v3/auth/tokens => generated 3250 bytes in 61 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 13 04:22:20.978898 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b8a1ef40-3fa9-4a06-80fc-94a790b9c2d6 service cinder] Authenticating user token {{(pid=81409) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:22:20.980363 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.fernet_utils [None req-b8a1ef40-3fa9-4a06-80fc-94a790b9c2d6 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81409) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:22:21.001023 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b8a1ef40-3fa9-4a06-80fc-94a790b9c2d6 service cinder] Validating token access rules against request {{(pid=81409) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:22:21.005874 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8a37fe89-2c67-4684-b75a-94f4b95a0a3a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8ba823f313144da3b683bacef8fd62e0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81409) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:22:21.006636 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a37fe89-2c67-4684-b75a-94f4b95a0a3a admin admin] REQUEST_METHOD: `DELETE` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:22:21.006903 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a37fe89-2c67-4684-b75a-94f4b95a0a3a admin admin] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:22:21.007175 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a37fe89-2c67-4684-b75a-94f4b95a0a3a admin admin] PATH_INFO: `/v3/users/70ff2e3f5a0f4e0da74af8d0ea6b1506` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:22:21.008516 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8a37fe89-2c67-4684-b75a-94f4b95a0a3a admin admin] RBAC: Authorizing `identity:delete_user(user_id=70ff2e3f5a0f4e0da74af8d0ea6b1506)` {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:22:21.010580 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8a37fe89-2c67-4684-b75a-94f4b95a0a3a admin admin] RBAC: Authorization granted {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:22:21.064213 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG dbcounter [-] [81409] Writing DB stats keystone:SELECT=93,keystone:INSERT=2,keystone:DELETE=5 {{(pid=81409) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:22:21.084129 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.notifications [None req-8a37fe89-2c67-4684-b75a-94f4b95a0a3a admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '70ff2e3f5a0f4e0da74af8d0ea6b1506'} {{(pid=81409) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 13 04:22:21.084796 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.notifications [None req-8a37fe89-2c67-4684-b75a-94f4b95a0a3a admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '70ff2e3f5a0f4e0da74af8d0ea6b1506'} {{(pid=81409) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 13 04:22:21.096305 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.notifications [None req-8a37fe89-2c67-4684-b75a-94f4b95a0a3a admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '70ff2e3f5a0f4e0da74af8d0ea6b1506'} {{(pid=81409) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 13 04:22:21.102633 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.notifications [None req-8a37fe89-2c67-4684-b75a-94f4b95a0a3a admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '70ff2e3f5a0f4e0da74af8d0ea6b1506'} {{(pid=81409) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 13 04:22:21.111694 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.notifications [None req-8a37fe89-2c67-4684-b75a-94f4b95a0a3a admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '70ff2e3f5a0f4e0da74af8d0ea6b1506'} {{(pid=81409) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 13 04:22:21.132245 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 894/1759] 10.222.0.27 () {58 vars in 1224 bytes} [Thu Oct 13 04:22:20 2022] DELETE /identity/v3/users/70ff2e3f5a0f4e0da74af8d0ea6b1506 => generated 0 bytes in 154 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 13 04:22:21.324566 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5aa410da-9137-4ad0-8a21-3f680cf9d50b None None] Authenticating user token {{(pid=81408) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:22:21.337249 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5aa410da-9137-4ad0-8a21-3f680cf9d50b None None] Validating token access rules against request {{(pid=81408) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:22:21.403552 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0cb63950-a291-40e2-9ba5-307d04b83551 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8ba823f313144da3b683bacef8fd62e0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81408) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:22:21.404265 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0cb63950-a291-40e2-9ba5-307d04b83551 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:22:21.404638 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0cb63950-a291-40e2-9ba5-307d04b83551 admin admin] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:22:21.404913 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0cb63950-a291-40e2-9ba5-307d04b83551 admin admin] PATH_INFO: `/v3/projects/2413ab4744c345bc94f68b6510ca3b44` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:22:21.406082 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0cb63950-a291-40e2-9ba5-307d04b83551 admin admin] RBAC: Authorizing `identity:delete_project(project_id=2413ab4744c345bc94f68b6510ca3b44)` {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:22:21.408066 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0cb63950-a291-40e2-9ba5-307d04b83551 admin admin] RBAC: Authorization granted {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:22:21.470275 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.notifications [None req-0cb63950-a291-40e2-9ba5-307d04b83551 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '2413ab4744c345bc94f68b6510ca3b44', 'request_id': 'req-0cb63950-a291-40e2-9ba5-307d04b83551'} {{(pid=81408) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 13 04:22:21.482226 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.notifications [None req-0cb63950-a291-40e2-9ba5-307d04b83551 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '2413ab4744c345bc94f68b6510ca3b44', 'request_id': 'req-0cb63950-a291-40e2-9ba5-307d04b83551'} {{(pid=81408) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 13 04:22:21.493674 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.notifications [None req-0cb63950-a291-40e2-9ba5-307d04b83551 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '2413ab4744c345bc94f68b6510ca3b44', 'request_id': 'req-0cb63950-a291-40e2-9ba5-307d04b83551'} {{(pid=81408) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 13 04:22:21.505622 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.notifications [None req-0cb63950-a291-40e2-9ba5-307d04b83551 admin admin] The token cache is being invalidate because project 2413ab4744c345bc94f68b6510ca3b44 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81408) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 13 04:22:21.505989 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.notifications [None req-0cb63950-a291-40e2-9ba5-307d04b83551 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81408) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 13 04:22:21.506681 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.notifications [None req-0cb63950-a291-40e2-9ba5-307d04b83551 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81408) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 13 04:22:21.508272 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 866/1760] 10.222.0.27 () {58 vars in 1233 bytes} [Thu Oct 13 04:22:21 2022] DELETE /identity/v3/projects/2413ab4744c345bc94f68b6510ca3b44 => generated 0 bytes in 184 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 13 04:22:21.513431 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-235e4a48-cb7a-4b7b-8b20-03ff005c66c5 None None] REQUEST_METHOD: `POST` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:22:21.513721 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-235e4a48-cb7a-4b7b-8b20-03ff005c66c5 None None] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:22:21.513993 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-235e4a48-cb7a-4b7b-8b20-03ff005c66c5 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:22:21.537460 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.auth.core [None req-235e4a48-cb7a-4b7b-8b20-03ff005c66c5 None None] MFA Rules not processed for user `0e2d979563d6421995451a0e86bf3022`. Rule list: `[]` (Enabled: `True`). {{(pid=81409) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 13 04:22:21.538709 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.fernet_utils [None req-235e4a48-cb7a-4b7b-8b20-03ff005c66c5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81409) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:22:21.664863 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 895/1761] 10.222.0.27 () {58 vars in 958 bytes} [Thu Oct 13 04:22:21 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 151 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 13 04:22:21.666034 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12a9462f-ebdc-4e03-9cc4-f51c26cf5f05 None None] REQUEST_METHOD: `POST` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:22:21.666471 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12a9462f-ebdc-4e03-9cc4-f51c26cf5f05 None None] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:22:21.666986 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12a9462f-ebdc-4e03-9cc4-f51c26cf5f05 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:22:21.705936 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.auth.core [None req-12a9462f-ebdc-4e03-9cc4-f51c26cf5f05 None None] MFA Rules not processed for user `0e2d979563d6421995451a0e86bf3022`. Rule list: `[]` (Enabled: `True`). {{(pid=81408) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 13 04:22:21.707252 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.fernet_utils [None req-12a9462f-ebdc-4e03-9cc4-f51c26cf5f05 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81408) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:22:21.718364 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 867/1762] 10.222.0.27 () {58 vars in 958 bytes} [Thu Oct 13 04:22:21 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 53 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 13 04:22:21.722783 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-235e4a48-cb7a-4b7b-8b20-03ff005c66c5 None None] Authenticating user token {{(pid=81409) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:22:21.735835 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-235e4a48-cb7a-4b7b-8b20-03ff005c66c5 None None] Validating token access rules against request {{(pid=81409) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:22:21.741970 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-203c0596-2024-4777-989a-f2b7fbd39bdb admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8ba823f313144da3b683bacef8fd62e0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81409) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:22:21.742736 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-203c0596-2024-4777-989a-f2b7fbd39bdb admin admin] REQUEST_METHOD: `GET` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:22:21.743056 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-203c0596-2024-4777-989a-f2b7fbd39bdb admin admin] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:22:21.743336 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-203c0596-2024-4777-989a-f2b7fbd39bdb admin admin] PATH_INFO: `/v3/domains` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:22:21.744087 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-203c0596-2024-4777-989a-f2b7fbd39bdb admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81409) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 13 04:22:21.744393 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-203c0596-2024-4777-989a-f2b7fbd39bdb admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:22:21.746390 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-203c0596-2024-4777-989a-f2b7fbd39bdb admin admin] RBAC: Authorization granted {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:22:21.765075 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 896/1763] 10.222.0.27 () {58 vars in 1153 bytes} [Thu Oct 13 04:22:21 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 43 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 13 04:22:21.769032 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-12a9462f-ebdc-4e03-9cc4-f51c26cf5f05 None None] Authenticating user token {{(pid=81408) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:22:21.781765 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-12a9462f-ebdc-4e03-9cc4-f51c26cf5f05 None None] Validating token access rules against request {{(pid=81408) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:22:21.787330 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a0d4c615-adbb-4ea9-9d47-ee750ee4e565 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8ba823f313144da3b683bacef8fd62e0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81408) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:22:21.787974 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0d4c615-adbb-4ea9-9d47-ee750ee4e565 admin admin] REQUEST_METHOD: `POST` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:22:21.788266 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0d4c615-adbb-4ea9-9d47-ee750ee4e565 admin admin] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:22:21.788612 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0d4c615-adbb-4ea9-9d47-ee750ee4e565 admin admin] PATH_INFO: `/v3/projects` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:22:21.789297 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a0d4c615-adbb-4ea9-9d47-ee750ee4e565 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:22:21.795534 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a0d4c615-adbb-4ea9-9d47-ee750ee4e565 admin admin] RBAC: Authorization granted {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:22:21.881830 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 868/1764] 10.222.0.27 () {60 vars in 1153 bytes} [Thu Oct 13 04:22:21 2022] POST /identity/v3/projects => generated 381 bytes in 111 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 13 04:22:21.883311 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-203c0596-2024-4777-989a-f2b7fbd39bdb admin admin] Authenticating user token {{(pid=81409) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:22:21.899746 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-203c0596-2024-4777-989a-f2b7fbd39bdb admin admin] Validating token access rules against request {{(pid=81409) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:22:21.980312 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-30582e80-048d-47a1-9e39-9bf09c12b5a1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8ba823f313144da3b683bacef8fd62e0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81409) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:22:21.981317 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30582e80-048d-47a1-9e39-9bf09c12b5a1 admin admin] REQUEST_METHOD: `POST` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:22:21.981998 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30582e80-048d-47a1-9e39-9bf09c12b5a1 admin admin] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:22:21.982376 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30582e80-048d-47a1-9e39-9bf09c12b5a1 admin admin] PATH_INFO: `/v3/users` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:22:21.983473 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-30582e80-048d-47a1-9e39-9bf09c12b5a1 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:22:21.985510 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-30582e80-048d-47a1-9e39-9bf09c12b5a1 admin admin] RBAC: Authorization granted {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:22:22.012753 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 897/1765] 10.222.0.27 () {60 vars in 1144 bytes} [Thu Oct 13 04:22:21 2022] POST /identity/v3/users => generated 291 bytes in 130 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 13 04:22:22.017347 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a0d4c615-adbb-4ea9-9d47-ee750ee4e565 admin admin] Authenticating user token {{(pid=81408) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:22:22.027672 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a0d4c615-adbb-4ea9-9d47-ee750ee4e565 admin admin] Validating token access rules against request {{(pid=81408) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:22:22.032626 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-06bb53b8-f1d8-4ff4-b35e-cc13de77516e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8ba823f313144da3b683bacef8fd62e0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81408) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:22:22.033225 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06bb53b8-f1d8-4ff4-b35e-cc13de77516e admin admin] REQUEST_METHOD: `GET` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:22:22.033496 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06bb53b8-f1d8-4ff4-b35e-cc13de77516e admin admin] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:22:22.033758 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06bb53b8-f1d8-4ff4-b35e-cc13de77516e admin admin] PATH_INFO: `/v3/roles` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:22:22.034498 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-06bb53b8-f1d8-4ff4-b35e-cc13de77516e admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:22:22.036113 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-06bb53b8-f1d8-4ff4-b35e-cc13de77516e admin admin] RBAC: Authorization granted {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:22:22.047495 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 869/1766] 10.222.0.27 () {58 vars in 1122 bytes} [Thu Oct 13 04:22:22 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 13 04:22:22.051696 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-30582e80-048d-47a1-9e39-9bf09c12b5a1 admin admin] Authenticating user token {{(pid=81409) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:22:22.062496 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-30582e80-048d-47a1-9e39-9bf09c12b5a1 admin admin] Validating token access rules against request {{(pid=81409) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:22:22.067314 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-03c76063-a22f-40e7-972d-7b0a91c559ba admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8ba823f313144da3b683bacef8fd62e0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81409) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:22:22.067942 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03c76063-a22f-40e7-972d-7b0a91c559ba admin admin] REQUEST_METHOD: `GET` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:22:22.068217 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03c76063-a22f-40e7-972d-7b0a91c559ba admin admin] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:22:22.068482 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03c76063-a22f-40e7-972d-7b0a91c559ba admin admin] PATH_INFO: `/v3/roles` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:22:22.069143 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-03c76063-a22f-40e7-972d-7b0a91c559ba admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:22:22.070839 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-03c76063-a22f-40e7-972d-7b0a91c559ba admin admin] RBAC: Authorization granted {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:22:22.086244 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 898/1767] 10.222.0.27 () {58 vars in 1122 bytes} [Thu Oct 13 04:22:22 2022] GET /identity/v3/roles => generated 1420 bytes in 33 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Oct 13 04:22:22.087968 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-06bb53b8-f1d8-4ff4-b35e-cc13de77516e admin admin] Authenticating user token {{(pid=81408) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:22:22.099377 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-06bb53b8-f1d8-4ff4-b35e-cc13de77516e admin admin] Validating token access rules against request {{(pid=81408) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:22:22.104395 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f82b18cd-642a-46c7-a769-ac4c26138a55 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8ba823f313144da3b683bacef8fd62e0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81408) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:22:22.105102 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f82b18cd-642a-46c7-a769-ac4c26138a55 admin admin] REQUEST_METHOD: `PUT` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:22:22.105378 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f82b18cd-642a-46c7-a769-ac4c26138a55 admin admin] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:22:22.105646 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f82b18cd-642a-46c7-a769-ac4c26138a55 admin admin] PATH_INFO: `/v3/projects/52f3ab53f6af494b8d6206784b97661e/users/666598590c6b40aa9ad02ecf6a9a63f5/roles/cc56c4ae13a7424ea771c15da4b153c1` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:22:22.148485 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f82b18cd-642a-46c7-a769-ac4c26138a55 admin admin] RBAC: Authorizing `identity:create_grant(project_id=52f3ab53f6af494b8d6206784b97661e, user_id=666598590c6b40aa9ad02ecf6a9a63f5, role_id=cc56c4ae13a7424ea771c15da4b153c1)` {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:22:22.151772 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f82b18cd-642a-46c7-a769-ac4c26138a55 admin admin] RBAC: Authorization granted {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:22:22.168036 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 870/1768] 10.222.0.27 () {60 vars in 1483 bytes} [Thu Oct 13 04:22:22 2022] PUT /identity/v3/projects/52f3ab53f6af494b8d6206784b97661e/users/666598590c6b40aa9ad02ecf6a9a63f5/roles/cc56c4ae13a7424ea771c15da4b153c1 => generated 0 bytes in 80 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 13 04:22:22.178641 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-941dc93b-3430-4ce4-9511-c54e44c2faf4 None None] REQUEST_METHOD: `POST` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:22:22.179167 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-941dc93b-3430-4ce4-9511-c54e44c2faf4 None None] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:22:22.179598 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-941dc93b-3430-4ce4-9511-c54e44c2faf4 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:22:22.209888 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.auth.core [None req-941dc93b-3430-4ce4-9511-c54e44c2faf4 None None] MFA Rules not processed for user `666598590c6b40aa9ad02ecf6a9a63f5`. Rule list: `[]` (Enabled: `True`). {{(pid=81409) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 13 04:22:22.211513 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.fernet_utils [None req-941dc93b-3430-4ce4-9511-c54e44c2faf4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81409) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:22:22.215817 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG dbcounter [-] [81409] Writing DB stats keystone:SELECT=91,keystone:DELETE=5,keystone:INSERT=4 {{(pid=81409) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:22:22.288210 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 899/1769] 10.222.0.27 () {58 vars in 958 bytes} [Thu Oct 13 04:22:22 2022] POST /identity/v3/auth/tokens => generated 3258 bytes in 111 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 13 04:22:22.292709 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f82b18cd-642a-46c7-a769-ac4c26138a55 admin admin] Authenticating user token {{(pid=81408) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:22:22.333276 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG dbcounter [-] [81408] Writing DB stats keystone:SELECT=90,keystone:DELETE=7,keystone:INSERT=3 {{(pid=81408) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:22:22.371043 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f82b18cd-642a-46c7-a769-ac4c26138a55 admin admin] Validating token access rules against request {{(pid=81408) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:22:22.376354 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1a386ac2-7631-4754-8d5c-0d3859001a0c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8ba823f313144da3b683bacef8fd62e0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81408) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:22:22.376961 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a386ac2-7631-4754-8d5c-0d3859001a0c admin admin] REQUEST_METHOD: `POST` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:22:22.377243 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a386ac2-7631-4754-8d5c-0d3859001a0c admin admin] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:22:22.377510 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a386ac2-7631-4754-8d5c-0d3859001a0c admin admin] PATH_INFO: `/v3/projects` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:22:22.378157 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1a386ac2-7631-4754-8d5c-0d3859001a0c admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:22:22.380116 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1a386ac2-7631-4754-8d5c-0d3859001a0c admin admin] RBAC: Authorization granted {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:22:22.419937 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 871/1770] 10.222.0.27 () {60 vars in 1153 bytes} [Thu Oct 13 04:22:22 2022] POST /identity/v3/projects => generated 379 bytes in 125 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 13 04:22:22.421089 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-941dc93b-3430-4ce4-9511-c54e44c2faf4 None None] Authenticating user token {{(pid=81409) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:22:22.433244 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-941dc93b-3430-4ce4-9511-c54e44c2faf4 None None] Validating token access rules against request {{(pid=81409) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:22:22.496845 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fc0162c4-940e-4bba-a55c-b80d97570cd4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8ba823f313144da3b683bacef8fd62e0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81409) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:22:22.498442 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc0162c4-940e-4bba-a55c-b80d97570cd4 admin admin] REQUEST_METHOD: `POST` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:22:22.498728 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc0162c4-940e-4bba-a55c-b80d97570cd4 admin admin] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:22:22.499020 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc0162c4-940e-4bba-a55c-b80d97570cd4 admin admin] PATH_INFO: `/v3/users` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:22:22.499656 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fc0162c4-940e-4bba-a55c-b80d97570cd4 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:22:22.501570 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fc0162c4-940e-4bba-a55c-b80d97570cd4 admin admin] RBAC: Authorization granted {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:22:22.523161 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 900/1771] 10.222.0.27 () {60 vars in 1144 bytes} [Thu Oct 13 04:22:22 2022] POST /identity/v3/users => generated 296 bytes in 102 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 13 04:22:22.527252 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1a386ac2-7631-4754-8d5c-0d3859001a0c admin admin] Authenticating user token {{(pid=81408) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:22:22.538381 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1a386ac2-7631-4754-8d5c-0d3859001a0c admin admin] Validating token access rules against request {{(pid=81408) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:22:22.542988 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a732f7d5-b467-46b8-8a45-c78f40bcc1dc admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8ba823f313144da3b683bacef8fd62e0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81408) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:22:22.543585 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a732f7d5-b467-46b8-8a45-c78f40bcc1dc admin admin] REQUEST_METHOD: `GET` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:22:22.543863 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a732f7d5-b467-46b8-8a45-c78f40bcc1dc admin admin] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:22:22.544168 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a732f7d5-b467-46b8-8a45-c78f40bcc1dc admin admin] PATH_INFO: `/v3/roles` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:22:22.544829 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a732f7d5-b467-46b8-8a45-c78f40bcc1dc admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:22:22.546457 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a732f7d5-b467-46b8-8a45-c78f40bcc1dc admin admin] RBAC: Authorization granted {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:22:22.557451 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 872/1772] 10.222.0.27 () {58 vars in 1122 bytes} [Thu Oct 13 04:22:22 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 13 04:22:22.562102 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fc0162c4-940e-4bba-a55c-b80d97570cd4 admin admin] Authenticating user token {{(pid=81409) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:22:22.572429 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fc0162c4-940e-4bba-a55c-b80d97570cd4 admin admin] Validating token access rules against request {{(pid=81409) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:22:22.577651 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b0e08ab2-0f4d-40b2-968a-09cf96f94673 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8ba823f313144da3b683bacef8fd62e0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81409) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:22:22.578326 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0e08ab2-0f4d-40b2-968a-09cf96f94673 admin admin] REQUEST_METHOD: `PUT` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:22:22.578598 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0e08ab2-0f4d-40b2-968a-09cf96f94673 admin admin] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:22:22.578861 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0e08ab2-0f4d-40b2-968a-09cf96f94673 admin admin] PATH_INFO: `/v3/projects/b77a1f15f6394e17b156d44ae63ce37b/users/ebc8b2c33998434cab1aa8195131e5cf/roles/ef55fbd94e1040be83695754075a7ed5` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:22:22.593700 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b0e08ab2-0f4d-40b2-968a-09cf96f94673 admin admin] RBAC: Authorizing `identity:create_grant(project_id=b77a1f15f6394e17b156d44ae63ce37b, user_id=ebc8b2c33998434cab1aa8195131e5cf, role_id=ef55fbd94e1040be83695754075a7ed5)` {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:22:22.596441 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b0e08ab2-0f4d-40b2-968a-09cf96f94673 admin admin] RBAC: Authorization granted {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:22:22.609950 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 901/1773] 10.222.0.27 () {60 vars in 1483 bytes} [Thu Oct 13 04:22:22 2022] PUT /identity/v3/projects/b77a1f15f6394e17b156d44ae63ce37b/users/ebc8b2c33998434cab1aa8195131e5cf/roles/ef55fbd94e1040be83695754075a7ed5 => generated 0 bytes in 48 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 13 04:22:22.614282 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a732f7d5-b467-46b8-8a45-c78f40bcc1dc admin admin] Authenticating user token {{(pid=81408) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:22:22.625113 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a732f7d5-b467-46b8-8a45-c78f40bcc1dc admin admin] Validating token access rules against request {{(pid=81408) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:22:22.681014 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e9672562-4064-4632-91f6-425c40384f76 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8ba823f313144da3b683bacef8fd62e0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81408) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:22:22.681610 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9672562-4064-4632-91f6-425c40384f76 admin admin] REQUEST_METHOD: `GET` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:22:22.681888 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9672562-4064-4632-91f6-425c40384f76 admin admin] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:22:22.682152 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9672562-4064-4632-91f6-425c40384f76 admin admin] PATH_INFO: `/v3/roles` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:22:22.682809 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e9672562-4064-4632-91f6-425c40384f76 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:22:22.684388 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e9672562-4064-4632-91f6-425c40384f76 admin admin] RBAC: Authorization granted {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:22:22.695707 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 873/1774] 10.222.0.27 () {58 vars in 1122 bytes} [Thu Oct 13 04:22:22 2022] GET /identity/v3/roles => generated 1420 bytes in 82 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 13 04:22:22.700147 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b0e08ab2-0f4d-40b2-968a-09cf96f94673 admin admin] Authenticating user token {{(pid=81409) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:22:22.711303 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b0e08ab2-0f4d-40b2-968a-09cf96f94673 admin admin] Validating token access rules against request {{(pid=81409) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:22:22.715813 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-875ce73e-da2c-44df-b9a1-a25cf8a7afda admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8ba823f313144da3b683bacef8fd62e0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81409) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:22:22.716727 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-875ce73e-da2c-44df-b9a1-a25cf8a7afda admin admin] REQUEST_METHOD: `PUT` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:22:22.716998 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-875ce73e-da2c-44df-b9a1-a25cf8a7afda admin admin] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:22:22.717260 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-875ce73e-da2c-44df-b9a1-a25cf8a7afda admin admin] PATH_INFO: `/v3/projects/b77a1f15f6394e17b156d44ae63ce37b/users/ebc8b2c33998434cab1aa8195131e5cf/roles/ef55fbd94e1040be83695754075a7ed5` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:22:22.719164 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-875ce73e-da2c-44df-b9a1-a25cf8a7afda admin admin] RBAC: Authorizing `identity:create_grant(project_id=b77a1f15f6394e17b156d44ae63ce37b, user_id=ebc8b2c33998434cab1aa8195131e5cf, role_id=ef55fbd94e1040be83695754075a7ed5)` {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:22:22.721822 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-875ce73e-da2c-44df-b9a1-a25cf8a7afda admin admin] RBAC: Authorization granted {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:22:22.732987 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 902/1775] 10.222.0.27 () {60 vars in 1483 bytes} [Thu Oct 13 04:22:22 2022] PUT /identity/v3/projects/b77a1f15f6394e17b156d44ae63ce37b/users/ebc8b2c33998434cab1aa8195131e5cf/roles/ef55fbd94e1040be83695754075a7ed5 => generated 0 bytes in 33 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 13 04:22:22.744315 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22d525b4-596a-4036-81f5-c1fe452d30fc None None] REQUEST_METHOD: `POST` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:22:22.744665 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22d525b4-596a-4036-81f5-c1fe452d30fc None None] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:22:22.744939 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22d525b4-596a-4036-81f5-c1fe452d30fc None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:22:22.776375 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.auth.core [None req-22d525b4-596a-4036-81f5-c1fe452d30fc None None] MFA Rules not processed for user `ebc8b2c33998434cab1aa8195131e5cf`. Rule list: `[]` (Enabled: `True`). {{(pid=81408) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 13 04:22:22.777500 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.fernet_utils [None req-22d525b4-596a-4036-81f5-c1fe452d30fc None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81408) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:22:22.848931 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 874/1776] 10.222.0.27 () {58 vars in 958 bytes} [Thu Oct 13 04:22:22 2022] POST /identity/v3/auth/tokens => generated 3323 bytes in 106 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 13 04:22:22.860792 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-875ce73e-da2c-44df-b9a1-a25cf8a7afda admin admin] Authenticating user token {{(pid=81409) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:22:22.862487 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.fernet_utils [None req-875ce73e-da2c-44df-b9a1-a25cf8a7afda admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81409) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:22:22.925899 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-875ce73e-da2c-44df-b9a1-a25cf8a7afda admin admin] Validating token access rules against request {{(pid=81409) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:22:22.929748 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a9f3a73e-728a-4cd6-a9fa-8ad91b48d6d8 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6141b8a6c3d54e179d6879bb019f4d12', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dcec0431fb6849499d6aec35a234daa6', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81409) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:22:22.930404 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9f3a73e-728a-4cd6-a9fa-8ad91b48d6d8 service cinder] REQUEST_METHOD: `GET` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:22:22.930679 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9f3a73e-728a-4cd6-a9fa-8ad91b48d6d8 service cinder] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:22:22.930951 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9f3a73e-728a-4cd6-a9fa-8ad91b48d6d8 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:22:22.947611 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a9f3a73e-728a-4cd6-a9fa-8ad91b48d6d8 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:22:22.949061 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a9f3a73e-728a-4cd6-a9fa-8ad91b48d6d8 service cinder] RBAC: Authorization granted {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:22:22.955360 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG dbcounter [-] [81409] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=81409) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:22:22.964230 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 903/1777] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Oct 13 04:22:22 2022] GET /identity/v3/auth/tokens => generated 3323 bytes in 104 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 13 04:22:23.046838 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-22d525b4-596a-4036-81f5-c1fe452d30fc None None] Authenticating user token {{(pid=81408) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:22:23.056798 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-22d525b4-596a-4036-81f5-c1fe452d30fc None None] Validating token access rules against request {{(pid=81408) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:22:23.060729 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f36f3e88-cb04-4636-85e5-e28fbfe59ee4 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6141b8a6c3d54e179d6879bb019f4d12', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dcec0431fb6849499d6aec35a234daa6', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81408) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:22:23.061340 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f36f3e88-cb04-4636-85e5-e28fbfe59ee4 service cinder] REQUEST_METHOD: `GET` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:22:23.061617 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f36f3e88-cb04-4636-85e5-e28fbfe59ee4 service cinder] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:22:23.061887 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f36f3e88-cb04-4636-85e5-e28fbfe59ee4 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:22:23.083209 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG dbcounter [-] [81408] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81408) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:22:23.123228 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f36f3e88-cb04-4636-85e5-e28fbfe59ee4 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:22:23.124666 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f36f3e88-cb04-4636-85e5-e28fbfe59ee4 service cinder] RBAC: Authorization granted {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:22:23.134000 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 875/1778] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Oct 13 04:22:23 2022] GET /identity/v3/auth/tokens => generated 3258 bytes in 88 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 13 04:22:33.134515 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG dbcounter [-] [81408] Writing DB stats keystone:SELECT=18 {{(pid=81408) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:22:37.374573 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a9f3a73e-728a-4cd6-a9fa-8ad91b48d6d8 service cinder] Authenticating user token {{(pid=81409) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:22:37.441964 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a9f3a73e-728a-4cd6-a9fa-8ad91b48d6d8 service cinder] Validating token access rules against request {{(pid=81409) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:22:37.446744 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-efd127c5-c5d4-4a6e-986a-63253a0b6002 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8ba823f313144da3b683bacef8fd62e0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81409) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:22:37.447378 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-efd127c5-c5d4-4a6e-986a-63253a0b6002 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:22:37.447654 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-efd127c5-c5d4-4a6e-986a-63253a0b6002 admin admin] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:22:37.448062 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-efd127c5-c5d4-4a6e-986a-63253a0b6002 admin admin] PATH_INFO: `/v3/users/666598590c6b40aa9ad02ecf6a9a63f5` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:22:37.449190 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-efd127c5-c5d4-4a6e-986a-63253a0b6002 admin admin] RBAC: Authorizing `identity:delete_user(user_id=666598590c6b40aa9ad02ecf6a9a63f5)` {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:22:37.451193 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-efd127c5-c5d4-4a6e-986a-63253a0b6002 admin admin] RBAC: Authorization granted {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:22:37.500208 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.notifications [None req-efd127c5-c5d4-4a6e-986a-63253a0b6002 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '666598590c6b40aa9ad02ecf6a9a63f5'} {{(pid=81409) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 13 04:22:37.500926 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.notifications [None req-efd127c5-c5d4-4a6e-986a-63253a0b6002 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '666598590c6b40aa9ad02ecf6a9a63f5'} {{(pid=81409) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 13 04:22:37.508029 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.notifications [None req-efd127c5-c5d4-4a6e-986a-63253a0b6002 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '666598590c6b40aa9ad02ecf6a9a63f5'} {{(pid=81409) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 13 04:22:37.508594 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.notifications [None req-efd127c5-c5d4-4a6e-986a-63253a0b6002 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '666598590c6b40aa9ad02ecf6a9a63f5'} {{(pid=81409) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 13 04:22:37.517022 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.notifications [None req-efd127c5-c5d4-4a6e-986a-63253a0b6002 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '666598590c6b40aa9ad02ecf6a9a63f5'} {{(pid=81409) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 13 04:22:37.538145 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 904/1779] 10.222.0.27 () {58 vars in 1224 bytes} [Thu Oct 13 04:22:37 2022] DELETE /identity/v3/users/666598590c6b40aa9ad02ecf6a9a63f5 => generated 0 bytes in 164 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 13 04:22:37.542587 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-86627443-ec37-478d-be04-b31f2f21e224 None None] REQUEST_METHOD: `POST` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:22:37.545164 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-86627443-ec37-478d-be04-b31f2f21e224 None None] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:22:37.545445 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-86627443-ec37-478d-be04-b31f2f21e224 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:22:37.569782 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.auth.core [None req-86627443-ec37-478d-be04-b31f2f21e224 None None] MFA Rules not processed for user `0e2d979563d6421995451a0e86bf3022`. Rule list: `[]` (Enabled: `True`). {{(pid=81408) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 13 04:22:37.571031 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.fernet_utils [None req-86627443-ec37-478d-be04-b31f2f21e224 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81408) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:22:37.652882 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 876/1780] 10.222.0.27 () {58 vars in 958 bytes} [Thu Oct 13 04:22:37 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 111 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 13 04:22:37.673628 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-efd127c5-c5d4-4a6e-986a-63253a0b6002 admin admin] Authenticating user token {{(pid=81409) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:22:37.677024 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.fernet_utils [None req-efd127c5-c5d4-4a6e-986a-63253a0b6002 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81409) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:22:37.767968 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-efd127c5-c5d4-4a6e-986a-63253a0b6002 admin admin] Validating token access rules against request {{(pid=81409) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:22:37.773612 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-894122fc-7f8f-4155-ac7e-b7486cc22868 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd8812c3974f84871bb2d761cf0016257', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dcec0431fb6849499d6aec35a234daa6', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81409) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:22:37.774443 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-894122fc-7f8f-4155-ac7e-b7486cc22868 service neutron] REQUEST_METHOD: `GET` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:22:37.774789 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-894122fc-7f8f-4155-ac7e-b7486cc22868 service neutron] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:22:37.775130 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-894122fc-7f8f-4155-ac7e-b7486cc22868 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:22:37.786461 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-894122fc-7f8f-4155-ac7e-b7486cc22868 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:22:37.788035 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-894122fc-7f8f-4155-ac7e-b7486cc22868 service neutron] RBAC: Authorization granted {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:22:37.810661 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 905/1781] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Oct 13 04:22:37 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 137 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 13 04:22:38.157400 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-86627443-ec37-478d-be04-b31f2f21e224 None None] Authenticating user token {{(pid=81408) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:22:38.159101 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.fernet_utils [None req-86627443-ec37-478d-be04-b31f2f21e224 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81408) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:22:38.172321 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-86627443-ec37-478d-be04-b31f2f21e224 None None] Validating token access rules against request {{(pid=81408) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:22:38.181560 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8f9bc40d-4750-47dd-bf2b-e9dda05912c3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8ba823f313144da3b683bacef8fd62e0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81408) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:22:38.182275 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f9bc40d-4750-47dd-bf2b-e9dda05912c3 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:22:38.182572 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f9bc40d-4750-47dd-bf2b-e9dda05912c3 admin admin] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:22:38.182825 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f9bc40d-4750-47dd-bf2b-e9dda05912c3 admin admin] PATH_INFO: `/v3/projects/52f3ab53f6af494b8d6206784b97661e` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:22:38.183998 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8f9bc40d-4750-47dd-bf2b-e9dda05912c3 admin admin] RBAC: Authorizing `identity:delete_project(project_id=52f3ab53f6af494b8d6206784b97661e)` {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:22:38.186660 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8f9bc40d-4750-47dd-bf2b-e9dda05912c3 admin admin] RBAC: Authorization granted {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:22:38.239259 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.notifications [None req-8f9bc40d-4750-47dd-bf2b-e9dda05912c3 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '52f3ab53f6af494b8d6206784b97661e', 'request_id': 'req-8f9bc40d-4750-47dd-bf2b-e9dda05912c3'} {{(pid=81408) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 13 04:22:38.250696 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.notifications [None req-8f9bc40d-4750-47dd-bf2b-e9dda05912c3 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '52f3ab53f6af494b8d6206784b97661e', 'request_id': 'req-8f9bc40d-4750-47dd-bf2b-e9dda05912c3'} {{(pid=81408) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 13 04:22:38.257486 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.notifications [None req-8f9bc40d-4750-47dd-bf2b-e9dda05912c3 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '52f3ab53f6af494b8d6206784b97661e', 'request_id': 'req-8f9bc40d-4750-47dd-bf2b-e9dda05912c3'} {{(pid=81408) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 13 04:22:38.276869 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.notifications [None req-8f9bc40d-4750-47dd-bf2b-e9dda05912c3 admin admin] The token cache is being invalidate because project 52f3ab53f6af494b8d6206784b97661e was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81408) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 13 04:22:38.277163 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.notifications [None req-8f9bc40d-4750-47dd-bf2b-e9dda05912c3 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81408) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 13 04:22:38.277828 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.notifications [None req-8f9bc40d-4750-47dd-bf2b-e9dda05912c3 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81408) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 13 04:22:38.280760 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 877/1782] 10.222.0.27 () {58 vars in 1233 bytes} [Thu Oct 13 04:22:38 2022] DELETE /identity/v3/projects/52f3ab53f6af494b8d6206784b97661e => generated 0 bytes in 124 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 13 04:22:38.284717 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-894122fc-7f8f-4155-ac7e-b7486cc22868 service neutron] Authenticating user token {{(pid=81409) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:22:38.295790 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-894122fc-7f8f-4155-ac7e-b7486cc22868 service neutron] Validating token access rules against request {{(pid=81409) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:22:38.323675 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG dbcounter [-] [81409] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81409) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:22:38.368461 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-caabe3a1-075b-48be-bd64-82eda484c1de admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8ba823f313144da3b683bacef8fd62e0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81409) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:22:38.369103 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-caabe3a1-075b-48be-bd64-82eda484c1de admin admin] REQUEST_METHOD: `DELETE` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:22:38.369375 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-caabe3a1-075b-48be-bd64-82eda484c1de admin admin] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:22:38.369647 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-caabe3a1-075b-48be-bd64-82eda484c1de admin admin] PATH_INFO: `/v3/users/ebc8b2c33998434cab1aa8195131e5cf` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:22:38.370775 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-caabe3a1-075b-48be-bd64-82eda484c1de admin admin] RBAC: Authorizing `identity:delete_user(user_id=ebc8b2c33998434cab1aa8195131e5cf)` {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:22:38.373672 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-caabe3a1-075b-48be-bd64-82eda484c1de admin admin] RBAC: Authorization granted {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:22:38.432347 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.notifications [None req-caabe3a1-075b-48be-bd64-82eda484c1de admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'ebc8b2c33998434cab1aa8195131e5cf'} {{(pid=81409) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 13 04:22:38.432994 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.notifications [None req-caabe3a1-075b-48be-bd64-82eda484c1de admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'ebc8b2c33998434cab1aa8195131e5cf'} {{(pid=81409) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 13 04:22:38.441369 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.notifications [None req-caabe3a1-075b-48be-bd64-82eda484c1de admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'ebc8b2c33998434cab1aa8195131e5cf'} {{(pid=81409) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 13 04:22:38.442023 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.notifications [None req-caabe3a1-075b-48be-bd64-82eda484c1de admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'ebc8b2c33998434cab1aa8195131e5cf'} {{(pid=81409) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 13 04:22:38.450445 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.notifications [None req-caabe3a1-075b-48be-bd64-82eda484c1de admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'ebc8b2c33998434cab1aa8195131e5cf'} {{(pid=81409) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 13 04:22:38.470305 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 906/1783] 10.222.0.27 () {58 vars in 1224 bytes} [Thu Oct 13 04:22:38 2022] DELETE /identity/v3/users/ebc8b2c33998434cab1aa8195131e5cf => generated 0 bytes in 186 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 13 04:22:38.764743 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8f9bc40d-4750-47dd-bf2b-e9dda05912c3 admin admin] Authenticating user token {{(pid=81408) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:22:38.766787 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.fernet_utils [None req-8f9bc40d-4750-47dd-bf2b-e9dda05912c3 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81408) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:22:38.782055 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8f9bc40d-4750-47dd-bf2b-e9dda05912c3 admin admin] Validating token access rules against request {{(pid=81408) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:22:38.866846 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ffabc80c-3141-46a5-8d4a-12614a6465fd admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8ba823f313144da3b683bacef8fd62e0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81408) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:22:38.867613 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ffabc80c-3141-46a5-8d4a-12614a6465fd admin admin] REQUEST_METHOD: `DELETE` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:22:38.867959 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ffabc80c-3141-46a5-8d4a-12614a6465fd admin admin] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:22:38.868304 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ffabc80c-3141-46a5-8d4a-12614a6465fd admin admin] PATH_INFO: `/v3/projects/b77a1f15f6394e17b156d44ae63ce37b` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:22:38.869570 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ffabc80c-3141-46a5-8d4a-12614a6465fd admin admin] RBAC: Authorizing `identity:delete_project(project_id=b77a1f15f6394e17b156d44ae63ce37b)` {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:22:38.871650 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ffabc80c-3141-46a5-8d4a-12614a6465fd admin admin] RBAC: Authorization granted {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:22:38.905715 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG dbcounter [-] [81408] Writing DB stats keystone:SELECT=91,keystone:DELETE=8,keystone:INSERT=1 {{(pid=81408) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:22:38.929251 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.notifications [None req-ffabc80c-3141-46a5-8d4a-12614a6465fd admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'b77a1f15f6394e17b156d44ae63ce37b', 'request_id': 'req-ffabc80c-3141-46a5-8d4a-12614a6465fd'} {{(pid=81408) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 13 04:22:38.937420 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.notifications [None req-ffabc80c-3141-46a5-8d4a-12614a6465fd admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'b77a1f15f6394e17b156d44ae63ce37b', 'request_id': 'req-ffabc80c-3141-46a5-8d4a-12614a6465fd'} {{(pid=81408) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 13 04:22:38.944534 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.notifications [None req-ffabc80c-3141-46a5-8d4a-12614a6465fd admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'b77a1f15f6394e17b156d44ae63ce37b', 'request_id': 'req-ffabc80c-3141-46a5-8d4a-12614a6465fd'} {{(pid=81408) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 13 04:22:38.955318 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.notifications [None req-ffabc80c-3141-46a5-8d4a-12614a6465fd admin admin] The token cache is being invalidate because project b77a1f15f6394e17b156d44ae63ce37b was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81408) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 13 04:22:38.955642 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.notifications [None req-ffabc80c-3141-46a5-8d4a-12614a6465fd admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81408) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 13 04:22:38.956284 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.notifications [None req-ffabc80c-3141-46a5-8d4a-12614a6465fd admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81408) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 13 04:22:38.957935 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 878/1784] 10.222.0.27 () {58 vars in 1233 bytes} [Thu Oct 13 04:22:38 2022] DELETE /identity/v3/projects/b77a1f15f6394e17b156d44ae63ce37b => generated 0 bytes in 194 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 13 04:22:38.962679 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-127b335d-77de-410c-9106-d520488ed7d6 None None] REQUEST_METHOD: `POST` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:22:38.963038 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-127b335d-77de-410c-9106-d520488ed7d6 None None] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:22:38.963318 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-127b335d-77de-410c-9106-d520488ed7d6 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:22:38.983901 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.auth.core [None req-127b335d-77de-410c-9106-d520488ed7d6 None None] MFA Rules not processed for user `0e2d979563d6421995451a0e86bf3022`. Rule list: `[]` (Enabled: `True`). {{(pid=81409) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 13 04:22:38.985098 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.fernet_utils [None req-127b335d-77de-410c-9106-d520488ed7d6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81409) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:22:39.047946 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 907/1785] 10.222.0.27 () {58 vars in 958 bytes} [Thu Oct 13 04:22:38 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 86 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 13 04:22:39.055856 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f523b18-add5-4a73-a33d-af52db1ea2cd None None] REQUEST_METHOD: `POST` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:22:39.056138 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f523b18-add5-4a73-a33d-af52db1ea2cd None None] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:22:39.056409 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f523b18-add5-4a73-a33d-af52db1ea2cd None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:22:39.077269 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.auth.core [None req-8f523b18-add5-4a73-a33d-af52db1ea2cd None None] MFA Rules not processed for user `0e2d979563d6421995451a0e86bf3022`. Rule list: `[]` (Enabled: `True`). {{(pid=81408) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 13 04:22:39.078729 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.fernet_utils [None req-8f523b18-add5-4a73-a33d-af52db1ea2cd None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81408) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:22:39.087059 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 879/1786] 10.222.0.27 () {58 vars in 958 bytes} [Thu Oct 13 04:22:39 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 32 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 13 04:22:39.091913 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-127b335d-77de-410c-9106-d520488ed7d6 None None] Authenticating user token {{(pid=81409) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:22:39.102684 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-127b335d-77de-410c-9106-d520488ed7d6 None None] Validating token access rules against request {{(pid=81409) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:22:39.108964 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a23de5f4-9fa2-46d8-8e17-b301477d5edb admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8ba823f313144da3b683bacef8fd62e0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81409) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:22:39.109580 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a23de5f4-9fa2-46d8-8e17-b301477d5edb admin admin] REQUEST_METHOD: `GET` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:22:39.110116 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a23de5f4-9fa2-46d8-8e17-b301477d5edb admin admin] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:22:39.110116 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a23de5f4-9fa2-46d8-8e17-b301477d5edb admin admin] PATH_INFO: `/v3/domains` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:22:39.110921 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a23de5f4-9fa2-46d8-8e17-b301477d5edb admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81409) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 13 04:22:39.111423 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a23de5f4-9fa2-46d8-8e17-b301477d5edb admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:22:39.113015 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a23de5f4-9fa2-46d8-8e17-b301477d5edb admin admin] RBAC: Authorization granted {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:22:39.134897 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 908/1787] 10.222.0.27 () {58 vars in 1153 bytes} [Thu Oct 13 04:22:39 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 44 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 13 04:22:39.135635 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8f523b18-add5-4a73-a33d-af52db1ea2cd None None] Authenticating user token {{(pid=81408) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:22:39.146406 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8f523b18-add5-4a73-a33d-af52db1ea2cd None None] Validating token access rules against request {{(pid=81408) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:22:39.151869 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d2a82a08-bd69-405b-8c9f-5d6c033b8073 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8ba823f313144da3b683bacef8fd62e0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81408) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:22:39.152485 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2a82a08-bd69-405b-8c9f-5d6c033b8073 admin admin] REQUEST_METHOD: `POST` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:22:39.153255 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2a82a08-bd69-405b-8c9f-5d6c033b8073 admin admin] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:22:39.154046 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2a82a08-bd69-405b-8c9f-5d6c033b8073 admin admin] PATH_INFO: `/v3/projects` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:22:39.154905 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d2a82a08-bd69-405b-8c9f-5d6c033b8073 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:22:39.158367 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d2a82a08-bd69-405b-8c9f-5d6c033b8073 admin admin] RBAC: Authorization granted {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:22:39.196188 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 880/1788] 10.222.0.27 () {60 vars in 1153 bytes} [Thu Oct 13 04:22:39 2022] POST /identity/v3/projects => generated 375 bytes in 61 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 13 04:22:39.200265 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a23de5f4-9fa2-46d8-8e17-b301477d5edb admin admin] Authenticating user token {{(pid=81409) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:22:39.211429 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a23de5f4-9fa2-46d8-8e17-b301477d5edb admin admin] Validating token access rules against request {{(pid=81409) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:22:39.250417 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG dbcounter [-] [81409] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81409) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:22:39.299183 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d2c727e1-b95d-4dbd-b6cd-2f12e37b9dca admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8ba823f313144da3b683bacef8fd62e0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81409) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:22:39.299818 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2c727e1-b95d-4dbd-b6cd-2f12e37b9dca admin admin] REQUEST_METHOD: `POST` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:22:39.300098 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2c727e1-b95d-4dbd-b6cd-2f12e37b9dca admin admin] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:22:39.300371 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2c727e1-b95d-4dbd-b6cd-2f12e37b9dca admin admin] PATH_INFO: `/v3/users` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:22:39.301073 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d2c727e1-b95d-4dbd-b6cd-2f12e37b9dca admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:22:39.303543 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d2c727e1-b95d-4dbd-b6cd-2f12e37b9dca admin admin] RBAC: Authorization granted {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:22:39.336061 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 909/1789] 10.222.0.27 () {60 vars in 1144 bytes} [Thu Oct 13 04:22:39 2022] POST /identity/v3/users => generated 288 bytes in 135 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 13 04:22:39.339211 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d2a82a08-bd69-405b-8c9f-5d6c033b8073 admin admin] Authenticating user token {{(pid=81408) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:22:39.351605 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d2a82a08-bd69-405b-8c9f-5d6c033b8073 admin admin] Validating token access rules against request {{(pid=81408) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:22:39.357860 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1db637f6-7b06-4088-9399-2d83983a9a6f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8ba823f313144da3b683bacef8fd62e0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81408) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:22:39.358511 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1db637f6-7b06-4088-9399-2d83983a9a6f admin admin] REQUEST_METHOD: `GET` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:22:39.358790 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1db637f6-7b06-4088-9399-2d83983a9a6f admin admin] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:22:39.359062 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1db637f6-7b06-4088-9399-2d83983a9a6f admin admin] PATH_INFO: `/v3/roles` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:22:39.359800 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1db637f6-7b06-4088-9399-2d83983a9a6f admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:22:39.361424 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1db637f6-7b06-4088-9399-2d83983a9a6f admin admin] RBAC: Authorization granted {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:22:39.372391 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 881/1790] 10.222.0.27 () {58 vars in 1122 bytes} [Thu Oct 13 04:22:39 2022] GET /identity/v3/roles => generated 1420 bytes in 34 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 13 04:22:39.376859 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d2c727e1-b95d-4dbd-b6cd-2f12e37b9dca admin admin] Authenticating user token {{(pid=81409) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:22:39.388821 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d2c727e1-b95d-4dbd-b6cd-2f12e37b9dca admin admin] Validating token access rules against request {{(pid=81409) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:22:39.408514 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6d2cd0cb-c21a-405b-8da3-c387591332f6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8ba823f313144da3b683bacef8fd62e0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81409) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:22:39.408856 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d2cd0cb-c21a-405b-8da3-c387591332f6 admin admin] REQUEST_METHOD: `GET` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:22:39.408856 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d2cd0cb-c21a-405b-8da3-c387591332f6 admin admin] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:22:39.409280 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d2cd0cb-c21a-405b-8da3-c387591332f6 admin admin] PATH_INFO: `/v3/roles` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:22:39.410266 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6d2cd0cb-c21a-405b-8da3-c387591332f6 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:22:39.411426 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6d2cd0cb-c21a-405b-8da3-c387591332f6 admin admin] RBAC: Authorization granted {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:22:39.435002 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 910/1791] 10.222.0.27 () {58 vars in 1122 bytes} [Thu Oct 13 04:22:39 2022] GET /identity/v3/roles => generated 1420 bytes in 56 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 13 04:22:39.436379 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1db637f6-7b06-4088-9399-2d83983a9a6f admin admin] Authenticating user token {{(pid=81408) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:22:39.446646 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1db637f6-7b06-4088-9399-2d83983a9a6f admin admin] Validating token access rules against request {{(pid=81408) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:22:39.452016 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4301fba0-2435-4d90-bb79-a3431bcfc1ae admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8ba823f313144da3b683bacef8fd62e0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81408) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:22:39.452668 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4301fba0-2435-4d90-bb79-a3431bcfc1ae admin admin] REQUEST_METHOD: `PUT` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:22:39.452943 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4301fba0-2435-4d90-bb79-a3431bcfc1ae admin admin] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:22:39.453214 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4301fba0-2435-4d90-bb79-a3431bcfc1ae admin admin] PATH_INFO: `/v3/projects/cadec20ad4334d1d942a95c19c8b6cc6/users/774c1931571e4f9c9797e9f06414126b/roles/cc56c4ae13a7424ea771c15da4b153c1` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:22:39.469286 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4301fba0-2435-4d90-bb79-a3431bcfc1ae admin admin] RBAC: Authorizing `identity:create_grant(project_id=cadec20ad4334d1d942a95c19c8b6cc6, user_id=774c1931571e4f9c9797e9f06414126b, role_id=cc56c4ae13a7424ea771c15da4b153c1)` {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:22:39.472909 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4301fba0-2435-4d90-bb79-a3431bcfc1ae admin admin] RBAC: Authorization granted {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:22:39.488362 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 882/1792] 10.222.0.27 () {60 vars in 1483 bytes} [Thu Oct 13 04:22:39 2022] PUT /identity/v3/projects/cadec20ad4334d1d942a95c19c8b6cc6/users/774c1931571e4f9c9797e9f06414126b/roles/cc56c4ae13a7424ea771c15da4b153c1 => generated 0 bytes in 52 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 13 04:22:39.499876 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2bff8621-5b3b-4755-868b-77774b6b4851 None None] REQUEST_METHOD: `POST` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:22:39.500286 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2bff8621-5b3b-4755-868b-77774b6b4851 None None] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:22:39.500628 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2bff8621-5b3b-4755-868b-77774b6b4851 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:22:39.532069 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.auth.core [None req-2bff8621-5b3b-4755-868b-77774b6b4851 None None] MFA Rules not processed for user `774c1931571e4f9c9797e9f06414126b`. Rule list: `[]` (Enabled: `True`). {{(pid=81409) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 13 04:22:39.533354 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.fernet_utils [None req-2bff8621-5b3b-4755-868b-77774b6b4851 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81409) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:22:39.600820 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 911/1793] 10.222.0.27 () {58 vars in 958 bytes} [Thu Oct 13 04:22:39 2022] POST /identity/v3/auth/tokens => generated 3252 bytes in 102 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 13 04:22:39.605354 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4301fba0-2435-4d90-bb79-a3431bcfc1ae admin admin] Authenticating user token {{(pid=81408) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:22:39.670483 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4301fba0-2435-4d90-bb79-a3431bcfc1ae admin admin] Validating token access rules against request {{(pid=81408) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:22:39.676344 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9f72e3d8-3791-4feb-9c62-117557e9b101 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8ba823f313144da3b683bacef8fd62e0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81408) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:22:39.677028 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f72e3d8-3791-4feb-9c62-117557e9b101 admin admin] REQUEST_METHOD: `POST` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:22:39.677308 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f72e3d8-3791-4feb-9c62-117557e9b101 admin admin] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:22:39.677573 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f72e3d8-3791-4feb-9c62-117557e9b101 admin admin] PATH_INFO: `/v3/projects` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:22:39.678223 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9f72e3d8-3791-4feb-9c62-117557e9b101 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:22:39.680147 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9f72e3d8-3791-4feb-9c62-117557e9b101 admin admin] RBAC: Authorization granted {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:22:39.707242 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 883/1794] 10.222.0.27 () {60 vars in 1153 bytes} [Thu Oct 13 04:22:39 2022] POST /identity/v3/projects => generated 373 bytes in 102 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 13 04:22:39.711279 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2bff8621-5b3b-4755-868b-77774b6b4851 None None] Authenticating user token {{(pid=81409) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:22:39.722926 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2bff8621-5b3b-4755-868b-77774b6b4851 None None] Validating token access rules against request {{(pid=81409) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:22:39.778986 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dd7dc15f-e18a-4336-8d63-e824e74f58df admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8ba823f313144da3b683bacef8fd62e0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81409) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:22:39.779582 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd7dc15f-e18a-4336-8d63-e824e74f58df admin admin] REQUEST_METHOD: `POST` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:22:39.779852 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd7dc15f-e18a-4336-8d63-e824e74f58df admin admin] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:22:39.780116 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd7dc15f-e18a-4336-8d63-e824e74f58df admin admin] PATH_INFO: `/v3/users` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:22:39.780747 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dd7dc15f-e18a-4336-8d63-e824e74f58df admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:22:39.782664 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dd7dc15f-e18a-4336-8d63-e824e74f58df admin admin] RBAC: Authorization granted {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:22:39.802873 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 912/1795] 10.222.0.27 () {60 vars in 1144 bytes} [Thu Oct 13 04:22:39 2022] POST /identity/v3/users => generated 293 bytes in 92 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 13 04:22:39.807122 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9f72e3d8-3791-4feb-9c62-117557e9b101 admin admin] Authenticating user token {{(pid=81408) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:22:39.818181 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9f72e3d8-3791-4feb-9c62-117557e9b101 admin admin] Validating token access rules against request {{(pid=81408) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:22:39.822795 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0b9803fc-1300-4f17-8f86-402ac9116bd8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8ba823f313144da3b683bacef8fd62e0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81408) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:22:39.823394 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b9803fc-1300-4f17-8f86-402ac9116bd8 admin admin] REQUEST_METHOD: `GET` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:22:39.823668 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b9803fc-1300-4f17-8f86-402ac9116bd8 admin admin] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:22:39.823938 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b9803fc-1300-4f17-8f86-402ac9116bd8 admin admin] PATH_INFO: `/v3/roles` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:22:39.824574 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0b9803fc-1300-4f17-8f86-402ac9116bd8 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:22:39.826845 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0b9803fc-1300-4f17-8f86-402ac9116bd8 admin admin] RBAC: Authorization granted {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:22:39.837687 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 884/1796] 10.222.0.27 () {58 vars in 1122 bytes} [Thu Oct 13 04:22:39 2022] GET /identity/v3/roles => generated 1420 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Oct 13 04:22:39.841902 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dd7dc15f-e18a-4336-8d63-e824e74f58df admin admin] Authenticating user token {{(pid=81409) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:22:39.852175 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dd7dc15f-e18a-4336-8d63-e824e74f58df admin admin] Validating token access rules against request {{(pid=81409) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:22:39.860092 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-341be633-38d5-48e5-bda7-e6701d8784b5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8ba823f313144da3b683bacef8fd62e0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81409) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:22:39.860758 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-341be633-38d5-48e5-bda7-e6701d8784b5 admin admin] REQUEST_METHOD: `PUT` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:22:39.861034 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-341be633-38d5-48e5-bda7-e6701d8784b5 admin admin] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:22:39.861305 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-341be633-38d5-48e5-bda7-e6701d8784b5 admin admin] PATH_INFO: `/v3/projects/74551efd6e4b4080867563d9d8bb0f24/users/a302cccfb0f44ed9b93eff96f503e433/roles/ef55fbd94e1040be83695754075a7ed5` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:22:39.877286 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-341be633-38d5-48e5-bda7-e6701d8784b5 admin admin] RBAC: Authorizing `identity:create_grant(project_id=74551efd6e4b4080867563d9d8bb0f24, user_id=a302cccfb0f44ed9b93eff96f503e433, role_id=ef55fbd94e1040be83695754075a7ed5)` {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:22:39.879997 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-341be633-38d5-48e5-bda7-e6701d8784b5 admin admin] RBAC: Authorization granted {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:22:39.895234 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 913/1797] 10.222.0.27 () {60 vars in 1483 bytes} [Thu Oct 13 04:22:39 2022] PUT /identity/v3/projects/74551efd6e4b4080867563d9d8bb0f24/users/a302cccfb0f44ed9b93eff96f503e433/roles/ef55fbd94e1040be83695754075a7ed5 => generated 0 bytes in 54 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 13 04:22:39.899283 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0b9803fc-1300-4f17-8f86-402ac9116bd8 admin admin] Authenticating user token {{(pid=81408) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:22:39.909678 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0b9803fc-1300-4f17-8f86-402ac9116bd8 admin admin] Validating token access rules against request {{(pid=81408) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:22:39.931326 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG dbcounter [-] [81408] Writing DB stats keystone:DELETE=6,keystone:SELECT=90,keystone:INSERT=4 {{(pid=81408) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:22:39.975312 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-42bb9e9a-d921-4f93-8e17-c6d32390b8d6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8ba823f313144da3b683bacef8fd62e0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81408) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:22:39.975945 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42bb9e9a-d921-4f93-8e17-c6d32390b8d6 admin admin] REQUEST_METHOD: `GET` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:22:39.976246 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42bb9e9a-d921-4f93-8e17-c6d32390b8d6 admin admin] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:22:39.976493 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42bb9e9a-d921-4f93-8e17-c6d32390b8d6 admin admin] PATH_INFO: `/v3/roles` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:22:39.977135 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-42bb9e9a-d921-4f93-8e17-c6d32390b8d6 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:22:39.978895 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-42bb9e9a-d921-4f93-8e17-c6d32390b8d6 admin admin] RBAC: Authorization granted {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:22:39.990962 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 885/1798] 10.222.0.27 () {58 vars in 1122 bytes} [Thu Oct 13 04:22:39 2022] GET /identity/v3/roles => generated 1420 bytes in 92 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 13 04:22:39.995539 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-341be633-38d5-48e5-bda7-e6701d8784b5 admin admin] Authenticating user token {{(pid=81409) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:22:40.001079 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG dbcounter [-] [81409] Writing DB stats keystone:SELECT=93,keystone:INSERT=7 {{(pid=81409) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:22:40.007270 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-341be633-38d5-48e5-bda7-e6701d8784b5 admin admin] Validating token access rules against request {{(pid=81409) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:22:40.012589 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0eaa1265-dfe3-42b2-a62c-f7915cf6c46e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8ba823f313144da3b683bacef8fd62e0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81409) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:22:40.013890 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0eaa1265-dfe3-42b2-a62c-f7915cf6c46e admin admin] REQUEST_METHOD: `PUT` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:22:40.014162 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0eaa1265-dfe3-42b2-a62c-f7915cf6c46e admin admin] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:22:40.014469 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0eaa1265-dfe3-42b2-a62c-f7915cf6c46e admin admin] PATH_INFO: `/v3/projects/74551efd6e4b4080867563d9d8bb0f24/users/a302cccfb0f44ed9b93eff96f503e433/roles/ef55fbd94e1040be83695754075a7ed5` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:22:40.016468 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0eaa1265-dfe3-42b2-a62c-f7915cf6c46e admin admin] RBAC: Authorizing `identity:create_grant(project_id=74551efd6e4b4080867563d9d8bb0f24, user_id=a302cccfb0f44ed9b93eff96f503e433, role_id=ef55fbd94e1040be83695754075a7ed5)` {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:22:40.019104 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0eaa1265-dfe3-42b2-a62c-f7915cf6c46e admin admin] RBAC: Authorization granted {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:22:40.031198 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 914/1799] 10.222.0.27 () {60 vars in 1483 bytes} [Thu Oct 13 04:22:39 2022] PUT /identity/v3/projects/74551efd6e4b4080867563d9d8bb0f24/users/a302cccfb0f44ed9b93eff96f503e433/roles/ef55fbd94e1040be83695754075a7ed5 => generated 0 bytes in 36 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 13 04:22:40.041787 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75236924-53b5-48ad-9deb-2d389aed7e20 None None] REQUEST_METHOD: `POST` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:22:40.042070 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75236924-53b5-48ad-9deb-2d389aed7e20 None None] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:22:40.042388 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75236924-53b5-48ad-9deb-2d389aed7e20 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:22:40.077820 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.auth.core [None req-75236924-53b5-48ad-9deb-2d389aed7e20 None None] MFA Rules not processed for user `a302cccfb0f44ed9b93eff96f503e433`. Rule list: `[]` (Enabled: `True`). {{(pid=81408) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 13 04:22:40.079050 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.fernet_utils [None req-75236924-53b5-48ad-9deb-2d389aed7e20 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81408) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:22:40.148664 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 886/1800] 10.222.0.27 () {58 vars in 958 bytes} [Thu Oct 13 04:22:40 2022] POST /identity/v3/auth/tokens => generated 3317 bytes in 108 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 13 04:22:40.159160 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0eaa1265-dfe3-42b2-a62c-f7915cf6c46e admin admin] Authenticating user token {{(pid=81409) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:22:40.160810 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.fernet_utils [None req-0eaa1265-dfe3-42b2-a62c-f7915cf6c46e admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81409) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:22:40.218616 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0eaa1265-dfe3-42b2-a62c-f7915cf6c46e admin admin] Validating token access rules against request {{(pid=81409) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:22:40.222245 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c7e87d43-8a25-4692-a299-2948d2570742 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6141b8a6c3d54e179d6879bb019f4d12', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dcec0431fb6849499d6aec35a234daa6', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81409) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:22:40.222843 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7e87d43-8a25-4692-a299-2948d2570742 service cinder] REQUEST_METHOD: `GET` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:22:40.223115 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7e87d43-8a25-4692-a299-2948d2570742 service cinder] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:22:40.223380 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7e87d43-8a25-4692-a299-2948d2570742 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:22:40.279711 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c7e87d43-8a25-4692-a299-2948d2570742 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:22:40.281145 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c7e87d43-8a25-4692-a299-2948d2570742 service cinder] RBAC: Authorization granted {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:22:40.290065 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 915/1801] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Oct 13 04:22:40 2022] GET /identity/v3/auth/tokens => generated 3252 bytes in 131 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 13 04:22:41.021994 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-75236924-53b5-48ad-9deb-2d389aed7e20 None None] Authenticating user token {{(pid=81408) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:22:41.023437 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.fernet_utils [None req-75236924-53b5-48ad-9deb-2d389aed7e20 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81408) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:22:41.091958 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-75236924-53b5-48ad-9deb-2d389aed7e20 None None] Validating token access rules against request {{(pid=81408) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:22:41.096538 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-53e3e704-3b3c-4250-b759-fed6fa674bc9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8ba823f313144da3b683bacef8fd62e0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81408) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:22:41.097345 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53e3e704-3b3c-4250-b759-fed6fa674bc9 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:22:41.097629 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53e3e704-3b3c-4250-b759-fed6fa674bc9 admin admin] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:22:41.097906 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53e3e704-3b3c-4250-b759-fed6fa674bc9 admin admin] PATH_INFO: `/v3/users/9cd41672e51247f487f557b1bdebf6d1` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:22:41.099073 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-53e3e704-3b3c-4250-b759-fed6fa674bc9 admin admin] RBAC: Authorizing `identity:delete_user(user_id=9cd41672e51247f487f557b1bdebf6d1)` {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:22:41.101017 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-53e3e704-3b3c-4250-b759-fed6fa674bc9 admin admin] RBAC: Authorization granted {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:22:41.146132 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.notifications [None req-53e3e704-3b3c-4250-b759-fed6fa674bc9 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '9cd41672e51247f487f557b1bdebf6d1'} {{(pid=81408) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 13 04:22:41.147396 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG dbcounter [-] [81408] Writing DB stats keystone:SELECT=93,keystone:DELETE=7 {{(pid=81408) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:22:41.156158 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.notifications [None req-53e3e704-3b3c-4250-b759-fed6fa674bc9 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '9cd41672e51247f487f557b1bdebf6d1'} {{(pid=81408) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 13 04:22:41.156861 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.notifications [None req-53e3e704-3b3c-4250-b759-fed6fa674bc9 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '9cd41672e51247f487f557b1bdebf6d1'} {{(pid=81408) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 13 04:22:41.157459 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.notifications [None req-53e3e704-3b3c-4250-b759-fed6fa674bc9 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '9cd41672e51247f487f557b1bdebf6d1'} {{(pid=81408) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 13 04:22:41.165041 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.notifications [None req-53e3e704-3b3c-4250-b759-fed6fa674bc9 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '9cd41672e51247f487f557b1bdebf6d1'} {{(pid=81408) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 13 04:22:41.183454 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 887/1802] 10.222.0.27 () {58 vars in 1224 bytes} [Thu Oct 13 04:22:41 2022] DELETE /identity/v3/users/9cd41672e51247f487f557b1bdebf6d1 => generated 0 bytes in 162 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 13 04:22:41.188685 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b65ad05-0363-4ee4-8d46-230219e3afe3 None None] REQUEST_METHOD: `POST` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:22:41.189150 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b65ad05-0363-4ee4-8d46-230219e3afe3 None None] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:22:41.189275 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b65ad05-0363-4ee4-8d46-230219e3afe3 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:22:41.208992 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.auth.core [None req-8b65ad05-0363-4ee4-8d46-230219e3afe3 None None] MFA Rules not processed for user `0e2d979563d6421995451a0e86bf3022`. Rule list: `[]` (Enabled: `True`). {{(pid=81409) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 13 04:22:41.210241 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.fernet_utils [None req-8b65ad05-0363-4ee4-8d46-230219e3afe3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81409) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:22:41.271610 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 916/1803] 10.222.0.27 () {58 vars in 958 bytes} [Thu Oct 13 04:22:41 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 84 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 13 04:22:41.285882 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-53e3e704-3b3c-4250-b759-fed6fa674bc9 admin admin] Authenticating user token {{(pid=81408) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:22:41.287560 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.fernet_utils [None req-53e3e704-3b3c-4250-b759-fed6fa674bc9 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81408) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:22:41.346564 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-53e3e704-3b3c-4250-b759-fed6fa674bc9 admin admin] Validating token access rules against request {{(pid=81408) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:22:41.350164 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bc26d0ee-2e39-4da7-a813-b28772f625b9 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd8812c3974f84871bb2d761cf0016257', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dcec0431fb6849499d6aec35a234daa6', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81408) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:22:41.350792 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc26d0ee-2e39-4da7-a813-b28772f625b9 service neutron] REQUEST_METHOD: `GET` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:22:41.351070 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc26d0ee-2e39-4da7-a813-b28772f625b9 service neutron] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:22:41.351340 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc26d0ee-2e39-4da7-a813-b28772f625b9 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:22:41.360617 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bc26d0ee-2e39-4da7-a813-b28772f625b9 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:22:41.362044 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bc26d0ee-2e39-4da7-a813-b28772f625b9 service neutron] RBAC: Authorization granted {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:22:41.371637 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 888/1804] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Oct 13 04:22:41 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 86 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 13 04:22:41.675159 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8b65ad05-0363-4ee4-8d46-230219e3afe3 None None] Authenticating user token {{(pid=81409) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:22:41.677727 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.fernet_utils [None req-8b65ad05-0363-4ee4-8d46-230219e3afe3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81409) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:22:41.697260 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8b65ad05-0363-4ee4-8d46-230219e3afe3 None None] Validating token access rules against request {{(pid=81409) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:22:41.703323 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a4d61b7c-550e-4b45-a097-52390cb0547a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8ba823f313144da3b683bacef8fd62e0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81409) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:22:41.704591 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4d61b7c-550e-4b45-a097-52390cb0547a admin admin] REQUEST_METHOD: `DELETE` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:22:41.704903 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4d61b7c-550e-4b45-a097-52390cb0547a admin admin] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:22:41.705223 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4d61b7c-550e-4b45-a097-52390cb0547a admin admin] PATH_INFO: `/v3/projects/4aac93da506f49528e48ef1c1d2afbfd` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:22:41.706475 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a4d61b7c-550e-4b45-a097-52390cb0547a admin admin] RBAC: Authorizing `identity:delete_project(project_id=4aac93da506f49528e48ef1c1d2afbfd)` {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:22:41.708790 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a4d61b7c-550e-4b45-a097-52390cb0547a admin admin] RBAC: Authorization granted {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:22:41.759108 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG dbcounter [-] [81409] Writing DB stats keystone:SELECT=96,keystone:INSERT=1,keystone:DELETE=3 {{(pid=81409) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:22:41.769778 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.notifications [None req-a4d61b7c-550e-4b45-a097-52390cb0547a admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '4aac93da506f49528e48ef1c1d2afbfd', 'request_id': 'req-a4d61b7c-550e-4b45-a097-52390cb0547a'} {{(pid=81409) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 13 04:22:41.781253 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.notifications [None req-a4d61b7c-550e-4b45-a097-52390cb0547a admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '4aac93da506f49528e48ef1c1d2afbfd', 'request_id': 'req-a4d61b7c-550e-4b45-a097-52390cb0547a'} {{(pid=81409) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 13 04:22:41.789859 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.notifications [None req-a4d61b7c-550e-4b45-a097-52390cb0547a admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '4aac93da506f49528e48ef1c1d2afbfd', 'request_id': 'req-a4d61b7c-550e-4b45-a097-52390cb0547a'} {{(pid=81409) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 13 04:22:41.802868 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.notifications [None req-a4d61b7c-550e-4b45-a097-52390cb0547a admin admin] The token cache is being invalidate because project 4aac93da506f49528e48ef1c1d2afbfd was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81409) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 13 04:22:41.803227 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.notifications [None req-a4d61b7c-550e-4b45-a097-52390cb0547a admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81409) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 13 04:22:41.803885 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.notifications [None req-a4d61b7c-550e-4b45-a097-52390cb0547a admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81409) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 13 04:22:41.805485 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 917/1805] 10.222.0.27 () {58 vars in 1233 bytes} [Thu Oct 13 04:22:41 2022] DELETE /identity/v3/projects/4aac93da506f49528e48ef1c1d2afbfd => generated 0 bytes in 131 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 13 04:22:46.008786 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bc26d0ee-2e39-4da7-a813-b28772f625b9 service neutron] Authenticating user token {{(pid=81408) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:22:46.010264 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.fernet_utils [None req-bc26d0ee-2e39-4da7-a813-b28772f625b9 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81408) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:22:46.069604 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bc26d0ee-2e39-4da7-a813-b28772f625b9 service neutron] Validating token access rules against request {{(pid=81408) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:22:46.122572 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2a06e5ca-06fe-4735-bc6f-0494a4682fbb service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6141b8a6c3d54e179d6879bb019f4d12', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dcec0431fb6849499d6aec35a234daa6', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81408) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:22:46.123191 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a06e5ca-06fe-4735-bc6f-0494a4682fbb service cinder] REQUEST_METHOD: `GET` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:22:46.123461 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a06e5ca-06fe-4735-bc6f-0494a4682fbb service cinder] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:22:46.123729 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a06e5ca-06fe-4735-bc6f-0494a4682fbb service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:22:46.125450 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.fernet_utils [None req-2a06e5ca-06fe-4735-bc6f-0494a4682fbb service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81408) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:22:46.163713 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG dbcounter [-] [81408] Writing DB stats keystone:SELECT=96,keystone:INSERT=1,keystone:DELETE=3 {{(pid=81408) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:22:46.189359 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2a06e5ca-06fe-4735-bc6f-0494a4682fbb service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:22:46.190896 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2a06e5ca-06fe-4735-bc6f-0494a4682fbb service cinder] RBAC: Authorization granted {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:22:46.200308 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 889/1806] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Oct 13 04:22:46 2022] GET /identity/v3/auth/tokens => generated 3317 bytes in 192 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 13 04:22:51.798494 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG dbcounter [-] [81409] Writing DB stats keystone:SELECT=10,keystone:DELETE=4,keystone:INSERT=1 {{(pid=81409) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:22:52.045484 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89841d28-d5b6-4bb7-a5e6-5d930c14d806 None None] REQUEST_METHOD: `POST` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:22:52.046001 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89841d28-d5b6-4bb7-a5e6-5d930c14d806 None None] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:22:52.046393 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89841d28-d5b6-4bb7-a5e6-5d930c14d806 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:22:52.067017 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.auth.core [None req-89841d28-d5b6-4bb7-a5e6-5d930c14d806 None None] MFA Rules not processed for user `9ee6eac2e6da4d83bd14324669dc7269`. Rule list: `[]` (Enabled: `True`). {{(pid=81409) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 13 04:22:52.068223 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.fernet_utils [None req-89841d28-d5b6-4bb7-a5e6-5d930c14d806 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81409) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:22:52.133063 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 918/1807] 10.222.0.27 () {58 vars in 958 bytes} [Thu Oct 13 04:22:52 2022] POST /identity/v3/auth/tokens => generated 3250 bytes in 89 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 13 04:22:56.115161 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2a06e5ca-06fe-4735-bc6f-0494a4682fbb service cinder] Authenticating user token {{(pid=81408) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:22:56.116587 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.fernet_utils [None req-2a06e5ca-06fe-4735-bc6f-0494a4682fbb service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81408) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:22:56.184271 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2a06e5ca-06fe-4735-bc6f-0494a4682fbb service cinder] Validating token access rules against request {{(pid=81408) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:22:56.194658 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-393818b7-10ae-419d-a854-72d19d39315c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8ba823f313144da3b683bacef8fd62e0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81408) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:22:56.195471 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-393818b7-10ae-419d-a854-72d19d39315c admin admin] REQUEST_METHOD: `DELETE` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:22:56.195583 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-393818b7-10ae-419d-a854-72d19d39315c admin admin] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:22:56.195883 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-393818b7-10ae-419d-a854-72d19d39315c admin admin] PATH_INFO: `/v3/users/ca7301586f634c90a52759f788ef0b46` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:22:56.197026 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-393818b7-10ae-419d-a854-72d19d39315c admin admin] RBAC: Authorizing `identity:delete_user(user_id=ca7301586f634c90a52759f788ef0b46)` {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:22:56.199140 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-393818b7-10ae-419d-a854-72d19d39315c admin admin] RBAC: Authorization granted {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:22:56.253191 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.notifications [None req-393818b7-10ae-419d-a854-72d19d39315c admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'ca7301586f634c90a52759f788ef0b46'} {{(pid=81408) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 13 04:22:56.260624 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.notifications [None req-393818b7-10ae-419d-a854-72d19d39315c admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'ca7301586f634c90a52759f788ef0b46'} {{(pid=81408) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 13 04:22:56.261300 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.notifications [None req-393818b7-10ae-419d-a854-72d19d39315c admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'ca7301586f634c90a52759f788ef0b46'} {{(pid=81408) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 13 04:22:56.261987 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.notifications [None req-393818b7-10ae-419d-a854-72d19d39315c admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'ca7301586f634c90a52759f788ef0b46'} {{(pid=81408) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 13 04:22:56.271628 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.notifications [None req-393818b7-10ae-419d-a854-72d19d39315c admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'ca7301586f634c90a52759f788ef0b46'} {{(pid=81408) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 13 04:22:56.293144 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 890/1808] 10.222.0.27 () {58 vars in 1224 bytes} [Thu Oct 13 04:22:56 2022] DELETE /identity/v3/users/ca7301586f634c90a52759f788ef0b46 => generated 0 bytes in 178 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 13 04:22:56.298365 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9727460f-e877-4edf-8bec-a7c49c138070 None None] REQUEST_METHOD: `POST` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:22:56.298838 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9727460f-e877-4edf-8bec-a7c49c138070 None None] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:22:56.298916 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9727460f-e877-4edf-8bec-a7c49c138070 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:22:56.321164 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.auth.core [None req-9727460f-e877-4edf-8bec-a7c49c138070 None None] MFA Rules not processed for user `0e2d979563d6421995451a0e86bf3022`. Rule list: `[]` (Enabled: `True`). {{(pid=81409) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 13 04:22:56.322752 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.fernet_utils [None req-9727460f-e877-4edf-8bec-a7c49c138070 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81409) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:22:56.420613 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 919/1809] 10.222.0.27 () {58 vars in 958 bytes} [Thu Oct 13 04:22:56 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 123 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 13 04:22:56.431562 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-393818b7-10ae-419d-a854-72d19d39315c admin admin] Authenticating user token {{(pid=81408) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:22:56.435879 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.fernet_utils [None req-393818b7-10ae-419d-a854-72d19d39315c admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81408) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:22:56.506142 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-393818b7-10ae-419d-a854-72d19d39315c admin admin] Validating token access rules against request {{(pid=81408) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:22:56.511074 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-66b353cc-bafc-4154-9fd0-18685d6b07dd service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd8812c3974f84871bb2d761cf0016257', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dcec0431fb6849499d6aec35a234daa6', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81408) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:22:56.511693 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66b353cc-bafc-4154-9fd0-18685d6b07dd service neutron] REQUEST_METHOD: `GET` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:22:56.511968 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66b353cc-bafc-4154-9fd0-18685d6b07dd service neutron] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:22:56.512236 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66b353cc-bafc-4154-9fd0-18685d6b07dd service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:22:56.519531 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG dbcounter [-] [81408] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81408) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:22:56.522027 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-66b353cc-bafc-4154-9fd0-18685d6b07dd service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:22:56.523532 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-66b353cc-bafc-4154-9fd0-18685d6b07dd service neutron] RBAC: Authorization granted {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:22:56.534421 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 891/1810] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Oct 13 04:22:56 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 103 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 13 04:22:56.835042 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9727460f-e877-4edf-8bec-a7c49c138070 None None] Authenticating user token {{(pid=81409) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:22:56.836590 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.fernet_utils [None req-9727460f-e877-4edf-8bec-a7c49c138070 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81409) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:22:56.851679 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9727460f-e877-4edf-8bec-a7c49c138070 None None] Validating token access rules against request {{(pid=81409) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:22:56.856833 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dd58dda9-a3e7-4cde-ac89-80e2d9c3bafa admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8ba823f313144da3b683bacef8fd62e0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81409) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:22:56.857554 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd58dda9-a3e7-4cde-ac89-80e2d9c3bafa admin admin] REQUEST_METHOD: `DELETE` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:22:56.857905 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd58dda9-a3e7-4cde-ac89-80e2d9c3bafa admin admin] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:22:56.858275 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd58dda9-a3e7-4cde-ac89-80e2d9c3bafa admin admin] PATH_INFO: `/v3/projects/188def90cec84de6b0527111babb5370` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:22:56.859632 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dd58dda9-a3e7-4cde-ac89-80e2d9c3bafa admin admin] RBAC: Authorizing `identity:delete_project(project_id=188def90cec84de6b0527111babb5370)` {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:22:56.861947 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dd58dda9-a3e7-4cde-ac89-80e2d9c3bafa admin admin] RBAC: Authorization granted {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:22:56.911592 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.notifications [None req-dd58dda9-a3e7-4cde-ac89-80e2d9c3bafa admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '188def90cec84de6b0527111babb5370', 'request_id': 'req-dd58dda9-a3e7-4cde-ac89-80e2d9c3bafa'} {{(pid=81409) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 13 04:22:56.920128 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.notifications [None req-dd58dda9-a3e7-4cde-ac89-80e2d9c3bafa admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '188def90cec84de6b0527111babb5370', 'request_id': 'req-dd58dda9-a3e7-4cde-ac89-80e2d9c3bafa'} {{(pid=81409) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 13 04:22:56.927719 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.notifications [None req-dd58dda9-a3e7-4cde-ac89-80e2d9c3bafa admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '188def90cec84de6b0527111babb5370', 'request_id': 'req-dd58dda9-a3e7-4cde-ac89-80e2d9c3bafa'} {{(pid=81409) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 13 04:22:56.939391 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.notifications [None req-dd58dda9-a3e7-4cde-ac89-80e2d9c3bafa admin admin] The token cache is being invalidate because project 188def90cec84de6b0527111babb5370 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81409) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 13 04:22:56.939680 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.notifications [None req-dd58dda9-a3e7-4cde-ac89-80e2d9c3bafa admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81409) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 13 04:22:56.940370 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.notifications [None req-dd58dda9-a3e7-4cde-ac89-80e2d9c3bafa admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81409) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 13 04:22:56.941891 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 920/1811] 10.222.0.27 () {58 vars in 1233 bytes} [Thu Oct 13 04:22:56 2022] DELETE /identity/v3/projects/188def90cec84de6b0527111babb5370 => generated 0 bytes in 108 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 13 04:22:56.946891 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95860332-86a2-45ed-9d9e-5a7ec0af04ff None None] REQUEST_METHOD: `POST` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:22:56.947206 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95860332-86a2-45ed-9d9e-5a7ec0af04ff None None] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:22:56.947510 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95860332-86a2-45ed-9d9e-5a7ec0af04ff None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:22:56.968028 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.auth.core [None req-95860332-86a2-45ed-9d9e-5a7ec0af04ff None None] MFA Rules not processed for user `0e2d979563d6421995451a0e86bf3022`. Rule list: `[]` (Enabled: `True`). {{(pid=81408) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 13 04:22:56.969237 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.fernet_utils [None req-95860332-86a2-45ed-9d9e-5a7ec0af04ff None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81408) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:22:57.034302 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 892/1812] 10.222.0.27 () {58 vars in 958 bytes} [Thu Oct 13 04:22:56 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 88 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 13 04:22:57.043069 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1fe80206-b15c-48c9-abcb-2ac4e91f9e0b None None] REQUEST_METHOD: `POST` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:22:57.043353 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1fe80206-b15c-48c9-abcb-2ac4e91f9e0b None None] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:22:57.043628 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1fe80206-b15c-48c9-abcb-2ac4e91f9e0b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:22:57.066316 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.auth.core [None req-1fe80206-b15c-48c9-abcb-2ac4e91f9e0b None None] MFA Rules not processed for user `0e2d979563d6421995451a0e86bf3022`. Rule list: `[]` (Enabled: `True`). {{(pid=81409) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 13 04:22:57.067542 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.fernet_utils [None req-1fe80206-b15c-48c9-abcb-2ac4e91f9e0b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81409) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:22:57.077279 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 921/1813] 10.222.0.27 () {58 vars in 958 bytes} [Thu Oct 13 04:22:57 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 35 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 13 04:22:57.081725 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-95860332-86a2-45ed-9d9e-5a7ec0af04ff None None] Authenticating user token {{(pid=81408) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:22:57.092242 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-95860332-86a2-45ed-9d9e-5a7ec0af04ff None None] Validating token access rules against request {{(pid=81408) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:22:57.097519 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-daed2f9f-8d9b-4765-b3dc-9ee037f0f73c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8ba823f313144da3b683bacef8fd62e0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81408) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:22:57.097691 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-daed2f9f-8d9b-4765-b3dc-9ee037f0f73c admin admin] REQUEST_METHOD: `GET` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:22:57.098668 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-daed2f9f-8d9b-4765-b3dc-9ee037f0f73c admin admin] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:22:57.098943 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-daed2f9f-8d9b-4765-b3dc-9ee037f0f73c admin admin] PATH_INFO: `/v3/domains` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:22:57.099613 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-daed2f9f-8d9b-4765-b3dc-9ee037f0f73c admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81408) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 13 04:22:57.099916 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-daed2f9f-8d9b-4765-b3dc-9ee037f0f73c admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:22:57.101546 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-daed2f9f-8d9b-4765-b3dc-9ee037f0f73c admin admin] RBAC: Authorization granted {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:22:57.114808 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 893/1814] 10.222.0.27 () {58 vars in 1153 bytes} [Thu Oct 13 04:22:57 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 33 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 13 04:22:57.119192 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1fe80206-b15c-48c9-abcb-2ac4e91f9e0b None None] Authenticating user token {{(pid=81409) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:22:57.129173 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1fe80206-b15c-48c9-abcb-2ac4e91f9e0b None None] Validating token access rules against request {{(pid=81409) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:22:57.133854 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-77106b83-c0b2-463c-84a1-e532dbb084a0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8ba823f313144da3b683bacef8fd62e0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81409) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:22:57.134695 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77106b83-c0b2-463c-84a1-e532dbb084a0 admin admin] REQUEST_METHOD: `POST` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:22:57.134974 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77106b83-c0b2-463c-84a1-e532dbb084a0 admin admin] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:22:57.135237 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77106b83-c0b2-463c-84a1-e532dbb084a0 admin admin] PATH_INFO: `/v3/projects` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:22:57.135990 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-77106b83-c0b2-463c-84a1-e532dbb084a0 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:22:57.137888 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-77106b83-c0b2-463c-84a1-e532dbb084a0 admin admin] RBAC: Authorization granted {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:22:57.163615 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG dbcounter [-] [81409] Writing DB stats keystone:SELECT=91,keystone:DELETE=7,keystone:INSERT=2 {{(pid=81409) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:22:57.176287 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 922/1815] 10.222.0.27 () {60 vars in 1153 bytes} [Thu Oct 13 04:22:57 2022] POST /identity/v3/projects => generated 371 bytes in 58 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 13 04:22:57.176731 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-daed2f9f-8d9b-4765-b3dc-9ee037f0f73c admin admin] Authenticating user token {{(pid=81408) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:22:57.187497 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-daed2f9f-8d9b-4765-b3dc-9ee037f0f73c admin admin] Validating token access rules against request {{(pid=81408) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:22:57.254774 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1c8ec8a4-5dbe-4cec-ae01-80eae202347e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8ba823f313144da3b683bacef8fd62e0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81408) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:22:57.255397 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c8ec8a4-5dbe-4cec-ae01-80eae202347e admin admin] REQUEST_METHOD: `POST` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:22:57.255758 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c8ec8a4-5dbe-4cec-ae01-80eae202347e admin admin] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:22:57.256039 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c8ec8a4-5dbe-4cec-ae01-80eae202347e admin admin] PATH_INFO: `/v3/users` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:22:57.258845 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1c8ec8a4-5dbe-4cec-ae01-80eae202347e admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:22:57.260854 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1c8ec8a4-5dbe-4cec-ae01-80eae202347e admin admin] RBAC: Authorization granted {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:22:57.286460 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 894/1816] 10.222.0.27 () {60 vars in 1144 bytes} [Thu Oct 13 04:22:57 2022] POST /identity/v3/users => generated 286 bytes in 110 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 13 04:22:57.291014 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-77106b83-c0b2-463c-84a1-e532dbb084a0 admin admin] Authenticating user token {{(pid=81409) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:22:57.309170 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-77106b83-c0b2-463c-84a1-e532dbb084a0 admin admin] Validating token access rules against request {{(pid=81409) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:22:57.315274 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1a053d0d-e2da-4cb5-a3d6-bd83df30336e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8ba823f313144da3b683bacef8fd62e0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81409) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:22:57.315908 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a053d0d-e2da-4cb5-a3d6-bd83df30336e admin admin] REQUEST_METHOD: `GET` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:22:57.316235 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a053d0d-e2da-4cb5-a3d6-bd83df30336e admin admin] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:22:57.316506 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a053d0d-e2da-4cb5-a3d6-bd83df30336e admin admin] PATH_INFO: `/v3/roles` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:22:57.317154 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1a053d0d-e2da-4cb5-a3d6-bd83df30336e admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:22:57.319275 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1a053d0d-e2da-4cb5-a3d6-bd83df30336e admin admin] RBAC: Authorization granted {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:22:57.333868 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 923/1817] 10.222.0.27 () {58 vars in 1122 bytes} [Thu Oct 13 04:22:57 2022] GET /identity/v3/roles => generated 1420 bytes in 41 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 13 04:22:57.335457 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1c8ec8a4-5dbe-4cec-ae01-80eae202347e admin admin] Authenticating user token {{(pid=81408) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:22:57.352713 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1c8ec8a4-5dbe-4cec-ae01-80eae202347e admin admin] Validating token access rules against request {{(pid=81408) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:22:57.358875 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b5e737dc-0f45-482d-8c48-8ebdcbff213a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8ba823f313144da3b683bacef8fd62e0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81408) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:22:57.359670 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5e737dc-0f45-482d-8c48-8ebdcbff213a admin admin] REQUEST_METHOD: `GET` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:22:57.360010 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5e737dc-0f45-482d-8c48-8ebdcbff213a admin admin] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:22:57.360348 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5e737dc-0f45-482d-8c48-8ebdcbff213a admin admin] PATH_INFO: `/v3/roles` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:22:57.361286 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b5e737dc-0f45-482d-8c48-8ebdcbff213a admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:22:57.362836 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b5e737dc-0f45-482d-8c48-8ebdcbff213a admin admin] RBAC: Authorization granted {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:22:57.379500 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 895/1818] 10.222.0.27 () {58 vars in 1122 bytes} [Thu Oct 13 04:22:57 2022] GET /identity/v3/roles => generated 1420 bytes in 44 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 13 04:22:57.380593 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1a053d0d-e2da-4cb5-a3d6-bd83df30336e admin admin] Authenticating user token {{(pid=81409) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:22:57.393645 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1a053d0d-e2da-4cb5-a3d6-bd83df30336e admin admin] Validating token access rules against request {{(pid=81409) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:22:57.399727 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0f695efa-0f01-4fd0-8a12-a60bb399d9a2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8ba823f313144da3b683bacef8fd62e0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81409) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:22:57.400435 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f695efa-0f01-4fd0-8a12-a60bb399d9a2 admin admin] REQUEST_METHOD: `PUT` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:22:57.400710 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f695efa-0f01-4fd0-8a12-a60bb399d9a2 admin admin] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:22:57.400982 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f695efa-0f01-4fd0-8a12-a60bb399d9a2 admin admin] PATH_INFO: `/v3/projects/fabffb6796a34773987a4f7e414352d0/users/242490d937324b31bc95ae1982ea0b39/roles/cc56c4ae13a7424ea771c15da4b153c1` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:22:57.419025 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0f695efa-0f01-4fd0-8a12-a60bb399d9a2 admin admin] RBAC: Authorizing `identity:create_grant(project_id=fabffb6796a34773987a4f7e414352d0, user_id=242490d937324b31bc95ae1982ea0b39, role_id=cc56c4ae13a7424ea771c15da4b153c1)` {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:22:57.422217 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0f695efa-0f01-4fd0-8a12-a60bb399d9a2 admin admin] RBAC: Authorization granted {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:22:57.441140 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b877c203-d683-45dd-ab40-89cbf1ae0245 None None] REQUEST_METHOD: `POST` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:22:57.441241 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b877c203-d683-45dd-ab40-89cbf1ae0245 None None] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:22:57.441536 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b877c203-d683-45dd-ab40-89cbf1ae0245 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:22:57.445615 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 924/1819] 10.222.0.27 () {60 vars in 1483 bytes} [Thu Oct 13 04:22:57 2022] PUT /identity/v3/projects/fabffb6796a34773987a4f7e414352d0/users/242490d937324b31bc95ae1982ea0b39/roles/cc56c4ae13a7424ea771c15da4b153c1 => generated 0 bytes in 56 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 13 04:22:57.473281 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.auth.core [None req-b877c203-d683-45dd-ab40-89cbf1ae0245 None None] MFA Rules not processed for user `0e2d979563d6421995451a0e86bf3022`. Rule list: `[]` (Enabled: `True`). {{(pid=81408) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 13 04:22:57.474500 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.fernet_utils [None req-b877c203-d683-45dd-ab40-89cbf1ae0245 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81408) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:22:57.546373 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 896/1820] 10.222.0.27 () {58 vars in 958 bytes} [Thu Oct 13 04:22:57 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 106 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 13 04:22:57.555865 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0f695efa-0f01-4fd0-8a12-a60bb399d9a2 admin admin] Authenticating user token {{(pid=81409) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:22:57.557752 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.fernet_utils [None req-0f695efa-0f01-4fd0-8a12-a60bb399d9a2 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81409) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:22:57.634853 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0f695efa-0f01-4fd0-8a12-a60bb399d9a2 admin admin] Validating token access rules against request {{(pid=81409) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:22:57.640417 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-35022f70-23d3-4e28-ad3c-33cdc56ae5ef service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd8812c3974f84871bb2d761cf0016257', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dcec0431fb6849499d6aec35a234daa6', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81409) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:22:57.641022 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35022f70-23d3-4e28-ad3c-33cdc56ae5ef service neutron] REQUEST_METHOD: `GET` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:22:57.641292 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35022f70-23d3-4e28-ad3c-33cdc56ae5ef service neutron] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:22:57.641558 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35022f70-23d3-4e28-ad3c-33cdc56ae5ef service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:22:57.652255 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-35022f70-23d3-4e28-ad3c-33cdc56ae5ef service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:22:57.653898 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-35022f70-23d3-4e28-ad3c-33cdc56ae5ef service neutron] RBAC: Authorization granted {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:22:57.664456 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 925/1821] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Oct 13 04:22:57 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 109 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 13 04:23:02.320212 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83357a43-2643-4462-a6e8-c068728a1cc1 None None] REQUEST_METHOD: `POST` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:23:02.320679 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83357a43-2643-4462-a6e8-c068728a1cc1 None None] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:23:02.320783 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83357a43-2643-4462-a6e8-c068728a1cc1 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:23:02.328469 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG dbcounter [-] [81408] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81408) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:23:02.356381 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.auth.core [None req-83357a43-2643-4462-a6e8-c068728a1cc1 None None] MFA Rules not processed for user `242490d937324b31bc95ae1982ea0b39`. Rule list: `[]` (Enabled: `True`). {{(pid=81408) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 13 04:23:02.358686 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.fernet_utils [None req-83357a43-2643-4462-a6e8-c068728a1cc1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81408) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:23:02.458648 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 897/1822] 10.222.0.27 () {58 vars in 958 bytes} [Thu Oct 13 04:23:02 2022] POST /identity/v3/auth/tokens => generated 3248 bytes in 139 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 13 04:23:02.463555 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-35022f70-23d3-4e28-ad3c-33cdc56ae5ef service neutron] Authenticating user token {{(pid=81409) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:23:02.484356 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-35022f70-23d3-4e28-ad3c-33cdc56ae5ef service neutron] Validating token access rules against request {{(pid=81409) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:23:02.490221 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b20e8c08-c176-4850-82a2-5e11b4038752 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8ba823f313144da3b683bacef8fd62e0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81409) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:23:02.498414 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b20e8c08-c176-4850-82a2-5e11b4038752 admin admin] REQUEST_METHOD: `POST` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:23:02.502669 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b20e8c08-c176-4850-82a2-5e11b4038752 admin admin] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:23:02.503022 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b20e8c08-c176-4850-82a2-5e11b4038752 admin admin] PATH_INFO: `/v3/projects` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:23:02.503859 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b20e8c08-c176-4850-82a2-5e11b4038752 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:23:02.506278 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b20e8c08-c176-4850-82a2-5e11b4038752 admin admin] RBAC: Authorization granted {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:23:02.551169 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 926/1823] 10.222.0.27 () {60 vars in 1153 bytes} [Thu Oct 13 04:23:02 2022] POST /identity/v3/projects => generated 371 bytes in 88 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 13 04:23:02.555348 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-83357a43-2643-4462-a6e8-c068728a1cc1 None None] Authenticating user token {{(pid=81408) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:23:02.568381 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-83357a43-2643-4462-a6e8-c068728a1cc1 None None] Validating token access rules against request {{(pid=81408) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:23:02.635787 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-16bcf4b9-5192-492d-99c1-673726bb0abe admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8ba823f313144da3b683bacef8fd62e0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81408) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:23:02.636459 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16bcf4b9-5192-492d-99c1-673726bb0abe admin admin] REQUEST_METHOD: `POST` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:23:02.636739 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16bcf4b9-5192-492d-99c1-673726bb0abe admin admin] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:23:02.637207 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16bcf4b9-5192-492d-99c1-673726bb0abe admin admin] PATH_INFO: `/v3/users` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:23:02.638732 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-16bcf4b9-5192-492d-99c1-673726bb0abe admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:23:02.641050 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-16bcf4b9-5192-492d-99c1-673726bb0abe admin admin] RBAC: Authorization granted {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:23:02.670249 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 898/1824] 10.222.0.27 () {60 vars in 1144 bytes} [Thu Oct 13 04:23:02 2022] POST /identity/v3/users => generated 292 bytes in 115 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 13 04:23:02.673650 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b20e8c08-c176-4850-82a2-5e11b4038752 admin admin] Authenticating user token {{(pid=81409) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:23:02.684570 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b20e8c08-c176-4850-82a2-5e11b4038752 admin admin] Validating token access rules against request {{(pid=81409) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:23:02.697413 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7eac9d28-be52-4ef4-8e62-4c01af323b50 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8ba823f313144da3b683bacef8fd62e0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81409) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:23:02.698087 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7eac9d28-be52-4ef4-8e62-4c01af323b50 admin admin] REQUEST_METHOD: `GET` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:23:02.698433 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7eac9d28-be52-4ef4-8e62-4c01af323b50 admin admin] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:23:02.698725 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7eac9d28-be52-4ef4-8e62-4c01af323b50 admin admin] PATH_INFO: `/v3/roles` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:23:02.699392 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7eac9d28-be52-4ef4-8e62-4c01af323b50 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:23:02.701388 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7eac9d28-be52-4ef4-8e62-4c01af323b50 admin admin] RBAC: Authorization granted {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:23:02.713199 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 927/1825] 10.222.0.27 () {58 vars in 1122 bytes} [Thu Oct 13 04:23:02 2022] GET /identity/v3/roles => generated 1420 bytes in 40 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 13 04:23:02.717646 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-16bcf4b9-5192-492d-99c1-673726bb0abe admin admin] Authenticating user token {{(pid=81408) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:23:02.727813 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-16bcf4b9-5192-492d-99c1-673726bb0abe admin admin] Validating token access rules against request {{(pid=81408) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:23:02.733801 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8686eaaa-130d-46d0-93e3-74a4f6851009 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8ba823f313144da3b683bacef8fd62e0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81408) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:23:02.734497 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8686eaaa-130d-46d0-93e3-74a4f6851009 admin admin] REQUEST_METHOD: `PUT` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:23:02.734774 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8686eaaa-130d-46d0-93e3-74a4f6851009 admin admin] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:23:02.735042 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8686eaaa-130d-46d0-93e3-74a4f6851009 admin admin] PATH_INFO: `/v3/projects/dc5dcbaa1692443fb71e454c5d148bf8/users/4617d83c9dac4f939091c1abfef5eb17/roles/ef55fbd94e1040be83695754075a7ed5` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:23:02.751661 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8686eaaa-130d-46d0-93e3-74a4f6851009 admin admin] RBAC: Authorizing `identity:create_grant(project_id=dc5dcbaa1692443fb71e454c5d148bf8, user_id=4617d83c9dac4f939091c1abfef5eb17, role_id=ef55fbd94e1040be83695754075a7ed5)` {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:23:02.754454 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8686eaaa-130d-46d0-93e3-74a4f6851009 admin admin] RBAC: Authorization granted {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:23:02.769071 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 899/1826] 10.222.0.27 () {60 vars in 1483 bytes} [Thu Oct 13 04:23:02 2022] PUT /identity/v3/projects/dc5dcbaa1692443fb71e454c5d148bf8/users/4617d83c9dac4f939091c1abfef5eb17/roles/ef55fbd94e1040be83695754075a7ed5 => generated 0 bytes in 52 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 13 04:23:02.772929 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7eac9d28-be52-4ef4-8e62-4c01af323b50 admin admin] Authenticating user token {{(pid=81409) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:23:02.784530 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7eac9d28-be52-4ef4-8e62-4c01af323b50 admin admin] Validating token access rules against request {{(pid=81409) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:23:02.854735 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b25a6e24-a119-4c08-8b9c-b29bc70d56fc admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8ba823f313144da3b683bacef8fd62e0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81409) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:23:02.855337 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b25a6e24-a119-4c08-8b9c-b29bc70d56fc admin admin] REQUEST_METHOD: `GET` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:23:02.855674 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b25a6e24-a119-4c08-8b9c-b29bc70d56fc admin admin] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:23:02.856003 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b25a6e24-a119-4c08-8b9c-b29bc70d56fc admin admin] PATH_INFO: `/v3/roles` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:23:02.857547 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b25a6e24-a119-4c08-8b9c-b29bc70d56fc admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:23:02.859594 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b25a6e24-a119-4c08-8b9c-b29bc70d56fc admin admin] RBAC: Authorization granted {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:23:02.876291 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 928/1827] 10.222.0.27 () {58 vars in 1122 bytes} [Thu Oct 13 04:23:02 2022] GET /identity/v3/roles => generated 1420 bytes in 104 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 13 04:23:02.880724 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8686eaaa-130d-46d0-93e3-74a4f6851009 admin admin] Authenticating user token {{(pid=81408) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:23:02.892086 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8686eaaa-130d-46d0-93e3-74a4f6851009 admin admin] Validating token access rules against request {{(pid=81408) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:23:02.896838 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4c8590dc-874e-4968-970e-ca05c3776f25 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8ba823f313144da3b683bacef8fd62e0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81408) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:23:02.897501 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c8590dc-874e-4968-970e-ca05c3776f25 admin admin] REQUEST_METHOD: `PUT` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:23:02.897772 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c8590dc-874e-4968-970e-ca05c3776f25 admin admin] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:23:02.898039 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c8590dc-874e-4968-970e-ca05c3776f25 admin admin] PATH_INFO: `/v3/projects/dc5dcbaa1692443fb71e454c5d148bf8/users/4617d83c9dac4f939091c1abfef5eb17/roles/ef55fbd94e1040be83695754075a7ed5` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:23:02.899991 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4c8590dc-874e-4968-970e-ca05c3776f25 admin admin] RBAC: Authorizing `identity:create_grant(project_id=dc5dcbaa1692443fb71e454c5d148bf8, user_id=4617d83c9dac4f939091c1abfef5eb17, role_id=ef55fbd94e1040be83695754075a7ed5)` {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:23:02.902697 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4c8590dc-874e-4968-970e-ca05c3776f25 admin admin] RBAC: Authorization granted {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:23:02.914442 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 900/1828] 10.222.0.27 () {60 vars in 1483 bytes} [Thu Oct 13 04:23:02 2022] PUT /identity/v3/projects/dc5dcbaa1692443fb71e454c5d148bf8/users/4617d83c9dac4f939091c1abfef5eb17/roles/ef55fbd94e1040be83695754075a7ed5 => generated 0 bytes in 34 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 13 04:23:08.696578 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d01483b5-8262-44a9-bbc2-6ea529fe10fb None None] REQUEST_METHOD: `POST` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:23:08.697058 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d01483b5-8262-44a9-bbc2-6ea529fe10fb None None] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:23:08.697058 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d01483b5-8262-44a9-bbc2-6ea529fe10fb None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:23:08.735651 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.auth.core [None req-d01483b5-8262-44a9-bbc2-6ea529fe10fb None None] MFA Rules not processed for user `4617d83c9dac4f939091c1abfef5eb17`. Rule list: `[]` (Enabled: `True`). {{(pid=81409) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 13 04:23:08.737248 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.fernet_utils [None req-d01483b5-8262-44a9-bbc2-6ea529fe10fb None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81409) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:23:08.767972 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG dbcounter [-] [81409] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=81409) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:23:08.832552 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 929/1829] 10.222.0.27 () {58 vars in 958 bytes} [Thu Oct 13 04:23:08 2022] POST /identity/v3/auth/tokens => generated 3315 bytes in 137 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 13 04:23:08.846752 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4c8590dc-874e-4968-970e-ca05c3776f25 admin admin] Authenticating user token {{(pid=81408) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:23:08.848615 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.fernet_utils [None req-4c8590dc-874e-4968-970e-ca05c3776f25 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81408) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:23:08.900791 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG dbcounter [-] [81408] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=81408) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:23:08.934694 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4c8590dc-874e-4968-970e-ca05c3776f25 admin admin] Validating token access rules against request {{(pid=81408) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:23:08.940581 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-72e6dc39-83f1-4461-b1a9-570d8090e4c6 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cb740ed7f7684b079e45f887be83f795', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dcec0431fb6849499d6aec35a234daa6', 'project_domain_id': 'default', 'roles': ['ResellerAdmin', 'admin', 'reader', 'service', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81408) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:23:08.941202 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72e6dc39-83f1-4461-b1a9-570d8090e4c6 service nova] REQUEST_METHOD: `GET` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:23:08.941481 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72e6dc39-83f1-4461-b1a9-570d8090e4c6 service nova] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:23:08.941762 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72e6dc39-83f1-4461-b1a9-570d8090e4c6 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:23:09.003408 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-72e6dc39-83f1-4461-b1a9-570d8090e4c6 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:23:09.005179 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-72e6dc39-83f1-4461-b1a9-570d8090e4c6 service nova] RBAC: Authorization granted {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:23:09.029773 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 901/1830] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Oct 13 04:23:08 2022] GET /identity/v3/auth/tokens => generated 3248 bytes in 183 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 13 04:23:10.578119 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b899e4d-9172-4f96-87cf-af7c32af8174 None None] REQUEST_METHOD: `POST` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:23:10.582653 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b899e4d-9172-4f96-87cf-af7c32af8174 None None] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:23:10.583199 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b899e4d-9172-4f96-87cf-af7c32af8174 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:23:10.608348 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.auth.core [None req-5b899e4d-9172-4f96-87cf-af7c32af8174 None None] MFA Rules not processed for user `242490d937324b31bc95ae1982ea0b39`. Rule list: `[]` (Enabled: `True`). {{(pid=81409) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 13 04:23:10.609892 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.fernet_utils [None req-5b899e4d-9172-4f96-87cf-af7c32af8174 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81409) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:23:10.618163 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 930/1831] 10.222.0.27 () {58 vars in 958 bytes} [Thu Oct 13 04:23:10 2022] POST /identity/v3/auth/tokens => generated 3248 bytes in 42 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 13 04:23:12.307666 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-72e6dc39-83f1-4461-b1a9-570d8090e4c6 service nova] Authenticating user token {{(pid=81408) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:23:12.310484 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.fernet_utils [None req-72e6dc39-83f1-4461-b1a9-570d8090e4c6 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81408) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:23:12.430872 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-72e6dc39-83f1-4461-b1a9-570d8090e4c6 service nova] Validating token access rules against request {{(pid=81408) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:23:12.436988 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4073fe09-93d8-4fcd-8cad-37bfb3683ffc admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8ba823f313144da3b683bacef8fd62e0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81408) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:23:12.437843 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4073fe09-93d8-4fcd-8cad-37bfb3683ffc admin admin] REQUEST_METHOD: `DELETE` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:23:12.438276 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4073fe09-93d8-4fcd-8cad-37bfb3683ffc admin admin] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:23:12.438559 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4073fe09-93d8-4fcd-8cad-37bfb3683ffc admin admin] PATH_INFO: `/v3/users/774c1931571e4f9c9797e9f06414126b` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:23:12.439871 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4073fe09-93d8-4fcd-8cad-37bfb3683ffc admin admin] RBAC: Authorizing `identity:delete_user(user_id=774c1931571e4f9c9797e9f06414126b)` {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:23:12.441821 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4073fe09-93d8-4fcd-8cad-37bfb3683ffc admin admin] RBAC: Authorization granted {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:23:12.520036 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.notifications [None req-4073fe09-93d8-4fcd-8cad-37bfb3683ffc admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '774c1931571e4f9c9797e9f06414126b'} {{(pid=81408) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 13 04:23:12.534432 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.notifications [None req-4073fe09-93d8-4fcd-8cad-37bfb3683ffc admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '774c1931571e4f9c9797e9f06414126b'} {{(pid=81408) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 13 04:23:12.534432 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.notifications [None req-4073fe09-93d8-4fcd-8cad-37bfb3683ffc admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '774c1931571e4f9c9797e9f06414126b'} {{(pid=81408) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 13 04:23:12.534432 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.notifications [None req-4073fe09-93d8-4fcd-8cad-37bfb3683ffc admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '774c1931571e4f9c9797e9f06414126b'} {{(pid=81408) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 13 04:23:12.539721 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.notifications [None req-4073fe09-93d8-4fcd-8cad-37bfb3683ffc admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '774c1931571e4f9c9797e9f06414126b'} {{(pid=81408) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 13 04:23:12.558112 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG dbcounter [-] [81408] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81408) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:23:12.570978 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 902/1832] 10.222.0.27 () {58 vars in 1224 bytes} [Thu Oct 13 04:23:12 2022] DELETE /identity/v3/users/774c1931571e4f9c9797e9f06414126b => generated 0 bytes in 264 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 13 04:23:12.577106 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ecf8602e-6721-4a2c-871a-16816e16c95d None None] REQUEST_METHOD: `POST` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:23:12.577681 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ecf8602e-6721-4a2c-871a-16816e16c95d None None] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:23:12.577681 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ecf8602e-6721-4a2c-871a-16816e16c95d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:23:12.599064 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.auth.core [None req-ecf8602e-6721-4a2c-871a-16816e16c95d None None] MFA Rules not processed for user `0e2d979563d6421995451a0e86bf3022`. Rule list: `[]` (Enabled: `True`). {{(pid=81409) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 13 04:23:12.600494 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.fernet_utils [None req-ecf8602e-6721-4a2c-871a-16816e16c95d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81409) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:23:12.714844 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 931/1833] 10.222.0.27 () {58 vars in 958 bytes} [Thu Oct 13 04:23:12 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 140 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 13 04:23:12.736800 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4073fe09-93d8-4fcd-8cad-37bfb3683ffc admin admin] Authenticating user token {{(pid=81408) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:23:12.740222 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.fernet_utils [None req-4073fe09-93d8-4fcd-8cad-37bfb3683ffc admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81408) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:23:12.815191 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4073fe09-93d8-4fcd-8cad-37bfb3683ffc admin admin] Validating token access rules against request {{(pid=81408) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:23:12.819653 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-30741e5b-ee29-4b07-9e02-050767fdadbd service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd8812c3974f84871bb2d761cf0016257', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dcec0431fb6849499d6aec35a234daa6', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81408) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:23:12.820261 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30741e5b-ee29-4b07-9e02-050767fdadbd service neutron] REQUEST_METHOD: `GET` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:23:12.820533 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30741e5b-ee29-4b07-9e02-050767fdadbd service neutron] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:23:12.820816 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30741e5b-ee29-4b07-9e02-050767fdadbd service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:23:12.832284 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-30741e5b-ee29-4b07-9e02-050767fdadbd service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:23:12.833712 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-30741e5b-ee29-4b07-9e02-050767fdadbd service neutron] RBAC: Authorization granted {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:23:12.845573 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 903/1834] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Oct 13 04:23:12 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 110 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 13 04:23:13.161941 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ecf8602e-6721-4a2c-871a-16816e16c95d None None] Authenticating user token {{(pid=81409) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:23:13.167221 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.fernet_utils [None req-ecf8602e-6721-4a2c-871a-16816e16c95d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81409) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:23:13.193449 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ecf8602e-6721-4a2c-871a-16816e16c95d None None] Validating token access rules against request {{(pid=81409) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:23:13.203922 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-96fe7123-aa75-4bcd-a010-d64cc62d6b63 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8ba823f313144da3b683bacef8fd62e0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81409) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:23:13.204866 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96fe7123-aa75-4bcd-a010-d64cc62d6b63 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:23:13.205449 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96fe7123-aa75-4bcd-a010-d64cc62d6b63 admin admin] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:23:13.206014 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96fe7123-aa75-4bcd-a010-d64cc62d6b63 admin admin] PATH_INFO: `/v3/projects/cadec20ad4334d1d942a95c19c8b6cc6` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:23:13.207692 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-96fe7123-aa75-4bcd-a010-d64cc62d6b63 admin admin] RBAC: Authorizing `identity:delete_project(project_id=cadec20ad4334d1d942a95c19c8b6cc6)` {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:23:13.210265 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-96fe7123-aa75-4bcd-a010-d64cc62d6b63 admin admin] RBAC: Authorization granted {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:23:13.292068 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.notifications [None req-96fe7123-aa75-4bcd-a010-d64cc62d6b63 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'cadec20ad4334d1d942a95c19c8b6cc6', 'request_id': 'req-96fe7123-aa75-4bcd-a010-d64cc62d6b63'} {{(pid=81409) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 13 04:23:13.301674 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.notifications [None req-96fe7123-aa75-4bcd-a010-d64cc62d6b63 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'cadec20ad4334d1d942a95c19c8b6cc6', 'request_id': 'req-96fe7123-aa75-4bcd-a010-d64cc62d6b63'} {{(pid=81409) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 13 04:23:13.308864 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.notifications [None req-96fe7123-aa75-4bcd-a010-d64cc62d6b63 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'cadec20ad4334d1d942a95c19c8b6cc6', 'request_id': 'req-96fe7123-aa75-4bcd-a010-d64cc62d6b63'} {{(pid=81409) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 13 04:23:13.319780 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.notifications [None req-96fe7123-aa75-4bcd-a010-d64cc62d6b63 admin admin] The token cache is being invalidate because project cadec20ad4334d1d942a95c19c8b6cc6 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81409) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 13 04:23:13.320067 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.notifications [None req-96fe7123-aa75-4bcd-a010-d64cc62d6b63 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81409) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 13 04:23:13.320734 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.notifications [None req-96fe7123-aa75-4bcd-a010-d64cc62d6b63 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81409) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 13 04:23:13.322500 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 932/1835] 10.222.0.27 () {58 vars in 1233 bytes} [Thu Oct 13 04:23:13 2022] DELETE /identity/v3/projects/cadec20ad4334d1d942a95c19c8b6cc6 => generated 0 bytes in 161 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 13 04:23:13.326644 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-30741e5b-ee29-4b07-9e02-050767fdadbd service neutron] Authenticating user token {{(pid=81408) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:23:13.337338 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-30741e5b-ee29-4b07-9e02-050767fdadbd service neutron] Validating token access rules against request {{(pid=81408) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:23:13.402592 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4a84917c-a0fc-4552-b4a8-4a466614f97b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8ba823f313144da3b683bacef8fd62e0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81408) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:23:13.403274 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a84917c-a0fc-4552-b4a8-4a466614f97b admin admin] REQUEST_METHOD: `DELETE` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:23:13.403549 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a84917c-a0fc-4552-b4a8-4a466614f97b admin admin] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:23:13.403853 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a84917c-a0fc-4552-b4a8-4a466614f97b admin admin] PATH_INFO: `/v3/users/a302cccfb0f44ed9b93eff96f503e433` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:23:13.405063 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4a84917c-a0fc-4552-b4a8-4a466614f97b admin admin] RBAC: Authorizing `identity:delete_user(user_id=a302cccfb0f44ed9b93eff96f503e433)` {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:23:13.407207 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4a84917c-a0fc-4552-b4a8-4a466614f97b admin admin] RBAC: Authorization granted {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:23:13.462844 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.notifications [None req-4a84917c-a0fc-4552-b4a8-4a466614f97b admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'a302cccfb0f44ed9b93eff96f503e433'} {{(pid=81408) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 13 04:23:13.476617 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.notifications [None req-4a84917c-a0fc-4552-b4a8-4a466614f97b admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'a302cccfb0f44ed9b93eff96f503e433'} {{(pid=81408) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 13 04:23:13.477468 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.notifications [None req-4a84917c-a0fc-4552-b4a8-4a466614f97b admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'a302cccfb0f44ed9b93eff96f503e433'} {{(pid=81408) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 13 04:23:13.478090 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.notifications [None req-4a84917c-a0fc-4552-b4a8-4a466614f97b admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'a302cccfb0f44ed9b93eff96f503e433'} {{(pid=81408) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 13 04:23:13.487144 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.notifications [None req-4a84917c-a0fc-4552-b4a8-4a466614f97b admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'a302cccfb0f44ed9b93eff96f503e433'} {{(pid=81408) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 13 04:23:13.514268 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 904/1836] 10.222.0.27 () {58 vars in 1224 bytes} [Thu Oct 13 04:23:13 2022] DELETE /identity/v3/users/a302cccfb0f44ed9b93eff96f503e433 => generated 0 bytes in 184 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 13 04:23:13.984522 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-96fe7123-aa75-4bcd-a010-d64cc62d6b63 admin admin] Authenticating user token {{(pid=81409) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:23:13.986907 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.fernet_utils [None req-96fe7123-aa75-4bcd-a010-d64cc62d6b63 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81409) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:23:14.003779 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-96fe7123-aa75-4bcd-a010-d64cc62d6b63 admin admin] Validating token access rules against request {{(pid=81409) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:23:14.054439 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG dbcounter [-] [81409] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=81409) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:23:14.085512 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ec8091c8-75aa-4d3f-9622-e42feedcb501 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8ba823f313144da3b683bacef8fd62e0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81409) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:23:14.086628 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec8091c8-75aa-4d3f-9622-e42feedcb501 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:23:14.087095 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec8091c8-75aa-4d3f-9622-e42feedcb501 admin admin] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:23:14.087519 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec8091c8-75aa-4d3f-9622-e42feedcb501 admin admin] PATH_INFO: `/v3/projects/74551efd6e4b4080867563d9d8bb0f24` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:23:14.088785 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ec8091c8-75aa-4d3f-9622-e42feedcb501 admin admin] RBAC: Authorizing `identity:delete_project(project_id=74551efd6e4b4080867563d9d8bb0f24)` {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:23:14.091444 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ec8091c8-75aa-4d3f-9622-e42feedcb501 admin admin] RBAC: Authorization granted {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:23:14.145851 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.notifications [None req-ec8091c8-75aa-4d3f-9622-e42feedcb501 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '74551efd6e4b4080867563d9d8bb0f24', 'request_id': 'req-ec8091c8-75aa-4d3f-9622-e42feedcb501'} {{(pid=81409) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 13 04:23:14.153126 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.notifications [None req-ec8091c8-75aa-4d3f-9622-e42feedcb501 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '74551efd6e4b4080867563d9d8bb0f24', 'request_id': 'req-ec8091c8-75aa-4d3f-9622-e42feedcb501'} {{(pid=81409) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 13 04:23:14.160740 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.notifications [None req-ec8091c8-75aa-4d3f-9622-e42feedcb501 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '74551efd6e4b4080867563d9d8bb0f24', 'request_id': 'req-ec8091c8-75aa-4d3f-9622-e42feedcb501'} {{(pid=81409) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 13 04:23:14.175108 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.notifications [None req-ec8091c8-75aa-4d3f-9622-e42feedcb501 admin admin] The token cache is being invalidate because project 74551efd6e4b4080867563d9d8bb0f24 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81409) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 13 04:23:14.175392 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.notifications [None req-ec8091c8-75aa-4d3f-9622-e42feedcb501 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81409) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 13 04:23:14.176048 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.notifications [None req-ec8091c8-75aa-4d3f-9622-e42feedcb501 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81409) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 13 04:23:14.177551 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 933/1837] 10.222.0.27 () {58 vars in 1233 bytes} [Thu Oct 13 04:23:13 2022] DELETE /identity/v3/projects/74551efd6e4b4080867563d9d8bb0f24 => generated 0 bytes in 194 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 13 04:23:14.182864 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2a97f3e-f7e7-423b-8852-61ce1f6a1f5b None None] REQUEST_METHOD: `POST` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:23:14.183153 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2a97f3e-f7e7-423b-8852-61ce1f6a1f5b None None] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:23:14.183419 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2a97f3e-f7e7-423b-8852-61ce1f6a1f5b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:23:14.204212 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.auth.core [None req-e2a97f3e-f7e7-423b-8852-61ce1f6a1f5b None None] MFA Rules not processed for user `0e2d979563d6421995451a0e86bf3022`. Rule list: `[]` (Enabled: `True`). {{(pid=81408) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 13 04:23:14.205889 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.fernet_utils [None req-e2a97f3e-f7e7-423b-8852-61ce1f6a1f5b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81408) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:23:14.224564 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG dbcounter [-] [81408] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81408) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:23:14.272303 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 905/1838] 10.222.0.27 () {58 vars in 958 bytes} [Thu Oct 13 04:23:14 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 91 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 13 04:23:14.281899 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fcfafb33-a462-4cc7-9f01-d3775cab22ac None None] REQUEST_METHOD: `POST` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:23:14.282197 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fcfafb33-a462-4cc7-9f01-d3775cab22ac None None] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:23:14.282491 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fcfafb33-a462-4cc7-9f01-d3775cab22ac None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:23:14.304248 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.auth.core [None req-fcfafb33-a462-4cc7-9f01-d3775cab22ac None None] MFA Rules not processed for user `0e2d979563d6421995451a0e86bf3022`. Rule list: `[]` (Enabled: `True`). {{(pid=81409) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 13 04:23:14.305521 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.fernet_utils [None req-fcfafb33-a462-4cc7-9f01-d3775cab22ac None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81409) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:23:14.318049 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 934/1839] 10.222.0.27 () {58 vars in 958 bytes} [Thu Oct 13 04:23:14 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 34 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 13 04:23:14.319248 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e2a97f3e-f7e7-423b-8852-61ce1f6a1f5b None None] Authenticating user token {{(pid=81408) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:23:14.330648 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e2a97f3e-f7e7-423b-8852-61ce1f6a1f5b None None] Validating token access rules against request {{(pid=81408) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:23:14.335274 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8e4d0260-d21f-432c-be5b-7bb2b8fc564f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8ba823f313144da3b683bacef8fd62e0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81408) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:23:14.335952 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e4d0260-d21f-432c-be5b-7bb2b8fc564f admin admin] REQUEST_METHOD: `GET` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:23:14.336175 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e4d0260-d21f-432c-be5b-7bb2b8fc564f admin admin] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:23:14.336444 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e4d0260-d21f-432c-be5b-7bb2b8fc564f admin admin] PATH_INFO: `/v3/domains` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:23:14.337085 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8e4d0260-d21f-432c-be5b-7bb2b8fc564f admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81408) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 13 04:23:14.337379 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8e4d0260-d21f-432c-be5b-7bb2b8fc564f admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:23:14.338992 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8e4d0260-d21f-432c-be5b-7bb2b8fc564f admin admin] RBAC: Authorization granted {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:23:14.350939 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 906/1840] 10.222.0.27 () {58 vars in 1153 bytes} [Thu Oct 13 04:23:14 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 13 04:23:14.355586 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fcfafb33-a462-4cc7-9f01-d3775cab22ac None None] Authenticating user token {{(pid=81409) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:23:14.366590 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fcfafb33-a462-4cc7-9f01-d3775cab22ac None None] Validating token access rules against request {{(pid=81409) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:23:14.371487 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7c173122-097f-4acc-8229-f07feff579a6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8ba823f313144da3b683bacef8fd62e0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81409) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:23:14.372887 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c173122-097f-4acc-8229-f07feff579a6 admin admin] REQUEST_METHOD: `POST` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:23:14.373357 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c173122-097f-4acc-8229-f07feff579a6 admin admin] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:23:14.373639 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c173122-097f-4acc-8229-f07feff579a6 admin admin] PATH_INFO: `/v3/projects` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:23:14.374299 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7c173122-097f-4acc-8229-f07feff579a6 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:23:14.376879 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7c173122-097f-4acc-8229-f07feff579a6 admin admin] RBAC: Authorization granted {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:23:14.405510 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 935/1841] 10.222.0.27 () {60 vars in 1153 bytes} [Thu Oct 13 04:23:14 2022] POST /identity/v3/projects => generated 409 bytes in 50 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 13 04:23:14.411340 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8e4d0260-d21f-432c-be5b-7bb2b8fc564f admin admin] Authenticating user token {{(pid=81408) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:23:14.424589 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8e4d0260-d21f-432c-be5b-7bb2b8fc564f admin admin] Validating token access rules against request {{(pid=81408) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:23:14.480932 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6ba07d7c-91fb-4dec-858f-ad1b29fc40ab admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8ba823f313144da3b683bacef8fd62e0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81408) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:23:14.482068 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ba07d7c-91fb-4dec-858f-ad1b29fc40ab admin admin] REQUEST_METHOD: `POST` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:23:14.482386 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ba07d7c-91fb-4dec-858f-ad1b29fc40ab admin admin] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:23:14.482658 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ba07d7c-91fb-4dec-858f-ad1b29fc40ab admin admin] PATH_INFO: `/v3/users` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:23:14.483291 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6ba07d7c-91fb-4dec-858f-ad1b29fc40ab admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:23:14.485216 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6ba07d7c-91fb-4dec-858f-ad1b29fc40ab admin admin] RBAC: Authorization granted {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:23:14.508568 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 907/1842] 10.222.0.27 () {60 vars in 1144 bytes} [Thu Oct 13 04:23:14 2022] POST /identity/v3/users => generated 305 bytes in 98 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 13 04:23:14.512793 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7c173122-097f-4acc-8229-f07feff579a6 admin admin] Authenticating user token {{(pid=81409) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:23:14.523545 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7c173122-097f-4acc-8229-f07feff579a6 admin admin] Validating token access rules against request {{(pid=81409) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:23:14.528366 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d96da683-01fd-43c0-b5c4-439b49c57dc9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8ba823f313144da3b683bacef8fd62e0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81409) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:23:14.529065 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d96da683-01fd-43c0-b5c4-439b49c57dc9 admin admin] REQUEST_METHOD: `GET` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:23:14.529348 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d96da683-01fd-43c0-b5c4-439b49c57dc9 admin admin] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:23:14.529622 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d96da683-01fd-43c0-b5c4-439b49c57dc9 admin admin] PATH_INFO: `/v3/roles` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:23:14.530287 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d96da683-01fd-43c0-b5c4-439b49c57dc9 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:23:14.531883 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d96da683-01fd-43c0-b5c4-439b49c57dc9 admin admin] RBAC: Authorization granted {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:23:14.541757 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 936/1843] 10.222.0.27 () {58 vars in 1122 bytes} [Thu Oct 13 04:23:14 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 13 04:23:14.546327 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6ba07d7c-91fb-4dec-858f-ad1b29fc40ab admin admin] Authenticating user token {{(pid=81408) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:23:14.557435 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6ba07d7c-91fb-4dec-858f-ad1b29fc40ab admin admin] Validating token access rules against request {{(pid=81408) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:23:14.562968 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-86f3f52e-e69e-4edd-861b-6e0758252f55 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8ba823f313144da3b683bacef8fd62e0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81408) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:23:14.563570 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-86f3f52e-e69e-4edd-861b-6e0758252f55 admin admin] REQUEST_METHOD: `GET` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:23:14.563849 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-86f3f52e-e69e-4edd-861b-6e0758252f55 admin admin] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:23:14.564120 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-86f3f52e-e69e-4edd-861b-6e0758252f55 admin admin] PATH_INFO: `/v3/roles` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:23:14.564784 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-86f3f52e-e69e-4edd-861b-6e0758252f55 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:23:14.566414 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-86f3f52e-e69e-4edd-861b-6e0758252f55 admin admin] RBAC: Authorization granted {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:23:14.577339 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 908/1844] 10.222.0.27 () {58 vars in 1122 bytes} [Thu Oct 13 04:23:14 2022] GET /identity/v3/roles => generated 1420 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 13 04:23:14.581775 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d96da683-01fd-43c0-b5c4-439b49c57dc9 admin admin] Authenticating user token {{(pid=81409) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:23:14.592889 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d96da683-01fd-43c0-b5c4-439b49c57dc9 admin admin] Validating token access rules against request {{(pid=81409) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:23:14.598137 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a39da606-b04b-494d-b488-0f7b3249cccc admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8ba823f313144da3b683bacef8fd62e0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81409) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:23:14.598852 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a39da606-b04b-494d-b488-0f7b3249cccc admin admin] REQUEST_METHOD: `PUT` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:23:14.599197 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a39da606-b04b-494d-b488-0f7b3249cccc admin admin] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:23:14.599471 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a39da606-b04b-494d-b488-0f7b3249cccc admin admin] PATH_INFO: `/v3/projects/971b044253204f13b70277745a3c3b2e/users/f5aa0535833a44dc81dbc6efc7d1efb8/roles/cc56c4ae13a7424ea771c15da4b153c1` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:23:14.615046 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a39da606-b04b-494d-b488-0f7b3249cccc admin admin] RBAC: Authorizing `identity:create_grant(project_id=971b044253204f13b70277745a3c3b2e, user_id=f5aa0535833a44dc81dbc6efc7d1efb8, role_id=cc56c4ae13a7424ea771c15da4b153c1)` {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:23:14.617736 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a39da606-b04b-494d-b488-0f7b3249cccc admin admin] RBAC: Authorization granted {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:23:14.630616 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 937/1845] 10.222.0.27 () {60 vars in 1483 bytes} [Thu Oct 13 04:23:14 2022] PUT /identity/v3/projects/971b044253204f13b70277745a3c3b2e/users/f5aa0535833a44dc81dbc6efc7d1efb8/roles/cc56c4ae13a7424ea771c15da4b153c1 => generated 0 bytes in 49 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 13 04:23:14.641498 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8c98b1c-ecb9-41f1-8054-bb5447f63b73 None None] REQUEST_METHOD: `POST` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:23:14.641936 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8c98b1c-ecb9-41f1-8054-bb5447f63b73 None None] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:23:14.642242 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8c98b1c-ecb9-41f1-8054-bb5447f63b73 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:23:14.672316 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.auth.core [None req-a8c98b1c-ecb9-41f1-8054-bb5447f63b73 None None] MFA Rules not processed for user `f5aa0535833a44dc81dbc6efc7d1efb8`. Rule list: `[]` (Enabled: `True`). {{(pid=81408) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 13 04:23:14.673804 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.fernet_utils [None req-a8c98b1c-ecb9-41f1-8054-bb5447f63b73 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81408) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:23:14.744380 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 909/1846] 10.222.0.27 () {58 vars in 958 bytes} [Thu Oct 13 04:23:14 2022] POST /identity/v3/auth/tokens => generated 3286 bytes in 104 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 13 04:23:14.749542 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a39da606-b04b-494d-b488-0f7b3249cccc admin admin] Authenticating user token {{(pid=81409) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:23:14.823892 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a39da606-b04b-494d-b488-0f7b3249cccc admin admin] Validating token access rules against request {{(pid=81409) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:23:14.829951 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0b466ebf-1188-45aa-a4ab-8638477e3f81 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8ba823f313144da3b683bacef8fd62e0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81409) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:23:14.830755 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b466ebf-1188-45aa-a4ab-8638477e3f81 admin admin] REQUEST_METHOD: `POST` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:23:14.831037 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b466ebf-1188-45aa-a4ab-8638477e3f81 admin admin] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:23:14.831309 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b466ebf-1188-45aa-a4ab-8638477e3f81 admin admin] PATH_INFO: `/v3/projects` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:23:14.831986 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0b466ebf-1188-45aa-a4ab-8638477e3f81 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:23:14.834079 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0b466ebf-1188-45aa-a4ab-8638477e3f81 admin admin] RBAC: Authorization granted {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:23:14.842260 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG dbcounter [-] [81409] Writing DB stats keystone:SELECT=90,keystone:DELETE=7,keystone:INSERT=3 {{(pid=81409) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:23:14.879699 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 938/1847] 10.222.0.27 () {60 vars in 1153 bytes} [Thu Oct 13 04:23:14 2022] POST /identity/v3/projects => generated 407 bytes in 130 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 13 04:23:14.883327 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a8c98b1c-ecb9-41f1-8054-bb5447f63b73 None None] Authenticating user token {{(pid=81408) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:23:14.896055 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a8c98b1c-ecb9-41f1-8054-bb5447f63b73 None None] Validating token access rules against request {{(pid=81408) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:23:14.936616 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG dbcounter [-] [81408] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81408) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:23:15.002974 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-44e390df-d21d-46b2-a740-b260bdb00d29 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8ba823f313144da3b683bacef8fd62e0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81408) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:23:15.003652 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44e390df-d21d-46b2-a740-b260bdb00d29 admin admin] REQUEST_METHOD: `POST` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:23:15.003937 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44e390df-d21d-46b2-a740-b260bdb00d29 admin admin] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:23:15.004218 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44e390df-d21d-46b2-a740-b260bdb00d29 admin admin] PATH_INFO: `/v3/users` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:23:15.004900 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-44e390df-d21d-46b2-a740-b260bdb00d29 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:23:15.006881 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-44e390df-d21d-46b2-a740-b260bdb00d29 admin admin] RBAC: Authorization granted {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:23:15.037068 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 910/1848] 10.222.0.27 () {60 vars in 1144 bytes} [Thu Oct 13 04:23:14 2022] POST /identity/v3/users => generated 310 bytes in 152 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 13 04:23:15.038394 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0b466ebf-1188-45aa-a4ab-8638477e3f81 admin admin] Authenticating user token {{(pid=81409) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:23:15.049653 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0b466ebf-1188-45aa-a4ab-8638477e3f81 admin admin] Validating token access rules against request {{(pid=81409) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:23:15.054324 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d78903a9-f1da-438a-9384-0e03de8aaecd admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8ba823f313144da3b683bacef8fd62e0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81409) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:23:15.054917 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d78903a9-f1da-438a-9384-0e03de8aaecd admin admin] REQUEST_METHOD: `GET` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:23:15.055189 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d78903a9-f1da-438a-9384-0e03de8aaecd admin admin] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:23:15.055591 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d78903a9-f1da-438a-9384-0e03de8aaecd admin admin] PATH_INFO: `/v3/roles` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:23:15.056228 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d78903a9-f1da-438a-9384-0e03de8aaecd admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:23:15.057900 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d78903a9-f1da-438a-9384-0e03de8aaecd admin admin] RBAC: Authorization granted {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:23:15.079299 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 939/1849] 10.222.0.27 () {58 vars in 1122 bytes} [Thu Oct 13 04:23:15 2022] GET /identity/v3/roles => generated 1420 bytes in 41 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 13 04:23:15.083776 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-44e390df-d21d-46b2-a740-b260bdb00d29 admin admin] Authenticating user token {{(pid=81408) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:23:15.095719 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-44e390df-d21d-46b2-a740-b260bdb00d29 admin admin] Validating token access rules against request {{(pid=81408) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:23:15.101762 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-182ab346-5629-4ce0-9e6c-d461ed08fb00 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8ba823f313144da3b683bacef8fd62e0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81408) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:23:15.102540 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-182ab346-5629-4ce0-9e6c-d461ed08fb00 admin admin] REQUEST_METHOD: `PUT` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:23:15.102882 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-182ab346-5629-4ce0-9e6c-d461ed08fb00 admin admin] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:23:15.103221 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-182ab346-5629-4ce0-9e6c-d461ed08fb00 admin admin] PATH_INFO: `/v3/projects/e533f12763c7431dada530f2f195e532/users/4ee58f19e3fb4b7e89fb118e9169f64b/roles/ef55fbd94e1040be83695754075a7ed5` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:23:15.123337 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-182ab346-5629-4ce0-9e6c-d461ed08fb00 admin admin] RBAC: Authorizing `identity:create_grant(project_id=e533f12763c7431dada530f2f195e532, user_id=4ee58f19e3fb4b7e89fb118e9169f64b, role_id=ef55fbd94e1040be83695754075a7ed5)` {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:23:15.126103 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-182ab346-5629-4ce0-9e6c-d461ed08fb00 admin admin] RBAC: Authorization granted {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:23:15.143304 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 911/1850] 10.222.0.27 () {60 vars in 1483 bytes} [Thu Oct 13 04:23:15 2022] PUT /identity/v3/projects/e533f12763c7431dada530f2f195e532/users/4ee58f19e3fb4b7e89fb118e9169f64b/roles/ef55fbd94e1040be83695754075a7ed5 => generated 0 bytes in 60 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 13 04:23:15.147113 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d78903a9-f1da-438a-9384-0e03de8aaecd admin admin] Authenticating user token {{(pid=81409) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:23:15.166038 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d78903a9-f1da-438a-9384-0e03de8aaecd admin admin] Validating token access rules against request {{(pid=81409) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:23:15.244158 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-01a8df50-57fb-49ca-986d-bcdc1507fc4a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8ba823f313144da3b683bacef8fd62e0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81409) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:23:15.244858 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01a8df50-57fb-49ca-986d-bcdc1507fc4a admin admin] REQUEST_METHOD: `GET` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:23:15.245161 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01a8df50-57fb-49ca-986d-bcdc1507fc4a admin admin] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:23:15.245440 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01a8df50-57fb-49ca-986d-bcdc1507fc4a admin admin] PATH_INFO: `/v3/roles` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:23:15.247451 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-01a8df50-57fb-49ca-986d-bcdc1507fc4a admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:23:15.248874 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-01a8df50-57fb-49ca-986d-bcdc1507fc4a admin admin] RBAC: Authorization granted {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:23:15.260886 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 940/1851] 10.222.0.27 () {58 vars in 1122 bytes} [Thu Oct 13 04:23:15 2022] GET /identity/v3/roles => generated 1420 bytes in 114 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 13 04:23:15.267451 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-182ab346-5629-4ce0-9e6c-d461ed08fb00 admin admin] Authenticating user token {{(pid=81408) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:23:15.280796 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-182ab346-5629-4ce0-9e6c-d461ed08fb00 admin admin] Validating token access rules against request {{(pid=81408) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:23:15.286387 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ca1e2d08-f4bc-4e7e-a15b-255a3c200fe0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8ba823f313144da3b683bacef8fd62e0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81408) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:23:15.287130 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca1e2d08-f4bc-4e7e-a15b-255a3c200fe0 admin admin] REQUEST_METHOD: `PUT` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:23:15.287474 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca1e2d08-f4bc-4e7e-a15b-255a3c200fe0 admin admin] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:23:15.287760 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca1e2d08-f4bc-4e7e-a15b-255a3c200fe0 admin admin] PATH_INFO: `/v3/projects/e533f12763c7431dada530f2f195e532/users/4ee58f19e3fb4b7e89fb118e9169f64b/roles/ef55fbd94e1040be83695754075a7ed5` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:23:15.289934 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ca1e2d08-f4bc-4e7e-a15b-255a3c200fe0 admin admin] RBAC: Authorizing `identity:create_grant(project_id=e533f12763c7431dada530f2f195e532, user_id=4ee58f19e3fb4b7e89fb118e9169f64b, role_id=ef55fbd94e1040be83695754075a7ed5)` {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:23:15.292717 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ca1e2d08-f4bc-4e7e-a15b-255a3c200fe0 admin admin] RBAC: Authorization granted {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:23:15.303440 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 912/1852] 10.222.0.27 () {60 vars in 1483 bytes} [Thu Oct 13 04:23:15 2022] PUT /identity/v3/projects/e533f12763c7431dada530f2f195e532/users/4ee58f19e3fb4b7e89fb118e9169f64b/roles/ef55fbd94e1040be83695754075a7ed5 => generated 0 bytes in 36 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 13 04:23:15.314277 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f0d0d50-ac61-4927-a090-a29c853d0cd2 None None] REQUEST_METHOD: `POST` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:23:15.314588 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f0d0d50-ac61-4927-a090-a29c853d0cd2 None None] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:23:15.314880 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f0d0d50-ac61-4927-a090-a29c853d0cd2 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:23:15.345821 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.auth.core [None req-5f0d0d50-ac61-4927-a090-a29c853d0cd2 None None] MFA Rules not processed for user `4ee58f19e3fb4b7e89fb118e9169f64b`. Rule list: `[]` (Enabled: `True`). {{(pid=81409) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 13 04:23:15.346918 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.fernet_utils [None req-5f0d0d50-ac61-4927-a090-a29c853d0cd2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81409) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:23:15.452869 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 941/1853] 10.222.0.27 () {58 vars in 958 bytes} [Thu Oct 13 04:23:15 2022] POST /identity/v3/auth/tokens => generated 3351 bytes in 140 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 13 04:23:15.463652 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ca1e2d08-f4bc-4e7e-a15b-255a3c200fe0 admin admin] Authenticating user token {{(pid=81408) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:23:15.465505 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.fernet_utils [None req-ca1e2d08-f4bc-4e7e-a15b-255a3c200fe0 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81408) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:23:15.525986 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ca1e2d08-f4bc-4e7e-a15b-255a3c200fe0 admin admin] Validating token access rules against request {{(pid=81408) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:23:15.529772 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f78cba27-a4d2-4ed1-bc7c-bdbb67998e6e service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6141b8a6c3d54e179d6879bb019f4d12', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dcec0431fb6849499d6aec35a234daa6', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81408) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:23:15.530418 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f78cba27-a4d2-4ed1-bc7c-bdbb67998e6e service cinder] REQUEST_METHOD: `GET` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:23:15.530698 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f78cba27-a4d2-4ed1-bc7c-bdbb67998e6e service cinder] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:23:15.530973 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f78cba27-a4d2-4ed1-bc7c-bdbb67998e6e service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:23:15.540540 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f78cba27-a4d2-4ed1-bc7c-bdbb67998e6e service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:23:15.542126 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f78cba27-a4d2-4ed1-bc7c-bdbb67998e6e service cinder] RBAC: Authorization granted {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:23:15.552011 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 913/1854] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Oct 13 04:23:15 2022] GET /identity/v3/auth/tokens => generated 3351 bytes in 89 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 13 04:23:15.712001 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f0d0d50-ac61-4927-a090-a29c853d0cd2 None None] Authenticating user token {{(pid=81409) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:23:15.722109 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f0d0d50-ac61-4927-a090-a29c853d0cd2 None None] Validating token access rules against request {{(pid=81409) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:23:15.725628 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-316f5658-1aa2-4769-9901-2655055a9c47 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6141b8a6c3d54e179d6879bb019f4d12', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dcec0431fb6849499d6aec35a234daa6', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81409) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:23:15.726242 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-316f5658-1aa2-4769-9901-2655055a9c47 service cinder] REQUEST_METHOD: `GET` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:23:15.726514 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-316f5658-1aa2-4769-9901-2655055a9c47 service cinder] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:23:15.726777 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-316f5658-1aa2-4769-9901-2655055a9c47 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:23:15.782521 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG dbcounter [-] [81409] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81409) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:23:15.787583 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-316f5658-1aa2-4769-9901-2655055a9c47 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:23:15.789339 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-316f5658-1aa2-4769-9901-2655055a9c47 service cinder] RBAC: Authorization granted {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:23:15.799176 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 942/1855] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Oct 13 04:23:15 2022] GET /identity/v3/auth/tokens => generated 3286 bytes in 88 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 13 04:23:18.828167 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6e161e9-79dd-45a5-9d39-334845650715 None None] REQUEST_METHOD: `POST` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:23:18.828661 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6e161e9-79dd-45a5-9d39-334845650715 None None] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:23:18.829002 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6e161e9-79dd-45a5-9d39-334845650715 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:23:18.853039 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.auth.core [None req-c6e161e9-79dd-45a5-9d39-334845650715 None None] MFA Rules not processed for user `999c89e1bc7a426886fc802e28d38596`. Rule list: `[]` (Enabled: `True`). {{(pid=81408) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 13 04:23:18.855225 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.fernet_utils [None req-c6e161e9-79dd-45a5-9d39-334845650715 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81408) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:23:18.944986 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 914/1856] 10.222.0.27 () {58 vars in 958 bytes} [Thu Oct 13 04:23:18 2022] POST /identity/v3/auth/tokens => generated 3260 bytes in 118 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 13 04:23:19.658018 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1630dc70-f6e8-43f2-8f2f-25cdd6ba8b9d None None] REQUEST_METHOD: `POST` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:23:19.658605 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1630dc70-f6e8-43f2-8f2f-25cdd6ba8b9d None None] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:23:19.658605 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1630dc70-f6e8-43f2-8f2f-25cdd6ba8b9d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:23:19.688597 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.auth.core [None req-1630dc70-f6e8-43f2-8f2f-25cdd6ba8b9d None None] MFA Rules not processed for user `da9c75191a3b4ef49438aae9e35fcf13`. Rule list: `[]` (Enabled: `True`). {{(pid=81409) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 13 04:23:19.690590 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.fernet_utils [None req-1630dc70-f6e8-43f2-8f2f-25cdd6ba8b9d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81409) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:23:19.780566 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 943/1857] 10.222.0.27 () {58 vars in 958 bytes} [Thu Oct 13 04:23:19 2022] POST /identity/v3/auth/tokens => generated 3270 bytes in 124 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 13 04:23:24.844671 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c6e161e9-79dd-45a5-9d39-334845650715 None None] Authenticating user token {{(pid=81408) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:23:24.865234 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG dbcounter [-] [81408] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=81408) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:23:24.918867 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c6e161e9-79dd-45a5-9d39-334845650715 None None] Validating token access rules against request {{(pid=81408) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:23:24.924125 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a7b7eb52-4490-4e43-93fd-c918a2326169 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8ba823f313144da3b683bacef8fd62e0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81408) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:23:24.924844 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7b7eb52-4490-4e43-93fd-c918a2326169 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:23:24.925116 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7b7eb52-4490-4e43-93fd-c918a2326169 admin admin] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:23:24.925394 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7b7eb52-4490-4e43-93fd-c918a2326169 admin admin] PATH_INFO: `/v3/users/f5aa0535833a44dc81dbc6efc7d1efb8` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:23:24.926533 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a7b7eb52-4490-4e43-93fd-c918a2326169 admin admin] RBAC: Authorizing `identity:delete_user(user_id=f5aa0535833a44dc81dbc6efc7d1efb8)` {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:23:24.929805 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a7b7eb52-4490-4e43-93fd-c918a2326169 admin admin] RBAC: Authorization granted {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:23:24.978363 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.notifications [None req-a7b7eb52-4490-4e43-93fd-c918a2326169 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'f5aa0535833a44dc81dbc6efc7d1efb8'} {{(pid=81408) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 13 04:23:24.988309 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.notifications [None req-a7b7eb52-4490-4e43-93fd-c918a2326169 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'f5aa0535833a44dc81dbc6efc7d1efb8'} {{(pid=81408) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 13 04:23:24.989307 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.notifications [None req-a7b7eb52-4490-4e43-93fd-c918a2326169 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'f5aa0535833a44dc81dbc6efc7d1efb8'} {{(pid=81408) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 13 04:23:24.990139 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.notifications [None req-a7b7eb52-4490-4e43-93fd-c918a2326169 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'f5aa0535833a44dc81dbc6efc7d1efb8'} {{(pid=81408) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 13 04:23:24.998880 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.notifications [None req-a7b7eb52-4490-4e43-93fd-c918a2326169 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'f5aa0535833a44dc81dbc6efc7d1efb8'} {{(pid=81408) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 13 04:23:25.022990 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 915/1858] 10.222.0.27 () {58 vars in 1224 bytes} [Thu Oct 13 04:23:24 2022] DELETE /identity/v3/users/f5aa0535833a44dc81dbc6efc7d1efb8 => generated 0 bytes in 179 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 13 04:23:25.027610 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8d693b5-021c-472f-81db-5979601c74fa None None] REQUEST_METHOD: `POST` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:23:25.028109 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8d693b5-021c-472f-81db-5979601c74fa None None] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:23:25.028188 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8d693b5-021c-472f-81db-5979601c74fa None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:23:25.057081 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.auth.core [None req-a8d693b5-021c-472f-81db-5979601c74fa None None] MFA Rules not processed for user `0e2d979563d6421995451a0e86bf3022`. Rule list: `[]` (Enabled: `True`). {{(pid=81409) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 13 04:23:25.058343 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.fernet_utils [None req-a8d693b5-021c-472f-81db-5979601c74fa None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81409) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:23:25.130499 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 944/1859] 10.222.0.27 () {58 vars in 958 bytes} [Thu Oct 13 04:23:25 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 104 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 13 04:23:25.140340 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a7b7eb52-4490-4e43-93fd-c918a2326169 admin admin] Authenticating user token {{(pid=81408) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:23:25.142539 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.fernet_utils [None req-a7b7eb52-4490-4e43-93fd-c918a2326169 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81408) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:23:25.237831 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a7b7eb52-4490-4e43-93fd-c918a2326169 admin admin] Validating token access rules against request {{(pid=81408) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:23:25.243748 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-732d5b92-f311-4585-a75a-4f44589c6beb service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd8812c3974f84871bb2d761cf0016257', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dcec0431fb6849499d6aec35a234daa6', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81408) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:23:25.244964 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-732d5b92-f311-4585-a75a-4f44589c6beb service neutron] REQUEST_METHOD: `GET` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:23:25.245351 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-732d5b92-f311-4585-a75a-4f44589c6beb service neutron] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:23:25.247984 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-732d5b92-f311-4585-a75a-4f44589c6beb service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:23:25.261022 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-732d5b92-f311-4585-a75a-4f44589c6beb service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:23:25.262546 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-732d5b92-f311-4585-a75a-4f44589c6beb service neutron] RBAC: Authorization granted {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:23:25.277867 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 916/1860] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Oct 13 04:23:25 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 138 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 13 04:23:26.062664 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a8d693b5-021c-472f-81db-5979601c74fa None None] Authenticating user token {{(pid=81409) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:23:26.064630 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.fernet_utils [None req-a8d693b5-021c-472f-81db-5979601c74fa None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81409) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:23:26.078127 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a8d693b5-021c-472f-81db-5979601c74fa None None] Validating token access rules against request {{(pid=81409) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:23:26.083764 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8dd22d19-1980-401a-9219-30ac6236c2a2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8ba823f313144da3b683bacef8fd62e0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81409) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:23:26.084626 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8dd22d19-1980-401a-9219-30ac6236c2a2 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:23:26.084900 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8dd22d19-1980-401a-9219-30ac6236c2a2 admin admin] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:23:26.085322 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8dd22d19-1980-401a-9219-30ac6236c2a2 admin admin] PATH_INFO: `/v3/projects/971b044253204f13b70277745a3c3b2e` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:23:26.086698 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8dd22d19-1980-401a-9219-30ac6236c2a2 admin admin] RBAC: Authorizing `identity:delete_project(project_id=971b044253204f13b70277745a3c3b2e)` {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:23:26.088687 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8dd22d19-1980-401a-9219-30ac6236c2a2 admin admin] RBAC: Authorization granted {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:23:26.199339 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.notifications [None req-8dd22d19-1980-401a-9219-30ac6236c2a2 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '971b044253204f13b70277745a3c3b2e', 'request_id': 'req-8dd22d19-1980-401a-9219-30ac6236c2a2'} {{(pid=81409) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 13 04:23:26.207269 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.notifications [None req-8dd22d19-1980-401a-9219-30ac6236c2a2 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '971b044253204f13b70277745a3c3b2e', 'request_id': 'req-8dd22d19-1980-401a-9219-30ac6236c2a2'} {{(pid=81409) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 13 04:23:26.218411 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.notifications [None req-8dd22d19-1980-401a-9219-30ac6236c2a2 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '971b044253204f13b70277745a3c3b2e', 'request_id': 'req-8dd22d19-1980-401a-9219-30ac6236c2a2'} {{(pid=81409) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 13 04:23:26.243541 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.notifications [None req-8dd22d19-1980-401a-9219-30ac6236c2a2 admin admin] The token cache is being invalidate because project 971b044253204f13b70277745a3c3b2e was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81409) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 13 04:23:26.244008 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.notifications [None req-8dd22d19-1980-401a-9219-30ac6236c2a2 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81409) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 13 04:23:26.244837 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.notifications [None req-8dd22d19-1980-401a-9219-30ac6236c2a2 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81409) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 13 04:23:26.248704 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 945/1861] 10.222.0.27 () {58 vars in 1233 bytes} [Thu Oct 13 04:23:26 2022] DELETE /identity/v3/projects/971b044253204f13b70277745a3c3b2e => generated 0 bytes in 186 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 13 04:23:26.252245 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-732d5b92-f311-4585-a75a-4f44589c6beb service neutron] Authenticating user token {{(pid=81408) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:23:26.263845 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-732d5b92-f311-4585-a75a-4f44589c6beb service neutron] Validating token access rules against request {{(pid=81408) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:23:26.312348 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG dbcounter [-] [81408] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81408) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:23:26.335783 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-69cbb8f9-4035-4ed2-b6c1-f24d4952fe75 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8ba823f313144da3b683bacef8fd62e0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81408) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:23:26.336714 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69cbb8f9-4035-4ed2-b6c1-f24d4952fe75 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:23:26.337194 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69cbb8f9-4035-4ed2-b6c1-f24d4952fe75 admin admin] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:23:26.337619 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69cbb8f9-4035-4ed2-b6c1-f24d4952fe75 admin admin] PATH_INFO: `/v3/users/4ee58f19e3fb4b7e89fb118e9169f64b` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:23:26.339161 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-69cbb8f9-4035-4ed2-b6c1-f24d4952fe75 admin admin] RBAC: Authorizing `identity:delete_user(user_id=4ee58f19e3fb4b7e89fb118e9169f64b)` {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:23:26.341402 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-69cbb8f9-4035-4ed2-b6c1-f24d4952fe75 admin admin] RBAC: Authorization granted {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:23:26.409189 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.notifications [None req-69cbb8f9-4035-4ed2-b6c1-f24d4952fe75 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '4ee58f19e3fb4b7e89fb118e9169f64b'} {{(pid=81408) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 13 04:23:26.424486 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.notifications [None req-69cbb8f9-4035-4ed2-b6c1-f24d4952fe75 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '4ee58f19e3fb4b7e89fb118e9169f64b'} {{(pid=81408) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 13 04:23:26.425629 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.notifications [None req-69cbb8f9-4035-4ed2-b6c1-f24d4952fe75 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '4ee58f19e3fb4b7e89fb118e9169f64b'} {{(pid=81408) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 13 04:23:26.427003 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.notifications [None req-69cbb8f9-4035-4ed2-b6c1-f24d4952fe75 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '4ee58f19e3fb4b7e89fb118e9169f64b'} {{(pid=81408) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 13 04:23:26.444157 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.notifications [None req-69cbb8f9-4035-4ed2-b6c1-f24d4952fe75 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '4ee58f19e3fb4b7e89fb118e9169f64b'} {{(pid=81408) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 13 04:23:26.474238 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 917/1862] 10.222.0.27 () {58 vars in 1224 bytes} [Thu Oct 13 04:23:26 2022] DELETE /identity/v3/users/4ee58f19e3fb4b7e89fb118e9169f64b => generated 0 bytes in 223 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 13 04:23:26.943203 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8dd22d19-1980-401a-9219-30ac6236c2a2 admin admin] Authenticating user token {{(pid=81409) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:23:26.944743 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.fernet_utils [None req-8dd22d19-1980-401a-9219-30ac6236c2a2 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81409) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:23:26.960216 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8dd22d19-1980-401a-9219-30ac6236c2a2 admin admin] Validating token access rules against request {{(pid=81409) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:23:26.982821 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG dbcounter [-] [81409] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=81409) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:23:27.026303 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-06878f8e-f2e2-4909-b551-47fd6583a893 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8ba823f313144da3b683bacef8fd62e0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81409) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:23:27.027034 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06878f8e-f2e2-4909-b551-47fd6583a893 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:23:27.027308 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06878f8e-f2e2-4909-b551-47fd6583a893 admin admin] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:23:27.027599 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06878f8e-f2e2-4909-b551-47fd6583a893 admin admin] PATH_INFO: `/v3/projects/e533f12763c7431dada530f2f195e532` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:23:27.028935 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-06878f8e-f2e2-4909-b551-47fd6583a893 admin admin] RBAC: Authorizing `identity:delete_project(project_id=e533f12763c7431dada530f2f195e532)` {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:23:27.032533 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-06878f8e-f2e2-4909-b551-47fd6583a893 admin admin] RBAC: Authorization granted {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:23:27.105987 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.notifications [None req-06878f8e-f2e2-4909-b551-47fd6583a893 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'e533f12763c7431dada530f2f195e532', 'request_id': 'req-06878f8e-f2e2-4909-b551-47fd6583a893'} {{(pid=81409) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 13 04:23:27.116926 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.notifications [None req-06878f8e-f2e2-4909-b551-47fd6583a893 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'e533f12763c7431dada530f2f195e532', 'request_id': 'req-06878f8e-f2e2-4909-b551-47fd6583a893'} {{(pid=81409) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 13 04:23:27.124048 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.notifications [None req-06878f8e-f2e2-4909-b551-47fd6583a893 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'e533f12763c7431dada530f2f195e532', 'request_id': 'req-06878f8e-f2e2-4909-b551-47fd6583a893'} {{(pid=81409) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 13 04:23:27.135971 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.notifications [None req-06878f8e-f2e2-4909-b551-47fd6583a893 admin admin] The token cache is being invalidate because project e533f12763c7431dada530f2f195e532 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81409) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 13 04:23:27.136260 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.notifications [None req-06878f8e-f2e2-4909-b551-47fd6583a893 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81409) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 13 04:23:27.136913 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.notifications [None req-06878f8e-f2e2-4909-b551-47fd6583a893 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81409) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 13 04:23:27.138628 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 946/1863] 10.222.0.27 () {58 vars in 1233 bytes} [Thu Oct 13 04:23:26 2022] DELETE /identity/v3/projects/e533f12763c7431dada530f2f195e532 => generated 0 bytes in 196 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 13 04:23:27.143523 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0e16c77-5d1d-4540-aaa7-0cb8665715da None None] REQUEST_METHOD: `POST` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:23:27.144223 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0e16c77-5d1d-4540-aaa7-0cb8665715da None None] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:23:27.144785 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0e16c77-5d1d-4540-aaa7-0cb8665715da None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:23:27.187655 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.auth.core [None req-b0e16c77-5d1d-4540-aaa7-0cb8665715da None None] MFA Rules not processed for user `0e2d979563d6421995451a0e86bf3022`. Rule list: `[]` (Enabled: `True`). {{(pid=81408) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 13 04:23:27.190193 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.fernet_utils [None req-b0e16c77-5d1d-4540-aaa7-0cb8665715da None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81408) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:23:27.276281 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 918/1864] 10.222.0.27 () {58 vars in 958 bytes} [Thu Oct 13 04:23:27 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 134 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 13 04:23:27.286012 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18fb85c3-f2a6-4ba0-817d-daedc3d66188 None None] REQUEST_METHOD: `POST` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:23:27.286417 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18fb85c3-f2a6-4ba0-817d-daedc3d66188 None None] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:23:27.286791 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18fb85c3-f2a6-4ba0-817d-daedc3d66188 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:23:27.308397 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.auth.core [None req-18fb85c3-f2a6-4ba0-817d-daedc3d66188 None None] MFA Rules not processed for user `0e2d979563d6421995451a0e86bf3022`. Rule list: `[]` (Enabled: `True`). {{(pid=81409) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 13 04:23:27.309738 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.fernet_utils [None req-18fb85c3-f2a6-4ba0-817d-daedc3d66188 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81409) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:23:27.321921 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 947/1865] 10.222.0.27 () {58 vars in 958 bytes} [Thu Oct 13 04:23:27 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 34 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 13 04:23:27.323534 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b0e16c77-5d1d-4540-aaa7-0cb8665715da None None] Authenticating user token {{(pid=81408) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:23:27.333968 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b0e16c77-5d1d-4540-aaa7-0cb8665715da None None] Validating token access rules against request {{(pid=81408) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:23:27.338598 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ea188f76-70ac-4f49-a804-c0fe8ec1a789 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8ba823f313144da3b683bacef8fd62e0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81408) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:23:27.339220 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea188f76-70ac-4f49-a804-c0fe8ec1a789 admin admin] REQUEST_METHOD: `GET` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:23:27.339480 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea188f76-70ac-4f49-a804-c0fe8ec1a789 admin admin] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:23:27.339755 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea188f76-70ac-4f49-a804-c0fe8ec1a789 admin admin] PATH_INFO: `/v3/domains` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:23:27.340409 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ea188f76-70ac-4f49-a804-c0fe8ec1a789 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81408) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 13 04:23:27.340714 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ea188f76-70ac-4f49-a804-c0fe8ec1a789 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:23:27.342379 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ea188f76-70ac-4f49-a804-c0fe8ec1a789 admin admin] RBAC: Authorization granted {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:23:27.354967 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 919/1866] 10.222.0.27 () {58 vars in 1153 bytes} [Thu Oct 13 04:23:27 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 13 04:23:27.359336 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-18fb85c3-f2a6-4ba0-817d-daedc3d66188 None None] Authenticating user token {{(pid=81409) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:23:27.368930 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-18fb85c3-f2a6-4ba0-817d-daedc3d66188 None None] Validating token access rules against request {{(pid=81409) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:23:27.373612 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1ba4acd2-3b50-404a-b607-9c97a3e8bca1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8ba823f313144da3b683bacef8fd62e0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81409) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:23:27.374305 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ba4acd2-3b50-404a-b607-9c97a3e8bca1 admin admin] REQUEST_METHOD: `POST` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:23:27.374589 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ba4acd2-3b50-404a-b607-9c97a3e8bca1 admin admin] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:23:27.374859 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ba4acd2-3b50-404a-b607-9c97a3e8bca1 admin admin] PATH_INFO: `/v3/projects` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:23:27.375504 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1ba4acd2-3b50-404a-b607-9c97a3e8bca1 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:23:27.377968 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1ba4acd2-3b50-404a-b607-9c97a3e8bca1 admin admin] RBAC: Authorization granted {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:23:27.407108 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 948/1867] 10.222.0.27 () {60 vars in 1153 bytes} [Thu Oct 13 04:23:27 2022] POST /identity/v3/projects => generated 373 bytes in 48 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 13 04:23:27.411163 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ea188f76-70ac-4f49-a804-c0fe8ec1a789 admin admin] Authenticating user token {{(pid=81408) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:23:27.421595 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ea188f76-70ac-4f49-a804-c0fe8ec1a789 admin admin] Validating token access rules against request {{(pid=81408) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:23:27.470967 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG dbcounter [-] [81408] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81408) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:23:27.493572 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bdb78f7c-fad5-44c1-ba59-758b98f606d1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8ba823f313144da3b683bacef8fd62e0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81408) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:23:27.494295 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bdb78f7c-fad5-44c1-ba59-758b98f606d1 admin admin] REQUEST_METHOD: `POST` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:23:27.494662 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bdb78f7c-fad5-44c1-ba59-758b98f606d1 admin admin] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:23:27.495040 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bdb78f7c-fad5-44c1-ba59-758b98f606d1 admin admin] PATH_INFO: `/v3/users` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:23:27.495784 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bdb78f7c-fad5-44c1-ba59-758b98f606d1 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:23:27.497917 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bdb78f7c-fad5-44c1-ba59-758b98f606d1 admin admin] RBAC: Authorization granted {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:23:27.523423 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 920/1868] 10.222.0.27 () {60 vars in 1144 bytes} [Thu Oct 13 04:23:27 2022] POST /identity/v3/users => generated 287 bytes in 113 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 13 04:23:27.527867 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1ba4acd2-3b50-404a-b607-9c97a3e8bca1 admin admin] Authenticating user token {{(pid=81409) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:23:27.543825 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1ba4acd2-3b50-404a-b607-9c97a3e8bca1 admin admin] Validating token access rules against request {{(pid=81409) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:23:27.550260 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e94892d8-1436-4f04-9993-a77249f947d1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8ba823f313144da3b683bacef8fd62e0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81409) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:23:27.551159 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e94892d8-1436-4f04-9993-a77249f947d1 admin admin] REQUEST_METHOD: `GET` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:23:27.551531 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e94892d8-1436-4f04-9993-a77249f947d1 admin admin] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:23:27.551887 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e94892d8-1436-4f04-9993-a77249f947d1 admin admin] PATH_INFO: `/v3/roles` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:23:27.552713 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e94892d8-1436-4f04-9993-a77249f947d1 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:23:27.554590 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e94892d8-1436-4f04-9993-a77249f947d1 admin admin] RBAC: Authorization granted {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:23:27.566149 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 949/1869] 10.222.0.27 () {58 vars in 1122 bytes} [Thu Oct 13 04:23:27 2022] GET /identity/v3/roles => generated 1420 bytes in 39 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 13 04:23:27.570327 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bdb78f7c-fad5-44c1-ba59-758b98f606d1 admin admin] Authenticating user token {{(pid=81408) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:23:27.582474 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bdb78f7c-fad5-44c1-ba59-758b98f606d1 admin admin] Validating token access rules against request {{(pid=81408) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:23:27.588363 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-328ef94b-ed24-4105-a280-d1648a369a6f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8ba823f313144da3b683bacef8fd62e0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81408) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:23:27.588987 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-328ef94b-ed24-4105-a280-d1648a369a6f admin admin] REQUEST_METHOD: `GET` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:23:27.589268 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-328ef94b-ed24-4105-a280-d1648a369a6f admin admin] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:23:27.590070 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-328ef94b-ed24-4105-a280-d1648a369a6f admin admin] PATH_INFO: `/v3/roles` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:23:27.590179 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-328ef94b-ed24-4105-a280-d1648a369a6f admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:23:27.591847 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-328ef94b-ed24-4105-a280-d1648a369a6f admin admin] RBAC: Authorization granted {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:23:27.605426 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 921/1870] 10.222.0.27 () {58 vars in 1122 bytes} [Thu Oct 13 04:23:27 2022] GET /identity/v3/roles => generated 1420 bytes in 35 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 13 04:23:27.611243 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e94892d8-1436-4f04-9993-a77249f947d1 admin admin] Authenticating user token {{(pid=81409) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:23:27.622959 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e94892d8-1436-4f04-9993-a77249f947d1 admin admin] Validating token access rules against request {{(pid=81409) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:23:27.628741 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1a3bc505-19e5-4555-b007-749684448774 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8ba823f313144da3b683bacef8fd62e0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81409) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:23:27.629573 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a3bc505-19e5-4555-b007-749684448774 admin admin] REQUEST_METHOD: `PUT` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:23:27.629848 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a3bc505-19e5-4555-b007-749684448774 admin admin] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:23:27.630134 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a3bc505-19e5-4555-b007-749684448774 admin admin] PATH_INFO: `/v3/projects/a663647c36ba40d3a5692cfa20b9234c/users/6e4cf0d322264234875014a3ecaab4b0/roles/cc56c4ae13a7424ea771c15da4b153c1` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:23:27.648820 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1a3bc505-19e5-4555-b007-749684448774 admin admin] RBAC: Authorizing `identity:create_grant(project_id=a663647c36ba40d3a5692cfa20b9234c, user_id=6e4cf0d322264234875014a3ecaab4b0, role_id=cc56c4ae13a7424ea771c15da4b153c1)` {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:23:27.653152 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1a3bc505-19e5-4555-b007-749684448774 admin admin] RBAC: Authorization granted {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:23:27.669242 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 950/1871] 10.222.0.27 () {60 vars in 1483 bytes} [Thu Oct 13 04:23:27 2022] PUT /identity/v3/projects/a663647c36ba40d3a5692cfa20b9234c/users/6e4cf0d322264234875014a3ecaab4b0/roles/cc56c4ae13a7424ea771c15da4b153c1 => generated 0 bytes in 60 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 13 04:23:27.675348 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f380e17-a18e-438b-acfe-328987373ef9 None None] REQUEST_METHOD: `POST` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:23:27.675718 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f380e17-a18e-438b-acfe-328987373ef9 None None] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:23:27.676092 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f380e17-a18e-438b-acfe-328987373ef9 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:23:27.712096 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.auth.core [None req-6f380e17-a18e-438b-acfe-328987373ef9 None None] MFA Rules not processed for user `0e2d979563d6421995451a0e86bf3022`. Rule list: `[]` (Enabled: `True`). {{(pid=81408) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 13 04:23:27.715803 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.fernet_utils [None req-6f380e17-a18e-438b-acfe-328987373ef9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81408) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:23:27.832308 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 922/1872] 10.222.0.27 () {58 vars in 958 bytes} [Thu Oct 13 04:23:27 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 158 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 13 04:23:27.853507 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1a3bc505-19e5-4555-b007-749684448774 admin admin] Authenticating user token {{(pid=81409) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:23:27.855934 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.fernet_utils [None req-1a3bc505-19e5-4555-b007-749684448774 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81409) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:23:27.940414 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG dbcounter [-] [81409] Writing DB stats keystone:SELECT=90,keystone:DELETE=7,keystone:INSERT=3 {{(pid=81409) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:23:27.947344 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1a3bc505-19e5-4555-b007-749684448774 admin admin] Validating token access rules against request {{(pid=81409) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:23:27.953476 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7079e2b1-bf4b-4c0b-ab76-2f6daa25eb86 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd8812c3974f84871bb2d761cf0016257', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dcec0431fb6849499d6aec35a234daa6', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81409) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:23:27.954945 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7079e2b1-bf4b-4c0b-ab76-2f6daa25eb86 service neutron] REQUEST_METHOD: `GET` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:23:27.955642 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7079e2b1-bf4b-4c0b-ab76-2f6daa25eb86 service neutron] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:23:27.956023 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7079e2b1-bf4b-4c0b-ab76-2f6daa25eb86 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:23:27.970887 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7079e2b1-bf4b-4c0b-ab76-2f6daa25eb86 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:23:27.972767 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7079e2b1-bf4b-4c0b-ab76-2f6daa25eb86 service neutron] RBAC: Authorization granted {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:23:27.987601 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 951/1873] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Oct 13 04:23:27 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 134 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 13 04:23:29.750446 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1723fcd4-7222-4e08-898a-faf0151c22e7 None None] REQUEST_METHOD: `POST` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:23:29.750985 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1723fcd4-7222-4e08-898a-faf0151c22e7 None None] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:23:29.751130 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1723fcd4-7222-4e08-898a-faf0151c22e7 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:23:29.779438 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.auth.core [None req-1723fcd4-7222-4e08-898a-faf0151c22e7 None None] MFA Rules not processed for user `6e4cf0d322264234875014a3ecaab4b0`. Rule list: `[]` (Enabled: `True`). {{(pid=81408) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 13 04:23:29.780651 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.fernet_utils [None req-1723fcd4-7222-4e08-898a-faf0151c22e7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81408) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:23:29.849549 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 923/1874] 10.222.0.27 () {58 vars in 958 bytes} [Thu Oct 13 04:23:29 2022] POST /identity/v3/auth/tokens => generated 3250 bytes in 100 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 13 04:23:29.854100 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7079e2b1-bf4b-4c0b-ab76-2f6daa25eb86 service neutron] Authenticating user token {{(pid=81409) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:23:29.866014 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7079e2b1-bf4b-4c0b-ab76-2f6daa25eb86 service neutron] Validating token access rules against request {{(pid=81409) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:23:29.870840 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f8bea225-9c08-47ca-b502-40d67f0e4c62 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8ba823f313144da3b683bacef8fd62e0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81409) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:23:29.871509 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8bea225-9c08-47ca-b502-40d67f0e4c62 admin admin] REQUEST_METHOD: `POST` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:23:29.871852 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8bea225-9c08-47ca-b502-40d67f0e4c62 admin admin] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:23:29.872189 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8bea225-9c08-47ca-b502-40d67f0e4c62 admin admin] PATH_INFO: `/v3/projects` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:23:29.873205 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f8bea225-9c08-47ca-b502-40d67f0e4c62 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:23:29.875282 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f8bea225-9c08-47ca-b502-40d67f0e4c62 admin admin] RBAC: Authorization granted {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:23:29.908752 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 952/1875] 10.222.0.27 () {60 vars in 1153 bytes} [Thu Oct 13 04:23:29 2022] POST /identity/v3/projects => generated 373 bytes in 55 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 13 04:23:29.912544 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1723fcd4-7222-4e08-898a-faf0151c22e7 None None] Authenticating user token {{(pid=81408) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:23:29.923711 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1723fcd4-7222-4e08-898a-faf0151c22e7 None None] Validating token access rules against request {{(pid=81408) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:23:29.986181 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG dbcounter [-] [81408] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81408) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:23:29.993855 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-91169d72-5539-45df-9a58-988bfb925496 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8ba823f313144da3b683bacef8fd62e0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81408) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:23:29.994487 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91169d72-5539-45df-9a58-988bfb925496 admin admin] REQUEST_METHOD: `POST` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:23:29.994764 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91169d72-5539-45df-9a58-988bfb925496 admin admin] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:23:29.995035 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91169d72-5539-45df-9a58-988bfb925496 admin admin] PATH_INFO: `/v3/users` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:23:29.995673 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-91169d72-5539-45df-9a58-988bfb925496 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:23:29.997597 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-91169d72-5539-45df-9a58-988bfb925496 admin admin] RBAC: Authorization granted {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:23:30.019643 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 924/1876] 10.222.0.27 () {60 vars in 1144 bytes} [Thu Oct 13 04:23:29 2022] POST /identity/v3/users => generated 293 bytes in 108 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 13 04:23:30.023682 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f8bea225-9c08-47ca-b502-40d67f0e4c62 admin admin] Authenticating user token {{(pid=81409) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:23:30.034926 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f8bea225-9c08-47ca-b502-40d67f0e4c62 admin admin] Validating token access rules against request {{(pid=81409) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:23:30.039430 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9a4a68c7-cd01-4c73-82bd-28093f6054cb admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8ba823f313144da3b683bacef8fd62e0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81409) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:23:30.040025 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a4a68c7-cd01-4c73-82bd-28093f6054cb admin admin] REQUEST_METHOD: `GET` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:23:30.040301 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a4a68c7-cd01-4c73-82bd-28093f6054cb admin admin] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:23:30.040571 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a4a68c7-cd01-4c73-82bd-28093f6054cb admin admin] PATH_INFO: `/v3/roles` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:23:30.041209 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9a4a68c7-cd01-4c73-82bd-28093f6054cb admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:23:30.042830 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9a4a68c7-cd01-4c73-82bd-28093f6054cb admin admin] RBAC: Authorization granted {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:23:30.054131 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 953/1877] 10.222.0.27 () {58 vars in 1122 bytes} [Thu Oct 13 04:23:30 2022] GET /identity/v3/roles => generated 1420 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 13 04:23:30.058174 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-91169d72-5539-45df-9a58-988bfb925496 admin admin] Authenticating user token {{(pid=81408) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:23:30.069371 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-91169d72-5539-45df-9a58-988bfb925496 admin admin] Validating token access rules against request {{(pid=81408) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:23:30.073956 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-abe39bd1-4b45-49ec-a1b5-f808e709c109 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8ba823f313144da3b683bacef8fd62e0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81408) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:23:30.074629 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-abe39bd1-4b45-49ec-a1b5-f808e709c109 admin admin] REQUEST_METHOD: `PUT` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:23:30.074903 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-abe39bd1-4b45-49ec-a1b5-f808e709c109 admin admin] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:23:30.075173 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-abe39bd1-4b45-49ec-a1b5-f808e709c109 admin admin] PATH_INFO: `/v3/projects/03b3c3b7eb4f400f867f9b928c239311/users/23ae25269c5c450c8582191cf4414729/roles/ef55fbd94e1040be83695754075a7ed5` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:23:30.090506 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-abe39bd1-4b45-49ec-a1b5-f808e709c109 admin admin] RBAC: Authorizing `identity:create_grant(project_id=03b3c3b7eb4f400f867f9b928c239311, user_id=23ae25269c5c450c8582191cf4414729, role_id=ef55fbd94e1040be83695754075a7ed5)` {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:23:30.093206 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-abe39bd1-4b45-49ec-a1b5-f808e709c109 admin admin] RBAC: Authorization granted {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:23:30.105438 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 925/1878] 10.222.0.27 () {60 vars in 1483 bytes} [Thu Oct 13 04:23:30 2022] PUT /identity/v3/projects/03b3c3b7eb4f400f867f9b928c239311/users/23ae25269c5c450c8582191cf4414729/roles/ef55fbd94e1040be83695754075a7ed5 => generated 0 bytes in 48 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 13 04:23:30.109727 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9a4a68c7-cd01-4c73-82bd-28093f6054cb admin admin] Authenticating user token {{(pid=81409) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:23:30.120378 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9a4a68c7-cd01-4c73-82bd-28093f6054cb admin admin] Validating token access rules against request {{(pid=81409) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:23:30.176009 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f41f9dd1-8fe8-46f5-89f8-2c60c488978e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8ba823f313144da3b683bacef8fd62e0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81409) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:23:30.176605 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f41f9dd1-8fe8-46f5-89f8-2c60c488978e admin admin] REQUEST_METHOD: `GET` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:23:30.176875 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f41f9dd1-8fe8-46f5-89f8-2c60c488978e admin admin] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:23:30.177156 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f41f9dd1-8fe8-46f5-89f8-2c60c488978e admin admin] PATH_INFO: `/v3/roles` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:23:30.177769 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f41f9dd1-8fe8-46f5-89f8-2c60c488978e admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:23:30.179360 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f41f9dd1-8fe8-46f5-89f8-2c60c488978e admin admin] RBAC: Authorization granted {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:23:30.189715 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 954/1879] 10.222.0.27 () {58 vars in 1122 bytes} [Thu Oct 13 04:23:30 2022] GET /identity/v3/roles => generated 1420 bytes in 80 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 13 04:23:30.193853 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-abe39bd1-4b45-49ec-a1b5-f808e709c109 admin admin] Authenticating user token {{(pid=81408) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:23:30.204427 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-abe39bd1-4b45-49ec-a1b5-f808e709c109 admin admin] Validating token access rules against request {{(pid=81408) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:23:30.209043 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-59cfc213-017e-4a5d-bf28-39c4e200965d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8ba823f313144da3b683bacef8fd62e0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81408) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:23:30.209685 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59cfc213-017e-4a5d-bf28-39c4e200965d admin admin] REQUEST_METHOD: `PUT` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:23:30.209962 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59cfc213-017e-4a5d-bf28-39c4e200965d admin admin] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:23:30.210261 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59cfc213-017e-4a5d-bf28-39c4e200965d admin admin] PATH_INFO: `/v3/projects/03b3c3b7eb4f400f867f9b928c239311/users/23ae25269c5c450c8582191cf4414729/roles/ef55fbd94e1040be83695754075a7ed5` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:23:30.212156 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-59cfc213-017e-4a5d-bf28-39c4e200965d admin admin] RBAC: Authorizing `identity:create_grant(project_id=03b3c3b7eb4f400f867f9b928c239311, user_id=23ae25269c5c450c8582191cf4414729, role_id=ef55fbd94e1040be83695754075a7ed5)` {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:23:30.214846 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-59cfc213-017e-4a5d-bf28-39c4e200965d admin admin] RBAC: Authorization granted {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:23:30.225462 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 926/1880] 10.222.0.27 () {60 vars in 1483 bytes} [Thu Oct 13 04:23:30 2022] PUT /identity/v3/projects/03b3c3b7eb4f400f867f9b928c239311/users/23ae25269c5c450c8582191cf4414729/roles/ef55fbd94e1040be83695754075a7ed5 => generated 0 bytes in 32 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 13 04:23:30.978788 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f00f9a7d-8eb3-41e5-96de-bbe45c34c8ea None None] REQUEST_METHOD: `POST` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:23:30.979283 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f00f9a7d-8eb3-41e5-96de-bbe45c34c8ea None None] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:23:30.979472 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f00f9a7d-8eb3-41e5-96de-bbe45c34c8ea None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:23:31.008510 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.auth.core [None req-f00f9a7d-8eb3-41e5-96de-bbe45c34c8ea None None] MFA Rules not processed for user `23ae25269c5c450c8582191cf4414729`. Rule list: `[]` (Enabled: `True`). {{(pid=81409) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 13 04:23:31.009721 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.fernet_utils [None req-f00f9a7d-8eb3-41e5-96de-bbe45c34c8ea None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81409) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:23:31.080292 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 955/1881] 10.222.0.27 () {58 vars in 958 bytes} [Thu Oct 13 04:23:30 2022] POST /identity/v3/auth/tokens => generated 3317 bytes in 102 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 13 04:23:31.093234 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-59cfc213-017e-4a5d-bf28-39c4e200965d admin admin] Authenticating user token {{(pid=81408) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:23:31.096559 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.fernet_utils [None req-59cfc213-017e-4a5d-bf28-39c4e200965d admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81408) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:23:31.160986 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-59cfc213-017e-4a5d-bf28-39c4e200965d admin admin] Validating token access rules against request {{(pid=81408) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:23:31.164737 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7ac1b5e6-ab01-4555-8b04-1d13359dadf4 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6141b8a6c3d54e179d6879bb019f4d12', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dcec0431fb6849499d6aec35a234daa6', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81408) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:23:31.165338 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ac1b5e6-ab01-4555-8b04-1d13359dadf4 service cinder] REQUEST_METHOD: `GET` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:23:31.165611 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ac1b5e6-ab01-4555-8b04-1d13359dadf4 service cinder] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:23:31.165875 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ac1b5e6-ab01-4555-8b04-1d13359dadf4 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:23:31.221908 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7ac1b5e6-ab01-4555-8b04-1d13359dadf4 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:23:31.223432 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7ac1b5e6-ab01-4555-8b04-1d13359dadf4 service cinder] RBAC: Authorization granted {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:23:31.232926 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 927/1882] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Oct 13 04:23:31 2022] GET /identity/v3/auth/tokens => generated 3250 bytes in 140 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 13 04:23:33.259658 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f93d439-4f6f-4381-b42b-67b0d78577ad None None] REQUEST_METHOD: `POST` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:23:33.260088 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f93d439-4f6f-4381-b42b-67b0d78577ad None None] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:23:33.260185 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f93d439-4f6f-4381-b42b-67b0d78577ad None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:23:33.280999 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.auth.core [None req-9f93d439-4f6f-4381-b42b-67b0d78577ad None None] MFA Rules not processed for user `487b1605fd4143cfaa422a3a7a0344c7`. Rule list: `[]` (Enabled: `True`). {{(pid=81409) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 13 04:23:33.282306 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.fernet_utils [None req-9f93d439-4f6f-4381-b42b-67b0d78577ad None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81409) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:23:33.318840 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG dbcounter [-] [81409] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81409) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:23:33.345862 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 956/1883] 10.222.0.27 () {58 vars in 1009 bytes} [Thu Oct 13 04:23:33 2022] POST /identity/v3/auth/tokens => generated 3195 bytes in 87 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 13 04:23:33.387016 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7ac1b5e6-ab01-4555-8b04-1d13359dadf4 service cinder] Authenticating user token {{(pid=81408) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:23:33.388431 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.fernet_utils [None req-7ac1b5e6-ab01-4555-8b04-1d13359dadf4 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81408) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:23:33.455472 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG dbcounter [-] [81408] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=81408) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:23:33.459913 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7ac1b5e6-ab01-4555-8b04-1d13359dadf4 service cinder] Validating token access rules against request {{(pid=81408) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:23:33.465652 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c6182165-57a8-483e-b919-18bb601629eb service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90a0f466b664458898cf9dc4112ae651', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dcec0431fb6849499d6aec35a234daa6', 'project_domain_id': 'default', 'roles': ['service', 'member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81408) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:23:33.466282 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6182165-57a8-483e-b919-18bb601629eb service swift] REQUEST_METHOD: `GET` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:23:33.466559 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6182165-57a8-483e-b919-18bb601629eb service swift] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:23:33.466828 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6182165-57a8-483e-b919-18bb601629eb service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:23:33.475356 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c6182165-57a8-483e-b919-18bb601629eb service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:23:33.476932 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c6182165-57a8-483e-b919-18bb601629eb service swift] RBAC: Authorization granted {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:23:33.485881 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 928/1884] 10.222.0.27 () {60 vars in 1384 bytes} [Thu Oct 13 04:23:33 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 99 msecs (HTTP/1.1 200) 6 headers in 379 bytes (2 switches on core 0) Oct 13 04:23:33.671405 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9f93d439-4f6f-4381-b42b-67b0d78577ad None None] Authenticating user token {{(pid=81409) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:23:33.673266 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.fernet_utils [None req-9f93d439-4f6f-4381-b42b-67b0d78577ad None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81409) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:23:33.685402 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9f93d439-4f6f-4381-b42b-67b0d78577ad None None] Validating token access rules against request {{(pid=81409) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:23:33.689510 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e4eb8e4d-38f1-473f-8768-3099186f86d6 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6141b8a6c3d54e179d6879bb019f4d12', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dcec0431fb6849499d6aec35a234daa6', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81409) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:23:33.690531 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4eb8e4d-38f1-473f-8768-3099186f86d6 service cinder] REQUEST_METHOD: `GET` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:23:33.690951 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4eb8e4d-38f1-473f-8768-3099186f86d6 service cinder] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:23:33.691395 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4eb8e4d-38f1-473f-8768-3099186f86d6 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:23:33.705450 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e4eb8e4d-38f1-473f-8768-3099186f86d6 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:23:33.708082 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e4eb8e4d-38f1-473f-8768-3099186f86d6 service cinder] RBAC: Authorization granted {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:23:33.723297 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 957/1885] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Oct 13 04:23:33 2022] GET /identity/v3/auth/tokens => generated 3317 bytes in 53 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Oct 13 04:23:38.722278 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e9b2444-5e4b-4428-a679-c89573409345 None None] REQUEST_METHOD: `POST` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:23:38.722896 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e9b2444-5e4b-4428-a679-c89573409345 None None] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:23:38.722896 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e9b2444-5e4b-4428-a679-c89573409345 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:23:38.780091 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.auth.core [None req-1e9b2444-5e4b-4428-a679-c89573409345 None None] MFA Rules not processed for user `487b1605fd4143cfaa422a3a7a0344c7`. Rule list: `[]` (Enabled: `True`). {{(pid=81408) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 13 04:23:38.782681 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.fernet_utils [None req-1e9b2444-5e4b-4428-a679-c89573409345 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81408) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:23:38.804546 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 929/1886] 10.222.0.27 () {58 vars in 1009 bytes} [Thu Oct 13 04:23:38 2022] POST /identity/v3/auth/tokens => generated 3195 bytes in 83 msecs (HTTP/1.1 201) 6 headers in 385 bytes (3 switches on core 0) Oct 13 04:23:38.839845 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e4eb8e4d-38f1-473f-8768-3099186f86d6 service cinder] Authenticating user token {{(pid=81409) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:23:38.861172 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e4eb8e4d-38f1-473f-8768-3099186f86d6 service cinder] Validating token access rules against request {{(pid=81409) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:23:38.895633 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4c43f1d0-0123-4ddb-8185-5a2ab55daee2 service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90a0f466b664458898cf9dc4112ae651', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dcec0431fb6849499d6aec35a234daa6', 'project_domain_id': 'default', 'roles': ['service', 'member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81409) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:23:38.897423 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c43f1d0-0123-4ddb-8185-5a2ab55daee2 service swift] REQUEST_METHOD: `GET` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:23:38.897714 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c43f1d0-0123-4ddb-8185-5a2ab55daee2 service swift] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:23:38.897988 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c43f1d0-0123-4ddb-8185-5a2ab55daee2 service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:23:38.919046 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4c43f1d0-0123-4ddb-8185-5a2ab55daee2 service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:23:38.920908 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4c43f1d0-0123-4ddb-8185-5a2ab55daee2 service swift] RBAC: Authorization granted {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:23:38.936191 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 958/1887] 10.222.0.27 () {60 vars in 1384 bytes} [Thu Oct 13 04:23:38 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 97 msecs (HTTP/1.1 200) 6 headers in 379 bytes (1 switches on core 0) Oct 13 04:23:48.764258 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG dbcounter [-] [81408] Writing DB stats keystone:SELECT=7 {{(pid=81408) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:23:48.931513 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG dbcounter [-] [81409] Writing DB stats keystone:SELECT=19 {{(pid=81409) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:23:51.612077 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e9b2444-5e4b-4428-a679-c89573409345 None None] Authenticating user token {{(pid=81408) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:23:51.785584 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e9b2444-5e4b-4428-a679-c89573409345 None None] Validating token access rules against request {{(pid=81408) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:23:51.824070 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7df04a9e-8409-4546-b601-9742432df6dd admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8ba823f313144da3b683bacef8fd62e0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81408) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:23:51.825408 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7df04a9e-8409-4546-b601-9742432df6dd admin admin] REQUEST_METHOD: `DELETE` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:23:51.831537 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7df04a9e-8409-4546-b601-9742432df6dd admin admin] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:23:51.832141 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7df04a9e-8409-4546-b601-9742432df6dd admin admin] PATH_INFO: `/v3/users/6e4cf0d322264234875014a3ecaab4b0` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:23:51.833733 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7df04a9e-8409-4546-b601-9742432df6dd admin admin] RBAC: Authorizing `identity:delete_user(user_id=6e4cf0d322264234875014a3ecaab4b0)` {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:23:51.842328 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7df04a9e-8409-4546-b601-9742432df6dd admin admin] RBAC: Authorization granted {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:23:52.020290 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.notifications [None req-7df04a9e-8409-4546-b601-9742432df6dd admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '6e4cf0d322264234875014a3ecaab4b0'} {{(pid=81408) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 13 04:23:52.037911 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.notifications [None req-7df04a9e-8409-4546-b601-9742432df6dd admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '6e4cf0d322264234875014a3ecaab4b0'} {{(pid=81408) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 13 04:23:52.038953 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.notifications [None req-7df04a9e-8409-4546-b601-9742432df6dd admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '6e4cf0d322264234875014a3ecaab4b0'} {{(pid=81408) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 13 04:23:52.040055 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.notifications [None req-7df04a9e-8409-4546-b601-9742432df6dd admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '6e4cf0d322264234875014a3ecaab4b0'} {{(pid=81408) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 13 04:23:52.069684 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.notifications [None req-7df04a9e-8409-4546-b601-9742432df6dd admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '6e4cf0d322264234875014a3ecaab4b0'} {{(pid=81408) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 13 04:23:52.115814 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 930/1888] 10.222.0.27 () {58 vars in 1224 bytes} [Thu Oct 13 04:23:51 2022] DELETE /identity/v3/users/6e4cf0d322264234875014a3ecaab4b0 => generated 0 bytes in 505 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 13 04:23:52.707860 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4c43f1d0-0123-4ddb-8185-5a2ab55daee2 service swift] Authenticating user token {{(pid=81409) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:23:52.733076 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4c43f1d0-0123-4ddb-8185-5a2ab55daee2 service swift] Validating token access rules against request {{(pid=81409) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:23:52.987420 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0a917026-9126-4c94-af6a-921cd1505365 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8ba823f313144da3b683bacef8fd62e0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81409) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:23:52.990440 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a917026-9126-4c94-af6a-921cd1505365 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:23:52.990834 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a917026-9126-4c94-af6a-921cd1505365 admin admin] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:23:52.991214 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a917026-9126-4c94-af6a-921cd1505365 admin admin] PATH_INFO: `/v3/projects/a663647c36ba40d3a5692cfa20b9234c` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:23:52.994486 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0a917026-9126-4c94-af6a-921cd1505365 admin admin] RBAC: Authorizing `identity:delete_project(project_id=a663647c36ba40d3a5692cfa20b9234c)` {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:23:53.002086 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0a917026-9126-4c94-af6a-921cd1505365 admin admin] RBAC: Authorization granted {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:23:53.140395 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.notifications [None req-0a917026-9126-4c94-af6a-921cd1505365 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'a663647c36ba40d3a5692cfa20b9234c', 'request_id': 'req-0a917026-9126-4c94-af6a-921cd1505365'} {{(pid=81409) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 13 04:23:53.159877 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.notifications [None req-0a917026-9126-4c94-af6a-921cd1505365 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'a663647c36ba40d3a5692cfa20b9234c', 'request_id': 'req-0a917026-9126-4c94-af6a-921cd1505365'} {{(pid=81409) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 13 04:23:53.194690 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.notifications [None req-0a917026-9126-4c94-af6a-921cd1505365 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'a663647c36ba40d3a5692cfa20b9234c', 'request_id': 'req-0a917026-9126-4c94-af6a-921cd1505365'} {{(pid=81409) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 13 04:23:53.240837 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.notifications [None req-0a917026-9126-4c94-af6a-921cd1505365 admin admin] The token cache is being invalidate because project a663647c36ba40d3a5692cfa20b9234c was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81409) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 13 04:23:53.241129 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.notifications [None req-0a917026-9126-4c94-af6a-921cd1505365 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81409) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 13 04:23:53.245695 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.notifications [None req-0a917026-9126-4c94-af6a-921cd1505365 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81409) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 13 04:23:53.250322 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 959/1889] 10.222.0.27 () {58 vars in 1233 bytes} [Thu Oct 13 04:23:52 2022] DELETE /identity/v3/projects/a663647c36ba40d3a5692cfa20b9234c => generated 0 bytes in 543 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 13 04:23:53.254234 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7df04a9e-8409-4546-b601-9742432df6dd admin admin] Authenticating user token {{(pid=81408) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:23:53.256690 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.fernet_utils [None req-7df04a9e-8409-4546-b601-9742432df6dd admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81408) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:23:53.286175 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7df04a9e-8409-4546-b601-9742432df6dd admin admin] Validating token access rules against request {{(pid=81408) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:23:53.443175 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2dc9b4a7-c480-4b88-bbd2-a05c6a986027 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8ba823f313144da3b683bacef8fd62e0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81408) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:23:53.444380 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2dc9b4a7-c480-4b88-bbd2-a05c6a986027 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:23:53.445585 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2dc9b4a7-c480-4b88-bbd2-a05c6a986027 admin admin] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:23:53.447703 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2dc9b4a7-c480-4b88-bbd2-a05c6a986027 admin admin] PATH_INFO: `/v3/users/23ae25269c5c450c8582191cf4414729` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:23:53.449149 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2dc9b4a7-c480-4b88-bbd2-a05c6a986027 admin admin] RBAC: Authorizing `identity:delete_user(user_id=23ae25269c5c450c8582191cf4414729)` {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:23:53.451737 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2dc9b4a7-c480-4b88-bbd2-a05c6a986027 admin admin] RBAC: Authorization granted {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:23:53.502025 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG dbcounter [-] [81408] Writing DB stats keystone:SELECT=86,keystone:DELETE=13,keystone:INSERT=1 {{(pid=81408) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:23:53.598527 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.notifications [None req-2dc9b4a7-c480-4b88-bbd2-a05c6a986027 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '23ae25269c5c450c8582191cf4414729'} {{(pid=81408) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 13 04:23:53.621063 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.notifications [None req-2dc9b4a7-c480-4b88-bbd2-a05c6a986027 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '23ae25269c5c450c8582191cf4414729'} {{(pid=81408) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 13 04:23:53.621063 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.notifications [None req-2dc9b4a7-c480-4b88-bbd2-a05c6a986027 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '23ae25269c5c450c8582191cf4414729'} {{(pid=81408) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 13 04:23:53.621368 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.notifications [None req-2dc9b4a7-c480-4b88-bbd2-a05c6a986027 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '23ae25269c5c450c8582191cf4414729'} {{(pid=81408) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 13 04:23:53.654133 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.notifications [None req-2dc9b4a7-c480-4b88-bbd2-a05c6a986027 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '23ae25269c5c450c8582191cf4414729'} {{(pid=81408) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 13 04:23:53.766092 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 931/1890] 10.222.0.27 () {58 vars in 1224 bytes} [Thu Oct 13 04:23:53 2022] DELETE /identity/v3/users/23ae25269c5c450c8582191cf4414729 => generated 0 bytes in 495 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 13 04:23:54.554533 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0a917026-9126-4c94-af6a-921cd1505365 admin admin] Authenticating user token {{(pid=81409) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:23:54.556238 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.fernet_utils [None req-0a917026-9126-4c94-af6a-921cd1505365 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81409) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:23:54.597135 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0a917026-9126-4c94-af6a-921cd1505365 admin admin] Validating token access rules against request {{(pid=81409) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:23:54.778361 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9cee245b-8974-4b69-bc47-b74fd30684e6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8ba823f313144da3b683bacef8fd62e0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81409) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:23:54.779000 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9cee245b-8974-4b69-bc47-b74fd30684e6 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:23:54.779273 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9cee245b-8974-4b69-bc47-b74fd30684e6 admin admin] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:23:54.779545 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9cee245b-8974-4b69-bc47-b74fd30684e6 admin admin] PATH_INFO: `/v3/projects/03b3c3b7eb4f400f867f9b928c239311` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:23:54.780684 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9cee245b-8974-4b69-bc47-b74fd30684e6 admin admin] RBAC: Authorizing `identity:delete_project(project_id=03b3c3b7eb4f400f867f9b928c239311)` {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:23:54.784126 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9cee245b-8974-4b69-bc47-b74fd30684e6 admin admin] RBAC: Authorization granted {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:23:54.836816 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG dbcounter [-] [81409] Writing DB stats keystone:SELECT=90,keystone:DELETE=9,keystone:INSERT=1 {{(pid=81409) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:23:54.863014 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.notifications [None req-9cee245b-8974-4b69-bc47-b74fd30684e6 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '03b3c3b7eb4f400f867f9b928c239311', 'request_id': 'req-9cee245b-8974-4b69-bc47-b74fd30684e6'} {{(pid=81409) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 13 04:23:54.877868 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.notifications [None req-9cee245b-8974-4b69-bc47-b74fd30684e6 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '03b3c3b7eb4f400f867f9b928c239311', 'request_id': 'req-9cee245b-8974-4b69-bc47-b74fd30684e6'} {{(pid=81409) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 13 04:23:54.893188 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.notifications [None req-9cee245b-8974-4b69-bc47-b74fd30684e6 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '03b3c3b7eb4f400f867f9b928c239311', 'request_id': 'req-9cee245b-8974-4b69-bc47-b74fd30684e6'} {{(pid=81409) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 13 04:23:54.922340 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.notifications [None req-9cee245b-8974-4b69-bc47-b74fd30684e6 admin admin] The token cache is being invalidate because project 03b3c3b7eb4f400f867f9b928c239311 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81409) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 13 04:23:54.923044 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.notifications [None req-9cee245b-8974-4b69-bc47-b74fd30684e6 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81409) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 13 04:23:54.924132 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.notifications [None req-9cee245b-8974-4b69-bc47-b74fd30684e6 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81409) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 13 04:23:54.931152 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 960/1891] 10.222.0.27 () {58 vars in 1233 bytes} [Thu Oct 13 04:23:54 2022] DELETE /identity/v3/projects/03b3c3b7eb4f400f867f9b928c239311 => generated 0 bytes in 372 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 13 04:23:54.932490 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4a1486b-0f09-4f5c-aaf6-01d4011fbc69 None None] REQUEST_METHOD: `POST` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:23:54.933190 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4a1486b-0f09-4f5c-aaf6-01d4011fbc69 None None] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:23:54.933190 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4a1486b-0f09-4f5c-aaf6-01d4011fbc69 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:23:54.970306 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.auth.core [None req-d4a1486b-0f09-4f5c-aaf6-01d4011fbc69 None None] MFA Rules not processed for user `0e2d979563d6421995451a0e86bf3022`. Rule list: `[]` (Enabled: `True`). {{(pid=81408) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 13 04:23:54.971842 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.fernet_utils [None req-d4a1486b-0f09-4f5c-aaf6-01d4011fbc69 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81408) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:23:55.072028 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 932/1892] 10.222.0.27 () {58 vars in 958 bytes} [Thu Oct 13 04:23:54 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 136 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 13 04:23:55.074025 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef5a448b-714c-496f-bbbc-18e25ac4a794 None None] REQUEST_METHOD: `POST` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:23:55.074344 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef5a448b-714c-496f-bbbc-18e25ac4a794 None None] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:23:55.074621 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef5a448b-714c-496f-bbbc-18e25ac4a794 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:23:55.127155 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.auth.core [None req-ef5a448b-714c-496f-bbbc-18e25ac4a794 None None] MFA Rules not processed for user `0e2d979563d6421995451a0e86bf3022`. Rule list: `[]` (Enabled: `True`). {{(pid=81409) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 13 04:23:55.129506 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.fernet_utils [None req-ef5a448b-714c-496f-bbbc-18e25ac4a794 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81409) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:23:55.144038 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 961/1893] 10.222.0.27 () {58 vars in 958 bytes} [Thu Oct 13 04:23:55 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 71 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 13 04:23:55.149825 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d4a1486b-0f09-4f5c-aaf6-01d4011fbc69 None None] Authenticating user token {{(pid=81408) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:23:55.177562 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d4a1486b-0f09-4f5c-aaf6-01d4011fbc69 None None] Validating token access rules against request {{(pid=81408) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:23:55.185611 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cd9f1557-483b-44f6-bc6e-71a61488c177 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8ba823f313144da3b683bacef8fd62e0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81408) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:23:55.187118 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd9f1557-483b-44f6-bc6e-71a61488c177 admin admin] REQUEST_METHOD: `GET` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:23:55.188786 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd9f1557-483b-44f6-bc6e-71a61488c177 admin admin] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:23:55.189535 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd9f1557-483b-44f6-bc6e-71a61488c177 admin admin] PATH_INFO: `/v3/domains` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:23:55.191365 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cd9f1557-483b-44f6-bc6e-71a61488c177 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81408) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 13 04:23:55.192355 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cd9f1557-483b-44f6-bc6e-71a61488c177 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:23:55.195999 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cd9f1557-483b-44f6-bc6e-71a61488c177 admin admin] RBAC: Authorization granted {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:23:55.215574 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 933/1894] 10.222.0.27 () {58 vars in 1153 bytes} [Thu Oct 13 04:23:55 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 66 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 13 04:23:55.221079 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ef5a448b-714c-496f-bbbc-18e25ac4a794 None None] Authenticating user token {{(pid=81409) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:23:55.238990 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ef5a448b-714c-496f-bbbc-18e25ac4a794 None None] Validating token access rules against request {{(pid=81409) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:23:55.255846 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f05504db-dcde-41b2-85de-fdd0dcbcf8d8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8ba823f313144da3b683bacef8fd62e0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81409) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:23:55.257417 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f05504db-dcde-41b2-85de-fdd0dcbcf8d8 admin admin] REQUEST_METHOD: `POST` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:23:55.257928 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f05504db-dcde-41b2-85de-fdd0dcbcf8d8 admin admin] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:23:55.258457 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f05504db-dcde-41b2-85de-fdd0dcbcf8d8 admin admin] PATH_INFO: `/v3/projects` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:23:55.259379 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f05504db-dcde-41b2-85de-fdd0dcbcf8d8 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:23:55.261520 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f05504db-dcde-41b2-85de-fdd0dcbcf8d8 admin admin] RBAC: Authorization granted {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:23:55.346654 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 962/1895] 10.222.0.27 () {60 vars in 1153 bytes} [Thu Oct 13 04:23:55 2022] POST /identity/v3/projects => generated 385 bytes in 126 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 13 04:23:55.350948 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cd9f1557-483b-44f6-bc6e-71a61488c177 admin admin] Authenticating user token {{(pid=81408) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:23:55.364929 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cd9f1557-483b-44f6-bc6e-71a61488c177 admin admin] Validating token access rules against request {{(pid=81408) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:23:55.586758 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1962cc9e-2771-4c66-9bec-1d17592e69b4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8ba823f313144da3b683bacef8fd62e0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81408) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:23:55.587844 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1962cc9e-2771-4c66-9bec-1d17592e69b4 admin admin] REQUEST_METHOD: `POST` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:23:55.588338 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1962cc9e-2771-4c66-9bec-1d17592e69b4 admin admin] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:23:55.588806 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1962cc9e-2771-4c66-9bec-1d17592e69b4 admin admin] PATH_INFO: `/v3/users` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:23:55.590024 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1962cc9e-2771-4c66-9bec-1d17592e69b4 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:23:55.592219 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1962cc9e-2771-4c66-9bec-1d17592e69b4 admin admin] RBAC: Authorization granted {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:23:55.687312 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f05504db-dcde-41b2-85de-fdd0dcbcf8d8 admin admin] Authenticating user token {{(pid=81409) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:23:55.696989 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 934/1896] 10.222.0.27 () {60 vars in 1144 bytes} [Thu Oct 13 04:23:55 2022] POST /identity/v3/users => generated 293 bytes in 346 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 13 04:23:55.701138 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f05504db-dcde-41b2-85de-fdd0dcbcf8d8 admin admin] Validating token access rules against request {{(pid=81409) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:23:55.718333 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cf2dae6d-650f-4941-96c3-439880427e9c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8ba823f313144da3b683bacef8fd62e0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81409) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:23:55.719824 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf2dae6d-650f-4941-96c3-439880427e9c admin admin] REQUEST_METHOD: `GET` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:23:55.720322 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf2dae6d-650f-4941-96c3-439880427e9c admin admin] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:23:55.720793 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf2dae6d-650f-4941-96c3-439880427e9c admin admin] PATH_INFO: `/v3/roles` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:23:55.721653 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cf2dae6d-650f-4941-96c3-439880427e9c admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:23:55.723543 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cf2dae6d-650f-4941-96c3-439880427e9c admin admin] RBAC: Authorization granted {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:23:55.762027 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1962cc9e-2771-4c66-9bec-1d17592e69b4 admin admin] Authenticating user token {{(pid=81408) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:23:55.768543 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 963/1897] 10.222.0.27 () {58 vars in 1122 bytes} [Thu Oct 13 04:23:55 2022] GET /identity/v3/roles => generated 1420 bytes in 82 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 13 04:23:55.796245 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1962cc9e-2771-4c66-9bec-1d17592e69b4 admin admin] Validating token access rules against request {{(pid=81408) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:23:55.812121 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7f2a6116-8901-44e4-9068-23631151c349 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8ba823f313144da3b683bacef8fd62e0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81408) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:23:55.814606 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f2a6116-8901-44e4-9068-23631151c349 admin admin] REQUEST_METHOD: `GET` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:23:55.818571 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f2a6116-8901-44e4-9068-23631151c349 admin admin] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:23:55.819094 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f2a6116-8901-44e4-9068-23631151c349 admin admin] PATH_INFO: `/v3/roles` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:23:55.820002 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7f2a6116-8901-44e4-9068-23631151c349 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:23:55.821868 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7f2a6116-8901-44e4-9068-23631151c349 admin admin] RBAC: Authorization granted {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:23:55.865254 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 935/1898] 10.222.0.27 () {58 vars in 1122 bytes} [Thu Oct 13 04:23:55 2022] GET /identity/v3/roles => generated 1420 bytes in 104 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 13 04:23:55.873778 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cf2dae6d-650f-4941-96c3-439880427e9c admin admin] Authenticating user token {{(pid=81409) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:23:55.944515 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cf2dae6d-650f-4941-96c3-439880427e9c admin admin] Validating token access rules against request {{(pid=81409) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:23:55.988613 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9f426ac9-b0b3-49cf-8ad3-cdc94a8440c7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8ba823f313144da3b683bacef8fd62e0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81409) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:23:55.989260 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f426ac9-b0b3-49cf-8ad3-cdc94a8440c7 admin admin] REQUEST_METHOD: `PUT` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:23:55.989545 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f426ac9-b0b3-49cf-8ad3-cdc94a8440c7 admin admin] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:23:55.989817 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f426ac9-b0b3-49cf-8ad3-cdc94a8440c7 admin admin] PATH_INFO: `/v3/projects/ca90bafb9f5b4ab9b47432d9da9ce184/users/d7ef5291265543e4bf426403fc9d8968/roles/cc56c4ae13a7424ea771c15da4b153c1` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:23:56.026089 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9f426ac9-b0b3-49cf-8ad3-cdc94a8440c7 admin admin] RBAC: Authorizing `identity:create_grant(project_id=ca90bafb9f5b4ab9b47432d9da9ce184, user_id=d7ef5291265543e4bf426403fc9d8968, role_id=cc56c4ae13a7424ea771c15da4b153c1)` {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:23:56.029679 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9f426ac9-b0b3-49cf-8ad3-cdc94a8440c7 admin admin] RBAC: Authorization granted {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:23:56.046474 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 964/1899] 10.222.0.27 () {60 vars in 1483 bytes} [Thu Oct 13 04:23:55 2022] PUT /identity/v3/projects/ca90bafb9f5b4ab9b47432d9da9ce184/users/d7ef5291265543e4bf426403fc9d8968/roles/cc56c4ae13a7424ea771c15da4b153c1 => generated 0 bytes in 173 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 13 04:23:56.058264 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c388f345-72fb-41d3-923c-79ae31023a18 None None] REQUEST_METHOD: `POST` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:23:56.058690 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c388f345-72fb-41d3-923c-79ae31023a18 None None] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:23:56.058771 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c388f345-72fb-41d3-923c-79ae31023a18 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:23:56.071596 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG dbcounter [-] [81408] Writing DB stats keystone:DELETE=7,keystone:SELECT=89,keystone:INSERT=4 {{(pid=81408) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:23:56.114758 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.auth.core [None req-c388f345-72fb-41d3-923c-79ae31023a18 None None] MFA Rules not processed for user `d7ef5291265543e4bf426403fc9d8968`. Rule list: `[]` (Enabled: `True`). {{(pid=81408) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 13 04:23:56.116052 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.fernet_utils [None req-c388f345-72fb-41d3-923c-79ae31023a18 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81408) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:23:56.235793 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 936/1900] 10.222.0.27 () {58 vars in 958 bytes} [Thu Oct 13 04:23:56 2022] POST /identity/v3/auth/tokens => generated 3262 bytes in 176 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 13 04:23:56.243565 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9f426ac9-b0b3-49cf-8ad3-cdc94a8440c7 admin admin] Authenticating user token {{(pid=81409) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:23:56.247201 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.fernet_utils [None req-9f426ac9-b0b3-49cf-8ad3-cdc94a8440c7 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81409) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:23:56.336960 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9f426ac9-b0b3-49cf-8ad3-cdc94a8440c7 admin admin] Validating token access rules against request {{(pid=81409) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:23:56.341577 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-001018cb-4e65-4092-b122-1244e29f32be service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6141b8a6c3d54e179d6879bb019f4d12', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dcec0431fb6849499d6aec35a234daa6', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81409) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:23:56.342233 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-001018cb-4e65-4092-b122-1244e29f32be service cinder] REQUEST_METHOD: `GET` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:23:56.342517 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-001018cb-4e65-4092-b122-1244e29f32be service cinder] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:23:56.342805 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-001018cb-4e65-4092-b122-1244e29f32be service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:23:56.354796 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-001018cb-4e65-4092-b122-1244e29f32be service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:23:56.357097 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-001018cb-4e65-4092-b122-1244e29f32be service cinder] RBAC: Authorization granted {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:23:56.380683 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 965/1901] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Oct 13 04:23:56 2022] GET /identity/v3/auth/tokens => generated 3262 bytes in 137 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 13 04:23:56.394123 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c388f345-72fb-41d3-923c-79ae31023a18 None None] Authenticating user token {{(pid=81408) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:23:56.525708 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c388f345-72fb-41d3-923c-79ae31023a18 None None] Validating token access rules against request {{(pid=81408) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:23:56.530610 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a0c4bac4-9101-492c-a2bf-ef710300835d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8ba823f313144da3b683bacef8fd62e0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81408) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:23:56.532156 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0c4bac4-9101-492c-a2bf-ef710300835d admin admin] REQUEST_METHOD: `DELETE` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:23:56.532561 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0c4bac4-9101-492c-a2bf-ef710300835d admin admin] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:23:56.532838 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0c4bac4-9101-492c-a2bf-ef710300835d admin admin] PATH_INFO: `/v3/users/d7ef5291265543e4bf426403fc9d8968` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:23:56.534007 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a0c4bac4-9101-492c-a2bf-ef710300835d admin admin] RBAC: Authorizing `identity:delete_user(user_id=d7ef5291265543e4bf426403fc9d8968)` {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:23:56.536418 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a0c4bac4-9101-492c-a2bf-ef710300835d admin admin] RBAC: Authorization granted {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:23:56.618844 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.notifications [None req-a0c4bac4-9101-492c-a2bf-ef710300835d admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'd7ef5291265543e4bf426403fc9d8968'} {{(pid=81408) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 13 04:23:56.647366 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.notifications [None req-a0c4bac4-9101-492c-a2bf-ef710300835d admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'd7ef5291265543e4bf426403fc9d8968'} {{(pid=81408) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 13 04:23:56.651346 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.notifications [None req-a0c4bac4-9101-492c-a2bf-ef710300835d admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'd7ef5291265543e4bf426403fc9d8968'} {{(pid=81408) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 13 04:23:56.653112 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.notifications [None req-a0c4bac4-9101-492c-a2bf-ef710300835d admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'd7ef5291265543e4bf426403fc9d8968'} {{(pid=81408) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 13 04:23:56.682073 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.notifications [None req-a0c4bac4-9101-492c-a2bf-ef710300835d admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'd7ef5291265543e4bf426403fc9d8968'} {{(pid=81408) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 13 04:23:56.730879 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 937/1902] 10.222.0.27 () {58 vars in 1224 bytes} [Thu Oct 13 04:23:56 2022] DELETE /identity/v3/users/d7ef5291265543e4bf426403fc9d8968 => generated 0 bytes in 335 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 13 04:23:56.733449 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61b5eed7-abd2-4677-8807-85e128bf941e None None] REQUEST_METHOD: `POST` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:23:56.733934 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61b5eed7-abd2-4677-8807-85e128bf941e None None] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:23:56.734061 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61b5eed7-abd2-4677-8807-85e128bf941e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:23:56.772471 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.auth.core [None req-61b5eed7-abd2-4677-8807-85e128bf941e None None] MFA Rules not processed for user `0e2d979563d6421995451a0e86bf3022`. Rule list: `[]` (Enabled: `True`). {{(pid=81409) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 13 04:23:56.773746 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.fernet_utils [None req-61b5eed7-abd2-4677-8807-85e128bf941e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81409) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:23:56.875282 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG dbcounter [-] [81409] Writing DB stats keystone:SELECT=92,keystone:DELETE=5,keystone:INSERT=3 {{(pid=81409) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:23:56.905354 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 966/1903] 10.222.0.27 () {58 vars in 958 bytes} [Thu Oct 13 04:23:56 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 173 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 13 04:23:56.916723 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a0c4bac4-9101-492c-a2bf-ef710300835d admin admin] Authenticating user token {{(pid=81408) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:23:56.918887 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.fernet_utils [None req-a0c4bac4-9101-492c-a2bf-ef710300835d admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81408) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:23:56.941845 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG dbcounter [-] [81408] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81408) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:23:57.078748 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a0c4bac4-9101-492c-a2bf-ef710300835d admin admin] Validating token access rules against request {{(pid=81408) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:23:57.093908 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2372eab4-756c-481f-a347-ad8f920a338e service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd8812c3974f84871bb2d761cf0016257', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dcec0431fb6849499d6aec35a234daa6', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81408) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:23:57.094897 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2372eab4-756c-481f-a347-ad8f920a338e service neutron] REQUEST_METHOD: `GET` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:23:57.095380 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2372eab4-756c-481f-a347-ad8f920a338e service neutron] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:23:57.095851 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2372eab4-756c-481f-a347-ad8f920a338e service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:23:57.114735 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2372eab4-756c-481f-a347-ad8f920a338e service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:23:57.116889 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2372eab4-756c-481f-a347-ad8f920a338e service neutron] RBAC: Authorization granted {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:23:57.133269 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 938/1904] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Oct 13 04:23:56 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 217 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 13 04:23:57.548186 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-61b5eed7-abd2-4677-8807-85e128bf941e None None] Authenticating user token {{(pid=81409) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:23:57.550557 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.fernet_utils [None req-61b5eed7-abd2-4677-8807-85e128bf941e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81409) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:23:57.573541 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-61b5eed7-abd2-4677-8807-85e128bf941e None None] Validating token access rules against request {{(pid=81409) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:23:57.579324 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6dc9d6ee-2f2c-4bcd-87b0-f02ecea8b8ab admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8ba823f313144da3b683bacef8fd62e0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81409) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:23:57.579987 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6dc9d6ee-2f2c-4bcd-87b0-f02ecea8b8ab admin admin] REQUEST_METHOD: `DELETE` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:23:57.580280 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6dc9d6ee-2f2c-4bcd-87b0-f02ecea8b8ab admin admin] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:23:57.580579 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6dc9d6ee-2f2c-4bcd-87b0-f02ecea8b8ab admin admin] PATH_INFO: `/v3/projects/ca90bafb9f5b4ab9b47432d9da9ce184` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:23:57.581976 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6dc9d6ee-2f2c-4bcd-87b0-f02ecea8b8ab admin admin] RBAC: Authorizing `identity:delete_project(project_id=ca90bafb9f5b4ab9b47432d9da9ce184)` {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:23:57.584020 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6dc9d6ee-2f2c-4bcd-87b0-f02ecea8b8ab admin admin] RBAC: Authorization granted {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:23:57.645350 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.notifications [None req-6dc9d6ee-2f2c-4bcd-87b0-f02ecea8b8ab admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'ca90bafb9f5b4ab9b47432d9da9ce184', 'request_id': 'req-6dc9d6ee-2f2c-4bcd-87b0-f02ecea8b8ab'} {{(pid=81409) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 13 04:23:57.654415 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.notifications [None req-6dc9d6ee-2f2c-4bcd-87b0-f02ecea8b8ab admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'ca90bafb9f5b4ab9b47432d9da9ce184', 'request_id': 'req-6dc9d6ee-2f2c-4bcd-87b0-f02ecea8b8ab'} {{(pid=81409) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 13 04:23:57.661824 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.notifications [None req-6dc9d6ee-2f2c-4bcd-87b0-f02ecea8b8ab admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'ca90bafb9f5b4ab9b47432d9da9ce184', 'request_id': 'req-6dc9d6ee-2f2c-4bcd-87b0-f02ecea8b8ab'} {{(pid=81409) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 13 04:23:57.675630 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.notifications [None req-6dc9d6ee-2f2c-4bcd-87b0-f02ecea8b8ab admin admin] The token cache is being invalidate because project ca90bafb9f5b4ab9b47432d9da9ce184 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81409) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 13 04:23:57.676026 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.notifications [None req-6dc9d6ee-2f2c-4bcd-87b0-f02ecea8b8ab admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81409) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 13 04:23:57.676779 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.notifications [None req-6dc9d6ee-2f2c-4bcd-87b0-f02ecea8b8ab admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81409) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 13 04:23:57.678640 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 967/1905] 10.222.0.27 () {58 vars in 1233 bytes} [Thu Oct 13 04:23:57 2022] DELETE /identity/v3/projects/ca90bafb9f5b4ab9b47432d9da9ce184 => generated 0 bytes in 131 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 13 04:23:57.683837 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8273bb98-1357-4513-b8bb-3d96aa56a080 None None] REQUEST_METHOD: `POST` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:23:57.684175 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8273bb98-1357-4513-b8bb-3d96aa56a080 None None] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:23:57.684511 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8273bb98-1357-4513-b8bb-3d96aa56a080 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:23:57.709133 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.auth.core [None req-8273bb98-1357-4513-b8bb-3d96aa56a080 None None] MFA Rules not processed for user `0e2d979563d6421995451a0e86bf3022`. Rule list: `[]` (Enabled: `True`). {{(pid=81408) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 13 04:23:57.710408 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.fernet_utils [None req-8273bb98-1357-4513-b8bb-3d96aa56a080 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81408) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:23:57.782254 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 939/1906] 10.222.0.27 () {58 vars in 958 bytes} [Thu Oct 13 04:23:57 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 98 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 13 04:23:57.789502 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b3057f6-a17b-4f1c-9d1d-893c90a77ca0 None None] REQUEST_METHOD: `POST` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:23:57.790010 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b3057f6-a17b-4f1c-9d1d-893c90a77ca0 None None] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:23:57.790010 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b3057f6-a17b-4f1c-9d1d-893c90a77ca0 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:23:57.810976 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.auth.core [None req-0b3057f6-a17b-4f1c-9d1d-893c90a77ca0 None None] MFA Rules not processed for user `0e2d979563d6421995451a0e86bf3022`. Rule list: `[]` (Enabled: `True`). {{(pid=81409) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 13 04:23:57.812784 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.fernet_utils [None req-0b3057f6-a17b-4f1c-9d1d-893c90a77ca0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81409) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:23:57.821516 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 968/1907] 10.222.0.27 () {58 vars in 958 bytes} [Thu Oct 13 04:23:57 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 33 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 13 04:23:57.825414 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8273bb98-1357-4513-b8bb-3d96aa56a080 None None] Authenticating user token {{(pid=81408) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:23:57.837356 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8273bb98-1357-4513-b8bb-3d96aa56a080 None None] Validating token access rules against request {{(pid=81408) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:23:57.842385 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5b7fafe5-99cd-480b-b8b5-47c000888d1e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8ba823f313144da3b683bacef8fd62e0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81408) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:23:57.843506 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b7fafe5-99cd-480b-b8b5-47c000888d1e admin admin] REQUEST_METHOD: `GET` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:23:57.844001 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b7fafe5-99cd-480b-b8b5-47c000888d1e admin admin] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:23:57.844282 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b7fafe5-99cd-480b-b8b5-47c000888d1e admin admin] PATH_INFO: `/v3/domains` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:23:57.845212 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5b7fafe5-99cd-480b-b8b5-47c000888d1e admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81408) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 13 04:23:57.845533 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5b7fafe5-99cd-480b-b8b5-47c000888d1e admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:23:57.848038 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5b7fafe5-99cd-480b-b8b5-47c000888d1e admin admin] RBAC: Authorization granted {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:23:57.862357 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 940/1908] 10.222.0.27 () {58 vars in 1153 bytes} [Thu Oct 13 04:23:57 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 37 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 13 04:23:57.866088 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0b3057f6-a17b-4f1c-9d1d-893c90a77ca0 None None] Authenticating user token {{(pid=81409) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:23:57.880947 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0b3057f6-a17b-4f1c-9d1d-893c90a77ca0 None None] Validating token access rules against request {{(pid=81409) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:23:57.887265 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-83ecedef-a9b0-4730-940e-a6beb674904d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8ba823f313144da3b683bacef8fd62e0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81409) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:23:57.887827 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83ecedef-a9b0-4730-940e-a6beb674904d admin admin] REQUEST_METHOD: `POST` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:23:57.888102 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83ecedef-a9b0-4730-940e-a6beb674904d admin admin] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:23:57.888369 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83ecedef-a9b0-4730-940e-a6beb674904d admin admin] PATH_INFO: `/v3/projects` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:23:57.889013 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-83ecedef-a9b0-4730-940e-a6beb674904d admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:23:57.892103 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-83ecedef-a9b0-4730-940e-a6beb674904d admin admin] RBAC: Authorization granted {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:23:57.947567 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 969/1909] 10.222.0.27 () {60 vars in 1153 bytes} [Thu Oct 13 04:23:57 2022] POST /identity/v3/projects => generated 371 bytes in 79 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 13 04:23:57.951838 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5b7fafe5-99cd-480b-b8b5-47c000888d1e admin admin] Authenticating user token {{(pid=81408) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:23:57.974255 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5b7fafe5-99cd-480b-b8b5-47c000888d1e admin admin] Validating token access rules against request {{(pid=81408) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:23:58.096158 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-61db728e-ffc8-4d1f-9cc5-2ad4e4b0329c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8ba823f313144da3b683bacef8fd62e0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81408) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:23:58.096915 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61db728e-ffc8-4d1f-9cc5-2ad4e4b0329c admin admin] REQUEST_METHOD: `POST` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:23:58.097340 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61db728e-ffc8-4d1f-9cc5-2ad4e4b0329c admin admin] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:23:58.097710 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61db728e-ffc8-4d1f-9cc5-2ad4e4b0329c admin admin] PATH_INFO: `/v3/users` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:23:58.113811 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-61db728e-ffc8-4d1f-9cc5-2ad4e4b0329c admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:23:58.113811 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-61db728e-ffc8-4d1f-9cc5-2ad4e4b0329c admin admin] RBAC: Authorization granted {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:23:58.146489 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 941/1910] 10.222.0.27 () {60 vars in 1144 bytes} [Thu Oct 13 04:23:57 2022] POST /identity/v3/users => generated 286 bytes in 192 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 13 04:23:58.147933 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-83ecedef-a9b0-4730-940e-a6beb674904d admin admin] Authenticating user token {{(pid=81409) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:23:58.162976 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-83ecedef-a9b0-4730-940e-a6beb674904d admin admin] Validating token access rules against request {{(pid=81409) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:23:58.173285 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c8ba9da4-d2c2-4bfa-ab51-8231b3157053 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8ba823f313144da3b683bacef8fd62e0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81409) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:23:58.174276 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8ba9da4-d2c2-4bfa-ab51-8231b3157053 admin admin] REQUEST_METHOD: `GET` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:23:58.174775 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8ba9da4-d2c2-4bfa-ab51-8231b3157053 admin admin] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:23:58.175228 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8ba9da4-d2c2-4bfa-ab51-8231b3157053 admin admin] PATH_INFO: `/v3/roles` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:23:58.176058 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c8ba9da4-d2c2-4bfa-ab51-8231b3157053 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:23:58.178249 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c8ba9da4-d2c2-4bfa-ab51-8231b3157053 admin admin] RBAC: Authorization granted {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:23:58.194702 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 970/1911] 10.222.0.27 () {58 vars in 1122 bytes} [Thu Oct 13 04:23:58 2022] GET /identity/v3/roles => generated 1420 bytes in 45 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 13 04:23:58.196172 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-61db728e-ffc8-4d1f-9cc5-2ad4e4b0329c admin admin] Authenticating user token {{(pid=81408) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:23:58.210035 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-61db728e-ffc8-4d1f-9cc5-2ad4e4b0329c admin admin] Validating token access rules against request {{(pid=81408) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:23:58.215842 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1cb1e256-a294-4eb8-96c1-d496150280fd admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8ba823f313144da3b683bacef8fd62e0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81408) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:23:58.216739 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1cb1e256-a294-4eb8-96c1-d496150280fd admin admin] REQUEST_METHOD: `GET` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:23:58.217215 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1cb1e256-a294-4eb8-96c1-d496150280fd admin admin] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:23:58.217677 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1cb1e256-a294-4eb8-96c1-d496150280fd admin admin] PATH_INFO: `/v3/roles` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:23:58.219282 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1cb1e256-a294-4eb8-96c1-d496150280fd admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:23:58.221255 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1cb1e256-a294-4eb8-96c1-d496150280fd admin admin] RBAC: Authorization granted {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:23:58.242407 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c8ba9da4-d2c2-4bfa-ab51-8231b3157053 admin admin] Authenticating user token {{(pid=81409) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:23:58.253986 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 942/1912] 10.222.0.27 () {58 vars in 1122 bytes} [Thu Oct 13 04:23:58 2022] GET /identity/v3/roles => generated 1420 bytes in 48 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 13 04:23:58.255816 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c8ba9da4-d2c2-4bfa-ab51-8231b3157053 admin admin] Validating token access rules against request {{(pid=81409) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:23:58.263642 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2dcafadf-78ca-4b82-835d-06c4ea878df3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8ba823f313144da3b683bacef8fd62e0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81409) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:23:58.265357 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2dcafadf-78ca-4b82-835d-06c4ea878df3 admin admin] REQUEST_METHOD: `PUT` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:23:58.266447 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2dcafadf-78ca-4b82-835d-06c4ea878df3 admin admin] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:23:58.267084 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2dcafadf-78ca-4b82-835d-06c4ea878df3 admin admin] PATH_INFO: `/v3/projects/8ef4864e03e84de3bd599556277337cc/users/cb3a3ad9aa7c4705a508dc2272819848/roles/cc56c4ae13a7424ea771c15da4b153c1` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:23:58.289238 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2dcafadf-78ca-4b82-835d-06c4ea878df3 admin admin] RBAC: Authorizing `identity:create_grant(project_id=8ef4864e03e84de3bd599556277337cc, user_id=cb3a3ad9aa7c4705a508dc2272819848, role_id=cc56c4ae13a7424ea771c15da4b153c1)` {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:23:58.292292 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2dcafadf-78ca-4b82-835d-06c4ea878df3 admin admin] RBAC: Authorization granted {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:23:58.322427 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8be5f64-79ab-4b4a-ae55-0243974f1b2a None None] REQUEST_METHOD: `POST` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:23:58.322852 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8be5f64-79ab-4b4a-ae55-0243974f1b2a None None] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:23:58.323225 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8be5f64-79ab-4b4a-ae55-0243974f1b2a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:23:58.333736 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 971/1913] 10.222.0.27 () {60 vars in 1483 bytes} [Thu Oct 13 04:23:58 2022] PUT /identity/v3/projects/8ef4864e03e84de3bd599556277337cc/users/cb3a3ad9aa7c4705a508dc2272819848/roles/cc56c4ae13a7424ea771c15da4b153c1 => generated 0 bytes in 70 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 13 04:23:58.369590 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.auth.core [None req-c8be5f64-79ab-4b4a-ae55-0243974f1b2a None None] MFA Rules not processed for user `cb3a3ad9aa7c4705a508dc2272819848`. Rule list: `[]` (Enabled: `True`). {{(pid=81408) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 13 04:23:58.372344 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.fernet_utils [None req-c8be5f64-79ab-4b4a-ae55-0243974f1b2a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81408) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:23:58.401562 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG dbcounter [-] [81408] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81408) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:23:58.549579 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 943/1914] 10.222.0.27 () {58 vars in 958 bytes} [Thu Oct 13 04:23:58 2022] POST /identity/v3/auth/tokens => generated 3248 bytes in 228 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 13 04:23:58.559120 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2dcafadf-78ca-4b82-835d-06c4ea878df3 admin admin] Authenticating user token {{(pid=81409) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:23:58.574476 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.fernet_utils [None req-2dcafadf-78ca-4b82-835d-06c4ea878df3 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81409) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:23:58.654446 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2dcafadf-78ca-4b82-835d-06c4ea878df3 admin admin] Validating token access rules against request {{(pid=81409) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:23:58.673125 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-412950db-4911-426f-ae4b-9c0d8a417aee service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6141b8a6c3d54e179d6879bb019f4d12', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dcec0431fb6849499d6aec35a234daa6', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81409) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:23:58.673125 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-412950db-4911-426f-ae4b-9c0d8a417aee service cinder] REQUEST_METHOD: `GET` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:23:58.673125 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-412950db-4911-426f-ae4b-9c0d8a417aee service cinder] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:23:58.673125 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-412950db-4911-426f-ae4b-9c0d8a417aee service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:23:58.686019 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-412950db-4911-426f-ae4b-9c0d8a417aee service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:23:58.694490 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-412950db-4911-426f-ae4b-9c0d8a417aee service cinder] RBAC: Authorization granted {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:23:58.707857 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 972/1915] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Oct 13 04:23:58 2022] GET /identity/v3/auth/tokens => generated 3248 bytes in 150 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 13 04:23:58.737944 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c8be5f64-79ab-4b4a-ae55-0243974f1b2a None None] Authenticating user token {{(pid=81408) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:23:58.837225 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c8be5f64-79ab-4b4a-ae55-0243974f1b2a None None] Validating token access rules against request {{(pid=81408) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:23:58.846675 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6b3abf47-ce35-41ed-a053-0fa9d9cb8448 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8ba823f313144da3b683bacef8fd62e0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81408) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:23:58.847486 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b3abf47-ce35-41ed-a053-0fa9d9cb8448 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:23:58.847881 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b3abf47-ce35-41ed-a053-0fa9d9cb8448 admin admin] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:23:58.848420 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b3abf47-ce35-41ed-a053-0fa9d9cb8448 admin admin] PATH_INFO: `/v3/users/cb3a3ad9aa7c4705a508dc2272819848` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:23:58.849781 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6b3abf47-ce35-41ed-a053-0fa9d9cb8448 admin admin] RBAC: Authorizing `identity:delete_user(user_id=cb3a3ad9aa7c4705a508dc2272819848)` {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:23:58.852107 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6b3abf47-ce35-41ed-a053-0fa9d9cb8448 admin admin] RBAC: Authorization granted {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:23:58.950421 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.notifications [None req-6b3abf47-ce35-41ed-a053-0fa9d9cb8448 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'cb3a3ad9aa7c4705a508dc2272819848'} {{(pid=81408) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 13 04:23:58.958359 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.notifications [None req-6b3abf47-ce35-41ed-a053-0fa9d9cb8448 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'cb3a3ad9aa7c4705a508dc2272819848'} {{(pid=81408) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 13 04:23:58.959269 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.notifications [None req-6b3abf47-ce35-41ed-a053-0fa9d9cb8448 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'cb3a3ad9aa7c4705a508dc2272819848'} {{(pid=81408) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 13 04:23:58.959699 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.notifications [None req-6b3abf47-ce35-41ed-a053-0fa9d9cb8448 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'cb3a3ad9aa7c4705a508dc2272819848'} {{(pid=81408) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 13 04:23:58.972343 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.notifications [None req-6b3abf47-ce35-41ed-a053-0fa9d9cb8448 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'cb3a3ad9aa7c4705a508dc2272819848'} {{(pid=81408) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 13 04:23:59.001253 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 944/1916] 10.222.0.27 () {58 vars in 1224 bytes} [Thu Oct 13 04:23:58 2022] DELETE /identity/v3/users/cb3a3ad9aa7c4705a508dc2272819848 => generated 0 bytes in 264 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 13 04:23:59.005587 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1696674-6428-40af-965d-007d788bcbe7 None None] REQUEST_METHOD: `POST` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:23:59.006137 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1696674-6428-40af-965d-007d788bcbe7 None None] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:23:59.006137 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1696674-6428-40af-965d-007d788bcbe7 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:23:59.043453 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.auth.core [None req-c1696674-6428-40af-965d-007d788bcbe7 None None] MFA Rules not processed for user `0e2d979563d6421995451a0e86bf3022`. Rule list: `[]` (Enabled: `True`). {{(pid=81409) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 13 04:23:59.045449 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.fernet_utils [None req-c1696674-6428-40af-965d-007d788bcbe7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81409) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:23:59.070515 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG dbcounter [-] [81409] Writing DB stats keystone:SELECT=90,keystone:DELETE=7,keystone:INSERT=3 {{(pid=81409) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:23:59.146020 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 973/1917] 10.222.0.27 () {58 vars in 958 bytes} [Thu Oct 13 04:23:59 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 141 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 13 04:23:59.155959 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6b3abf47-ce35-41ed-a053-0fa9d9cb8448 admin admin] Authenticating user token {{(pid=81408) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:23:59.157837 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.fernet_utils [None req-6b3abf47-ce35-41ed-a053-0fa9d9cb8448 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81408) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:23:59.220641 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG dbcounter [-] [81408] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81408) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:23:59.267897 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6b3abf47-ce35-41ed-a053-0fa9d9cb8448 admin admin] Validating token access rules against request {{(pid=81408) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:23:59.273829 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-76570312-0ee9-4bd6-9e8b-0f17780ed0c1 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd8812c3974f84871bb2d761cf0016257', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dcec0431fb6849499d6aec35a234daa6', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81408) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:23:59.274835 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76570312-0ee9-4bd6-9e8b-0f17780ed0c1 service neutron] REQUEST_METHOD: `GET` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:23:59.275491 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76570312-0ee9-4bd6-9e8b-0f17780ed0c1 service neutron] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:23:59.275885 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76570312-0ee9-4bd6-9e8b-0f17780ed0c1 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:23:59.290807 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-76570312-0ee9-4bd6-9e8b-0f17780ed0c1 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:23:59.293806 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-76570312-0ee9-4bd6-9e8b-0f17780ed0c1 service neutron] RBAC: Authorization granted {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:23:59.310739 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 945/1918] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Oct 13 04:23:59 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 155 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 13 04:23:59.689271 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c1696674-6428-40af-965d-007d788bcbe7 None None] Authenticating user token {{(pid=81409) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:23:59.691248 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.fernet_utils [None req-c1696674-6428-40af-965d-007d788bcbe7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81409) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:23:59.712603 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c1696674-6428-40af-965d-007d788bcbe7 None None] Validating token access rules against request {{(pid=81409) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:23:59.730958 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-663e0466-21fb-43ef-b25a-2d688b10b654 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8ba823f313144da3b683bacef8fd62e0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81409) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:23:59.732415 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-663e0466-21fb-43ef-b25a-2d688b10b654 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:23:59.733384 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-663e0466-21fb-43ef-b25a-2d688b10b654 admin admin] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:23:59.733938 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-663e0466-21fb-43ef-b25a-2d688b10b654 admin admin] PATH_INFO: `/v3/projects/8ef4864e03e84de3bd599556277337cc` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:23:59.738384 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-663e0466-21fb-43ef-b25a-2d688b10b654 admin admin] RBAC: Authorizing `identity:delete_project(project_id=8ef4864e03e84de3bd599556277337cc)` {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:23:59.740742 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-663e0466-21fb-43ef-b25a-2d688b10b654 admin admin] RBAC: Authorization granted {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:23:59.864410 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.notifications [None req-663e0466-21fb-43ef-b25a-2d688b10b654 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '8ef4864e03e84de3bd599556277337cc', 'request_id': 'req-663e0466-21fb-43ef-b25a-2d688b10b654'} {{(pid=81409) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 13 04:23:59.880550 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.notifications [None req-663e0466-21fb-43ef-b25a-2d688b10b654 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '8ef4864e03e84de3bd599556277337cc', 'request_id': 'req-663e0466-21fb-43ef-b25a-2d688b10b654'} {{(pid=81409) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 13 04:23:59.889580 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.notifications [None req-663e0466-21fb-43ef-b25a-2d688b10b654 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '8ef4864e03e84de3bd599556277337cc', 'request_id': 'req-663e0466-21fb-43ef-b25a-2d688b10b654'} {{(pid=81409) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 13 04:23:59.906159 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.notifications [None req-663e0466-21fb-43ef-b25a-2d688b10b654 admin admin] The token cache is being invalidate because project 8ef4864e03e84de3bd599556277337cc was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81409) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 13 04:23:59.906840 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.notifications [None req-663e0466-21fb-43ef-b25a-2d688b10b654 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81409) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 13 04:23:59.908346 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.notifications [None req-663e0466-21fb-43ef-b25a-2d688b10b654 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81409) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 13 04:23:59.910054 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 974/1919] 10.222.0.27 () {58 vars in 1233 bytes} [Thu Oct 13 04:23:59 2022] DELETE /identity/v3/projects/8ef4864e03e84de3bd599556277337cc => generated 0 bytes in 222 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 13 04:23:59.914958 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f99df2c0-c780-40e8-9ff3-a6dcfbb70830 None None] REQUEST_METHOD: `POST` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:23:59.915240 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f99df2c0-c780-40e8-9ff3-a6dcfbb70830 None None] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:23:59.915514 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f99df2c0-c780-40e8-9ff3-a6dcfbb70830 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:23:59.959997 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.auth.core [None req-f99df2c0-c780-40e8-9ff3-a6dcfbb70830 None None] MFA Rules not processed for user `0e2d979563d6421995451a0e86bf3022`. Rule list: `[]` (Enabled: `True`). {{(pid=81408) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 13 04:23:59.961461 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.fernet_utils [None req-f99df2c0-c780-40e8-9ff3-a6dcfbb70830 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81408) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:24:00.028886 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 946/1920] 10.222.0.27 () {58 vars in 958 bytes} [Thu Oct 13 04:23:59 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 115 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 13 04:24:00.036544 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6adaa213-c5c2-4dce-a841-e1fc12d43fbb None None] REQUEST_METHOD: `POST` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:24:00.036993 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6adaa213-c5c2-4dce-a841-e1fc12d43fbb None None] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:24:00.037084 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6adaa213-c5c2-4dce-a841-e1fc12d43fbb None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:24:00.057437 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.auth.core [None req-6adaa213-c5c2-4dce-a841-e1fc12d43fbb None None] MFA Rules not processed for user `0e2d979563d6421995451a0e86bf3022`. Rule list: `[]` (Enabled: `True`). {{(pid=81409) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 13 04:24:00.058675 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.fernet_utils [None req-6adaa213-c5c2-4dce-a841-e1fc12d43fbb None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81409) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:24:00.067363 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 975/1921] 10.222.0.27 () {58 vars in 958 bytes} [Thu Oct 13 04:24:00 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 32 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 13 04:24:00.071438 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f99df2c0-c780-40e8-9ff3-a6dcfbb70830 None None] Authenticating user token {{(pid=81408) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:24:00.081737 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f99df2c0-c780-40e8-9ff3-a6dcfbb70830 None None] Validating token access rules against request {{(pid=81408) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:24:00.086288 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0b28208b-5c66-436f-8a5e-90f8b7f3f345 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8ba823f313144da3b683bacef8fd62e0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81408) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:24:00.086885 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b28208b-5c66-436f-8a5e-90f8b7f3f345 admin admin] REQUEST_METHOD: `GET` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:24:00.087164 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b28208b-5c66-436f-8a5e-90f8b7f3f345 admin admin] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:24:00.087436 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b28208b-5c66-436f-8a5e-90f8b7f3f345 admin admin] PATH_INFO: `/v3/domains` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:24:00.088078 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0b28208b-5c66-436f-8a5e-90f8b7f3f345 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81408) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 13 04:24:00.088379 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0b28208b-5c66-436f-8a5e-90f8b7f3f345 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:24:00.089976 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0b28208b-5c66-436f-8a5e-90f8b7f3f345 admin admin] RBAC: Authorization granted {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:24:00.101956 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 947/1922] 10.222.0.27 () {58 vars in 1153 bytes} [Thu Oct 13 04:24:00 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 13 04:24:00.107146 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6adaa213-c5c2-4dce-a841-e1fc12d43fbb None None] Authenticating user token {{(pid=81409) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:24:00.116709 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6adaa213-c5c2-4dce-a841-e1fc12d43fbb None None] Validating token access rules against request {{(pid=81409) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:24:00.121367 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b11ee3b8-4e81-46cb-9648-10fb9791fcae admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8ba823f313144da3b683bacef8fd62e0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81409) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:24:00.121962 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b11ee3b8-4e81-46cb-9648-10fb9791fcae admin admin] REQUEST_METHOD: `POST` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:24:00.122272 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b11ee3b8-4e81-46cb-9648-10fb9791fcae admin admin] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:24:00.122549 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b11ee3b8-4e81-46cb-9648-10fb9791fcae admin admin] PATH_INFO: `/v3/projects` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:24:00.123175 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b11ee3b8-4e81-46cb-9648-10fb9791fcae admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:24:00.125127 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b11ee3b8-4e81-46cb-9648-10fb9791fcae admin admin] RBAC: Authorization granted {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:24:00.151623 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 976/1923] 10.222.0.27 () {60 vars in 1153 bytes} [Thu Oct 13 04:24:00 2022] POST /identity/v3/projects => generated 399 bytes in 45 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 13 04:24:00.155653 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0b28208b-5c66-436f-8a5e-90f8b7f3f345 admin admin] Authenticating user token {{(pid=81408) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:24:00.165686 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0b28208b-5c66-436f-8a5e-90f8b7f3f345 admin admin] Validating token access rules against request {{(pid=81408) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:24:00.221147 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ebc2fcba-07af-463b-8e78-7eddc0251b2b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8ba823f313144da3b683bacef8fd62e0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81408) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:24:00.221660 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ebc2fcba-07af-463b-8e78-7eddc0251b2b admin admin] REQUEST_METHOD: `POST` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:24:00.221935 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ebc2fcba-07af-463b-8e78-7eddc0251b2b admin admin] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:24:00.222226 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ebc2fcba-07af-463b-8e78-7eddc0251b2b admin admin] PATH_INFO: `/v3/users` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:24:00.222862 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ebc2fcba-07af-463b-8e78-7eddc0251b2b admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:24:00.224767 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ebc2fcba-07af-463b-8e78-7eddc0251b2b admin admin] RBAC: Authorization granted {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:24:00.245341 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 948/1924] 10.222.0.27 () {60 vars in 1144 bytes} [Thu Oct 13 04:24:00 2022] POST /identity/v3/users => generated 300 bytes in 90 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 13 04:24:00.249403 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b11ee3b8-4e81-46cb-9648-10fb9791fcae admin admin] Authenticating user token {{(pid=81409) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:24:00.260110 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b11ee3b8-4e81-46cb-9648-10fb9791fcae admin admin] Validating token access rules against request {{(pid=81409) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:24:00.264636 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6f6bbd25-1213-4bb0-a7d8-908f96a5ddd2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8ba823f313144da3b683bacef8fd62e0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81409) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:24:00.265953 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f6bbd25-1213-4bb0-a7d8-908f96a5ddd2 admin admin] REQUEST_METHOD: `GET` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:24:00.266345 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f6bbd25-1213-4bb0-a7d8-908f96a5ddd2 admin admin] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:24:00.266535 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f6bbd25-1213-4bb0-a7d8-908f96a5ddd2 admin admin] PATH_INFO: `/v3/roles` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:24:00.267165 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6f6bbd25-1213-4bb0-a7d8-908f96a5ddd2 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:24:00.268753 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6f6bbd25-1213-4bb0-a7d8-908f96a5ddd2 admin admin] RBAC: Authorization granted {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:24:00.278521 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 977/1925] 10.222.0.27 () {58 vars in 1122 bytes} [Thu Oct 13 04:24:00 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 13 04:24:00.282591 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ebc2fcba-07af-463b-8e78-7eddc0251b2b admin admin] Authenticating user token {{(pid=81408) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:24:00.292349 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ebc2fcba-07af-463b-8e78-7eddc0251b2b admin admin] Validating token access rules against request {{(pid=81408) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:24:00.296963 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eb804eac-9b4b-4c8c-8752-2f16942940b5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8ba823f313144da3b683bacef8fd62e0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81408) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:24:00.297545 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb804eac-9b4b-4c8c-8752-2f16942940b5 admin admin] REQUEST_METHOD: `GET` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:24:00.297823 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb804eac-9b4b-4c8c-8752-2f16942940b5 admin admin] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:24:00.298100 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb804eac-9b4b-4c8c-8752-2f16942940b5 admin admin] PATH_INFO: `/v3/roles` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:24:00.298761 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eb804eac-9b4b-4c8c-8752-2f16942940b5 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:24:00.300379 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eb804eac-9b4b-4c8c-8752-2f16942940b5 admin admin] RBAC: Authorization granted {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:24:00.310874 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 949/1926] 10.222.0.27 () {58 vars in 1122 bytes} [Thu Oct 13 04:24:00 2022] GET /identity/v3/roles => generated 1420 bytes in 29 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 13 04:24:00.315421 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6f6bbd25-1213-4bb0-a7d8-908f96a5ddd2 admin admin] Authenticating user token {{(pid=81409) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:24:00.325338 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6f6bbd25-1213-4bb0-a7d8-908f96a5ddd2 admin admin] Validating token access rules against request {{(pid=81409) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:24:00.330954 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e923f354-622c-4726-9a06-51f9c70174b9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8ba823f313144da3b683bacef8fd62e0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81409) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:24:00.331829 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e923f354-622c-4726-9a06-51f9c70174b9 admin admin] REQUEST_METHOD: `PUT` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:24:00.332198 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e923f354-622c-4726-9a06-51f9c70174b9 admin admin] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:24:00.332542 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e923f354-622c-4726-9a06-51f9c70174b9 admin admin] PATH_INFO: `/v3/projects/bc0632ce0f1648bb81b10c1c8147c5e3/users/02da85c0dde349ce9ee51943f64f76cf/roles/cc56c4ae13a7424ea771c15da4b153c1` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:24:00.348685 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e923f354-622c-4726-9a06-51f9c70174b9 admin admin] RBAC: Authorizing `identity:create_grant(project_id=bc0632ce0f1648bb81b10c1c8147c5e3, user_id=02da85c0dde349ce9ee51943f64f76cf, role_id=cc56c4ae13a7424ea771c15da4b153c1)` {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:24:00.351670 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e923f354-622c-4726-9a06-51f9c70174b9 admin admin] RBAC: Authorization granted {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:24:00.378235 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 978/1927] 10.222.0.27 () {60 vars in 1483 bytes} [Thu Oct 13 04:24:00 2022] PUT /identity/v3/projects/bc0632ce0f1648bb81b10c1c8147c5e3/users/02da85c0dde349ce9ee51943f64f76cf/roles/cc56c4ae13a7424ea771c15da4b153c1 => generated 0 bytes in 54 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 13 04:24:00.383179 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca2c2ba4-2ea0-4040-a4d0-af51b49740ff None None] REQUEST_METHOD: `POST` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:24:00.383504 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca2c2ba4-2ea0-4040-a4d0-af51b49740ff None None] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:24:00.383766 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca2c2ba4-2ea0-4040-a4d0-af51b49740ff None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:24:00.423608 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.auth.core [None req-ca2c2ba4-2ea0-4040-a4d0-af51b49740ff None None] MFA Rules not processed for user `02da85c0dde349ce9ee51943f64f76cf`. Rule list: `[]` (Enabled: `True`). {{(pid=81408) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 13 04:24:00.424847 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.fernet_utils [None req-ca2c2ba4-2ea0-4040-a4d0-af51b49740ff None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81408) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:24:00.460413 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG dbcounter [-] [81408] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81408) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:24:00.504338 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d92c328-d1cb-46e0-afd6-0c39389e9095 None None] REQUEST_METHOD: `POST` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:24:00.504620 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d92c328-d1cb-46e0-afd6-0c39389e9095 None None] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:24:00.504893 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d92c328-d1cb-46e0-afd6-0c39389e9095 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:24:00.524529 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.auth.core [None req-2d92c328-d1cb-46e0-afd6-0c39389e9095 None None] MFA Rules not processed for user `487b1605fd4143cfaa422a3a7a0344c7`. Rule list: `[]` (Enabled: `True`). {{(pid=81409) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 13 04:24:00.525749 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.fernet_utils [None req-2d92c328-d1cb-46e0-afd6-0c39389e9095 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81409) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:24:00.529441 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 950/1928] 10.222.0.27 () {58 vars in 958 bytes} [Thu Oct 13 04:24:00 2022] POST /identity/v3/auth/tokens => generated 3276 bytes in 148 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 13 04:24:00.541593 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ca2c2ba4-2ea0-4040-a4d0-af51b49740ff None None] Authenticating user token {{(pid=81408) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:24:00.542976 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.fernet_utils [None req-ca2c2ba4-2ea0-4040-a4d0-af51b49740ff None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81408) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:24:00.567155 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG dbcounter [-] [81409] Writing DB stats keystone:SELECT=90,keystone:DELETE=7,keystone:INSERT=3 {{(pid=81409) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:24:00.586331 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 979/1929] 10.222.0.27 () {58 vars in 967 bytes} [Thu Oct 13 04:24:00 2022] POST /identity/v3/auth/tokens => generated 3195 bytes in 83 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 13 04:24:00.599600 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2d92c328-d1cb-46e0-afd6-0c39389e9095 None None] Authenticating user token {{(pid=81409) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:24:00.600595 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ca2c2ba4-2ea0-4040-a4d0-af51b49740ff None None] Validating token access rules against request {{(pid=81408) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:24:00.600918 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.fernet_utils [None req-2d92c328-d1cb-46e0-afd6-0c39389e9095 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81409) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:24:00.604331 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dbd5ecf9-000e-4263-b21c-2e7407991314 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6141b8a6c3d54e179d6879bb019f4d12', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dcec0431fb6849499d6aec35a234daa6', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81408) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:24:00.604942 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dbd5ecf9-000e-4263-b21c-2e7407991314 service cinder] REQUEST_METHOD: `GET` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:24:00.605221 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dbd5ecf9-000e-4263-b21c-2e7407991314 service cinder] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:24:00.605662 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dbd5ecf9-000e-4263-b21c-2e7407991314 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:24:00.614873 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dbd5ecf9-000e-4263-b21c-2e7407991314 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:24:00.616302 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dbd5ecf9-000e-4263-b21c-2e7407991314 service cinder] RBAC: Authorization granted {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:24:00.625361 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 951/1930] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Oct 13 04:24:00 2022] GET /identity/v3/auth/tokens => generated 3276 bytes in 84 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Oct 13 04:24:00.674422 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2d92c328-d1cb-46e0-afd6-0c39389e9095 None None] Validating token access rules against request {{(pid=81409) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:24:00.679465 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cf07ed73-953b-404a-9630-8261f875ea1b service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90a0f466b664458898cf9dc4112ae651', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dcec0431fb6849499d6aec35a234daa6', 'project_domain_id': 'default', 'roles': ['service', 'member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81409) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:24:00.680206 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf07ed73-953b-404a-9630-8261f875ea1b service swift] REQUEST_METHOD: `GET` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:24:00.680490 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf07ed73-953b-404a-9630-8261f875ea1b service swift] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:24:00.680765 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf07ed73-953b-404a-9630-8261f875ea1b service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:24:00.689399 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cf07ed73-953b-404a-9630-8261f875ea1b service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:24:00.691038 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cf07ed73-953b-404a-9630-8261f875ea1b service swift] RBAC: Authorization granted {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:24:00.699090 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 980/1931] 10.222.0.27 () {60 vars in 1384 bytes} [Thu Oct 13 04:24:00 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 100 msecs (HTTP/1.1 200) 6 headers in 379 bytes (2 switches on core 0) Oct 13 04:24:03.208975 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dbd5ecf9-000e-4263-b21c-2e7407991314 service cinder] Authenticating user token {{(pid=81408) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:24:03.214925 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.fernet_utils [None req-dbd5ecf9-000e-4263-b21c-2e7407991314 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81408) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:24:03.268750 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb63bbf9-59e3-43d0-9d58-b107642c0a7e None None] REQUEST_METHOD: `GET` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:24:03.269527 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb63bbf9-59e3-43d0-9d58-b107642c0a7e None None] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:24:03.269812 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb63bbf9-59e3-43d0-9d58-b107642c0a7e None None] PATH_INFO: `/` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:24:03.274954 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 981/1932] 10.222.0.27 () {54 vars in 921 bytes} [Thu Oct 13 04:24:03 2022] GET /identity => generated 270 bytes in 8 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Oct 13 04:24:03.281589 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69e31a09-fbe8-424f-876c-ba9ecb3ca6ac None None] REQUEST_METHOD: `POST` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:24:03.281944 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69e31a09-fbe8-424f-876c-ba9ecb3ca6ac None None] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:24:03.282515 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69e31a09-fbe8-424f-876c-ba9ecb3ca6ac None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:24:03.298523 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dbd5ecf9-000e-4263-b21c-2e7407991314 service cinder] Validating token access rules against request {{(pid=81408) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:24:03.303396 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fe4c3d3c-1930-4228-9e9b-839cf15d3566 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8ba823f313144da3b683bacef8fd62e0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81408) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:24:03.304120 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe4c3d3c-1930-4228-9e9b-839cf15d3566 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:24:03.304467 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe4c3d3c-1930-4228-9e9b-839cf15d3566 admin admin] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:24:03.304807 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe4c3d3c-1930-4228-9e9b-839cf15d3566 admin admin] PATH_INFO: `/v3/users/9ee6eac2e6da4d83bd14324669dc7269` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:24:03.306106 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fe4c3d3c-1930-4228-9e9b-839cf15d3566 admin admin] RBAC: Authorizing `identity:delete_user(user_id=9ee6eac2e6da4d83bd14324669dc7269)` {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:24:03.307991 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.auth.core [None req-69e31a09-fbe8-424f-876c-ba9ecb3ca6ac None None] MFA Rules not processed for user `cb740ed7f7684b079e45f887be83f795`. Rule list: `[]` (Enabled: `True`). {{(pid=81409) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 13 04:24:03.308201 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fe4c3d3c-1930-4228-9e9b-839cf15d3566 admin admin] RBAC: Authorization granted {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:24:03.309438 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.fernet_utils [None req-69e31a09-fbe8-424f-876c-ba9ecb3ca6ac None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81409) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:24:03.372253 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.notifications [None req-fe4c3d3c-1930-4228-9e9b-839cf15d3566 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '9ee6eac2e6da4d83bd14324669dc7269'} {{(pid=81408) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 13 04:24:03.386587 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.notifications [None req-fe4c3d3c-1930-4228-9e9b-839cf15d3566 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '9ee6eac2e6da4d83bd14324669dc7269'} {{(pid=81408) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 13 04:24:03.387429 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.notifications [None req-fe4c3d3c-1930-4228-9e9b-839cf15d3566 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '9ee6eac2e6da4d83bd14324669dc7269'} {{(pid=81408) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 13 04:24:03.388091 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.notifications [None req-fe4c3d3c-1930-4228-9e9b-839cf15d3566 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '9ee6eac2e6da4d83bd14324669dc7269'} {{(pid=81408) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 13 04:24:03.406593 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.notifications [None req-fe4c3d3c-1930-4228-9e9b-839cf15d3566 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '9ee6eac2e6da4d83bd14324669dc7269'} {{(pid=81408) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 13 04:24:03.416052 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 982/1933] 10.222.0.27 () {58 vars in 1018 bytes} [Thu Oct 13 04:24:03 2022] POST /identity/v3/auth/tokens => generated 3372 bytes in 136 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 13 04:24:03.419681 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG dbcounter [-] [81408] Writing DB stats keystone:SELECT=90,keystone:DELETE=9,keystone:INSERT=1 {{(pid=81408) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:24:03.431349 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-69e31a09-fbe8-424f-876c-ba9ecb3ca6ac None None] Authenticating user token {{(pid=81409) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:24:03.433098 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.fernet_utils [None req-69e31a09-fbe8-424f-876c-ba9ecb3ca6ac None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81409) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:24:03.455356 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 952/1934] 10.222.0.27 () {58 vars in 1224 bytes} [Thu Oct 13 04:24:03 2022] DELETE /identity/v3/users/9ee6eac2e6da4d83bd14324669dc7269 => generated 0 bytes in 244 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 13 04:24:03.506168 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-69e31a09-fbe8-424f-876c-ba9ecb3ca6ac None None] Validating token access rules against request {{(pid=81409) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:24:03.528116 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG dbcounter [-] [81409] Writing DB stats keystone:SELECT=100 {{(pid=81409) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:24:03.558702 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d53db7c1-6f23-4593-b313-abe3dcc108a5 service glance] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a6700f49b86543f6a5b92cad8faaba42', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dcec0431fb6849499d6aec35a234daa6', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81409) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:24:03.559311 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d53db7c1-6f23-4593-b313-abe3dcc108a5 service glance] REQUEST_METHOD: `GET` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:24:03.559586 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d53db7c1-6f23-4593-b313-abe3dcc108a5 service glance] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:24:03.559850 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d53db7c1-6f23-4593-b313-abe3dcc108a5 service glance] PATH_INFO: `/v3/auth/tokens` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:24:03.631422 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d53db7c1-6f23-4593-b313-abe3dcc108a5 service glance] RBAC: Authorizing `identity:validate_token()` {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:24:03.632894 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d53db7c1-6f23-4593-b313-abe3dcc108a5 service glance] RBAC: Authorization granted {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:24:03.644581 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 983/1935] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Oct 13 04:24:03 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 213 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 13 04:24:03.659449 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fe4c3d3c-1930-4228-9e9b-839cf15d3566 admin admin] Authenticating user token {{(pid=81408) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:24:03.660909 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.fernet_utils [None req-fe4c3d3c-1930-4228-9e9b-839cf15d3566 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81408) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:24:03.719785 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-740da2c1-de18-442b-a3ec-8486aa75e02a None None] REQUEST_METHOD: `GET` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:24:03.720065 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-740da2c1-de18-442b-a3ec-8486aa75e02a None None] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:24:03.720339 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-740da2c1-de18-442b-a3ec-8486aa75e02a None None] PATH_INFO: `/` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:24:03.721705 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 984/1936] 10.222.0.27 () {54 vars in 921 bytes} [Thu Oct 13 04:24:03 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Oct 13 04:24:03.726238 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-893ce103-8034-4f18-b595-303a60aea8a6 None None] REQUEST_METHOD: `POST` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:24:03.726520 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-893ce103-8034-4f18-b595-303a60aea8a6 None None] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:24:03.726976 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-893ce103-8034-4f18-b595-303a60aea8a6 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:24:03.747203 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fe4c3d3c-1930-4228-9e9b-839cf15d3566 admin admin] Validating token access rules against request {{(pid=81408) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:24:03.748617 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.auth.core [None req-893ce103-8034-4f18-b595-303a60aea8a6 None None] MFA Rules not processed for user `d8812c3974f84871bb2d761cf0016257`. Rule list: `[]` (Enabled: `True`). {{(pid=81409) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 13 04:24:03.750016 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.fernet_utils [None req-893ce103-8034-4f18-b595-303a60aea8a6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81409) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:24:03.751977 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a6c5aecf-a180-4c54-accd-e8aa736e4385 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8ba823f313144da3b683bacef8fd62e0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81408) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:24:03.752633 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6c5aecf-a180-4c54-accd-e8aa736e4385 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:24:03.752911 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6c5aecf-a180-4c54-accd-e8aa736e4385 admin admin] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:24:03.753181 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6c5aecf-a180-4c54-accd-e8aa736e4385 admin admin] PATH_INFO: `/v3/projects/e7d1e420eec84d1f8b78333b7d814f3c` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:24:03.754382 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a6c5aecf-a180-4c54-accd-e8aa736e4385 admin admin] RBAC: Authorizing `identity:delete_project(project_id=e7d1e420eec84d1f8b78333b7d814f3c)` {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:24:03.756311 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a6c5aecf-a180-4c54-accd-e8aa736e4385 admin admin] RBAC: Authorization granted {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:24:03.803337 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 985/1937] 10.222.0.27 () {58 vars in 1018 bytes} [Thu Oct 13 04:24:03 2022] POST /identity/v3/auth/tokens => generated 3121 bytes in 78 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 13 04:24:03.804984 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.notifications [None req-a6c5aecf-a180-4c54-accd-e8aa736e4385 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'e7d1e420eec84d1f8b78333b7d814f3c', 'request_id': 'req-a6c5aecf-a180-4c54-accd-e8aa736e4385'} {{(pid=81408) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 13 04:24:03.813185 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.notifications [None req-a6c5aecf-a180-4c54-accd-e8aa736e4385 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'e7d1e420eec84d1f8b78333b7d814f3c', 'request_id': 'req-a6c5aecf-a180-4c54-accd-e8aa736e4385'} {{(pid=81408) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 13 04:24:03.816732 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-893ce103-8034-4f18-b595-303a60aea8a6 None None] Authenticating user token {{(pid=81409) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:24:03.818087 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.fernet_utils [None req-893ce103-8034-4f18-b595-303a60aea8a6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81409) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:24:03.819861 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.notifications [None req-a6c5aecf-a180-4c54-accd-e8aa736e4385 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'e7d1e420eec84d1f8b78333b7d814f3c', 'request_id': 'req-a6c5aecf-a180-4c54-accd-e8aa736e4385'} {{(pid=81408) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 13 04:24:03.829732 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-893ce103-8034-4f18-b595-303a60aea8a6 None None] Validating token access rules against request {{(pid=81409) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:24:03.832564 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.notifications [None req-a6c5aecf-a180-4c54-accd-e8aa736e4385 admin admin] The token cache is being invalidate because project e7d1e420eec84d1f8b78333b7d814f3c was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81408) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 13 04:24:03.832836 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.notifications [None req-a6c5aecf-a180-4c54-accd-e8aa736e4385 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81408) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 13 04:24:03.833616 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.notifications [None req-a6c5aecf-a180-4c54-accd-e8aa736e4385 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81408) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 13 04:24:03.835290 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 953/1938] 10.222.0.27 () {58 vars in 1233 bytes} [Thu Oct 13 04:24:03 2022] DELETE /identity/v3/projects/e7d1e420eec84d1f8b78333b7d814f3c => generated 0 bytes in 176 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 13 04:24:03.840373 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22a11917-d18f-4900-abb2-8671a4ae10ec None None] REQUEST_METHOD: `POST` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:24:03.840652 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22a11917-d18f-4900-abb2-8671a4ae10ec None None] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:24:03.840921 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22a11917-d18f-4900-abb2-8671a4ae10ec None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:24:03.861029 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.auth.core [None req-22a11917-d18f-4900-abb2-8671a4ae10ec None None] MFA Rules not processed for user `0e2d979563d6421995451a0e86bf3022`. Rule list: `[]` (Enabled: `True`). {{(pid=81408) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 13 04:24:03.862254 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.fernet_utils [None req-22a11917-d18f-4900-abb2-8671a4ae10ec None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81408) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:24:03.880788 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-250badf6-a078-418d-a177-43dbcbb7e063 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd8812c3974f84871bb2d761cf0016257', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dcec0431fb6849499d6aec35a234daa6', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81409) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:24:03.881400 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-250badf6-a078-418d-a177-43dbcbb7e063 service neutron] REQUEST_METHOD: `GET` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:24:03.882073 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-250badf6-a078-418d-a177-43dbcbb7e063 service neutron] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:24:03.882387 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-250badf6-a078-418d-a177-43dbcbb7e063 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:24:03.884161 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.fernet_utils [None req-250badf6-a078-418d-a177-43dbcbb7e063 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81409) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:24:03.893737 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-250badf6-a078-418d-a177-43dbcbb7e063 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:24:03.895345 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-250badf6-a078-418d-a177-43dbcbb7e063 service neutron] RBAC: Authorization granted {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:24:03.903535 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 986/1939] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Oct 13 04:24:03 2022] GET /identity/v3/auth/tokens => generated 3121 bytes in 88 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Oct 13 04:24:03.920888 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 954/1940] 10.222.0.27 () {58 vars in 958 bytes} [Thu Oct 13 04:24:03 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 81 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 13 04:24:03.929082 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d28944b4-6a33-4a63-8b87-5673ad340470 None None] REQUEST_METHOD: `POST` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:24:03.929367 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d28944b4-6a33-4a63-8b87-5673ad340470 None None] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:24:03.929638 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d28944b4-6a33-4a63-8b87-5673ad340470 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:24:03.950623 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.auth.core [None req-d28944b4-6a33-4a63-8b87-5673ad340470 None None] MFA Rules not processed for user `0e2d979563d6421995451a0e86bf3022`. Rule list: `[]` (Enabled: `True`). {{(pid=81409) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 13 04:24:03.951841 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.fernet_utils [None req-d28944b4-6a33-4a63-8b87-5673ad340470 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81409) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:24:03.960575 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 987/1941] 10.222.0.27 () {58 vars in 958 bytes} [Thu Oct 13 04:24:03 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 32 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 13 04:24:03.964757 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-22a11917-d18f-4900-abb2-8671a4ae10ec None None] Authenticating user token {{(pid=81408) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:24:03.974834 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-22a11917-d18f-4900-abb2-8671a4ae10ec None None] Validating token access rules against request {{(pid=81408) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:24:03.979449 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a0f2294d-0ebc-43b4-b3e1-ec0b3e36218f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8ba823f313144da3b683bacef8fd62e0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81408) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:24:03.980043 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0f2294d-0ebc-43b4-b3e1-ec0b3e36218f admin admin] REQUEST_METHOD: `GET` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:24:03.980317 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0f2294d-0ebc-43b4-b3e1-ec0b3e36218f admin admin] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:24:03.980583 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0f2294d-0ebc-43b4-b3e1-ec0b3e36218f admin admin] PATH_INFO: `/v3/domains` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:24:03.981512 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a0f2294d-0ebc-43b4-b3e1-ec0b3e36218f admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81408) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 13 04:24:03.981807 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a0f2294d-0ebc-43b4-b3e1-ec0b3e36218f admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:24:03.983474 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a0f2294d-0ebc-43b4-b3e1-ec0b3e36218f admin admin] RBAC: Authorization granted {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:24:03.989159 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6312a523-08ae-42eb-94e2-a5f7fdacae7d None None] REQUEST_METHOD: `GET` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:24:03.989440 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6312a523-08ae-42eb-94e2-a5f7fdacae7d None None] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:24:03.989711 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6312a523-08ae-42eb-94e2-a5f7fdacae7d None None] PATH_INFO: `/` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:24:03.991613 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 988/1942] 10.222.0.27 () {54 vars in 921 bytes} [Thu Oct 13 04:24:03 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Oct 13 04:24:03.995613 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 955/1943] 10.222.0.27 () {58 vars in 1153 bytes} [Thu Oct 13 04:24:03 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 13 04:24:03.996817 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b831ffb-4bc8-451f-bf49-1b96663e1f63 None None] REQUEST_METHOD: `POST` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:24:03.997092 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b831ffb-4bc8-451f-bf49-1b96663e1f63 None None] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:24:03.997360 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b831ffb-4bc8-451f-bf49-1b96663e1f63 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:24:04.000315 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a0f2294d-0ebc-43b4-b3e1-ec0b3e36218f admin admin] Authenticating user token {{(pid=81408) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:24:04.010405 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a0f2294d-0ebc-43b4-b3e1-ec0b3e36218f admin admin] Validating token access rules against request {{(pid=81408) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:24:04.015299 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4995c7b5-7a46-47f2-a72f-2c3a3e9021fa admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8ba823f313144da3b683bacef8fd62e0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81408) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:24:04.016030 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4995c7b5-7a46-47f2-a72f-2c3a3e9021fa admin admin] REQUEST_METHOD: `POST` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:24:04.016309 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4995c7b5-7a46-47f2-a72f-2c3a3e9021fa admin admin] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:24:04.016582 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4995c7b5-7a46-47f2-a72f-2c3a3e9021fa admin admin] PATH_INFO: `/v3/projects` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:24:04.017210 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4995c7b5-7a46-47f2-a72f-2c3a3e9021fa admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:24:04.019155 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4995c7b5-7a46-47f2-a72f-2c3a3e9021fa admin admin] RBAC: Authorization granted {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:24:04.019337 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.auth.core [None req-4b831ffb-4bc8-451f-bf49-1b96663e1f63 None None] MFA Rules not processed for user `cb740ed7f7684b079e45f887be83f795`. Rule list: `[]` (Enabled: `True`). {{(pid=81409) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 13 04:24:04.020533 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.fernet_utils [None req-4b831ffb-4bc8-451f-bf49-1b96663e1f63 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81409) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:24:04.024798 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG dbcounter [-] [81408] Writing DB stats keystone:SELECT=91,keystone:DELETE=8,keystone:INSERT=1 {{(pid=81408) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:24:04.026944 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG dbcounter [-] [81409] Writing DB stats keystone:SELECT=100 {{(pid=81409) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:24:04.054664 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 956/1944] 10.222.0.27 () {60 vars in 1153 bytes} [Thu Oct 13 04:24:03 2022] POST /identity/v3/projects => generated 385 bytes in 52 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 13 04:24:04.055747 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4995c7b5-7a46-47f2-a72f-2c3a3e9021fa admin admin] Authenticating user token {{(pid=81408) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:24:04.103867 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 989/1945] 10.222.0.27 () {58 vars in 1018 bytes} [Thu Oct 13 04:24:03 2022] POST /identity/v3/auth/tokens => generated 3372 bytes in 107 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 13 04:24:04.117121 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4b831ffb-4bc8-451f-bf49-1b96663e1f63 None None] Authenticating user token {{(pid=81409) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:24:04.118548 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.fernet_utils [None req-4b831ffb-4bc8-451f-bf49-1b96663e1f63 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81409) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:24:04.127640 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4995c7b5-7a46-47f2-a72f-2c3a3e9021fa admin admin] Validating token access rules against request {{(pid=81408) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:24:04.130584 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4b831ffb-4bc8-451f-bf49-1b96663e1f63 None None] Validating token access rules against request {{(pid=81409) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:24:04.132726 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-392b20f2-4b05-4fb1-866a-c1dcd6719246 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8ba823f313144da3b683bacef8fd62e0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81408) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:24:04.133354 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-392b20f2-4b05-4fb1-866a-c1dcd6719246 admin admin] REQUEST_METHOD: `POST` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:24:04.133632 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-392b20f2-4b05-4fb1-866a-c1dcd6719246 admin admin] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:24:04.133904 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-392b20f2-4b05-4fb1-866a-c1dcd6719246 admin admin] PATH_INFO: `/v3/users` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:24:04.134566 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-392b20f2-4b05-4fb1-866a-c1dcd6719246 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:24:04.136527 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-392b20f2-4b05-4fb1-866a-c1dcd6719246 admin admin] RBAC: Authorization granted {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:24:04.161216 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 957/1946] 10.222.0.27 () {60 vars in 1144 bytes} [Thu Oct 13 04:24:04 2022] POST /identity/v3/users => generated 293 bytes in 106 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 13 04:24:04.165064 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-392b20f2-4b05-4fb1-866a-c1dcd6719246 admin admin] Authenticating user token {{(pid=81408) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:24:04.175525 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-392b20f2-4b05-4fb1-866a-c1dcd6719246 admin admin] Validating token access rules against request {{(pid=81408) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:24:04.181979 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-894b798d-dc70-4618-9b24-c05975e550c7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8ba823f313144da3b683bacef8fd62e0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81408) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:24:04.182610 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-894b798d-dc70-4618-9b24-c05975e550c7 admin admin] REQUEST_METHOD: `GET` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:24:04.182888 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-894b798d-dc70-4618-9b24-c05975e550c7 admin admin] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:24:04.183158 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-894b798d-dc70-4618-9b24-c05975e550c7 admin admin] PATH_INFO: `/v3/roles` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:24:04.183793 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-894b798d-dc70-4618-9b24-c05975e550c7 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:24:04.184077 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-96b159aa-8394-48c2-be53-98767d2e253e service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd8812c3974f84871bb2d761cf0016257', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dcec0431fb6849499d6aec35a234daa6', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81409) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:24:04.184709 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96b159aa-8394-48c2-be53-98767d2e253e service neutron] REQUEST_METHOD: `GET` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:24:04.184962 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96b159aa-8394-48c2-be53-98767d2e253e service neutron] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:24:04.185231 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96b159aa-8394-48c2-be53-98767d2e253e service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:24:04.185389 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-894b798d-dc70-4618-9b24-c05975e550c7 admin admin] RBAC: Authorization granted {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:24:04.198866 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 958/1947] 10.222.0.27 () {58 vars in 1122 bytes} [Thu Oct 13 04:24:04 2022] GET /identity/v3/roles => generated 1420 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Oct 13 04:24:04.199654 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-894b798d-dc70-4618-9b24-c05975e550c7 admin admin] Authenticating user token {{(pid=81408) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:24:04.210367 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-894b798d-dc70-4618-9b24-c05975e550c7 admin admin] Validating token access rules against request {{(pid=81408) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:24:04.215313 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3e04a8b7-46b7-42e7-8e47-b8de629608a1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8ba823f313144da3b683bacef8fd62e0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81408) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:24:04.216016 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e04a8b7-46b7-42e7-8e47-b8de629608a1 admin admin] REQUEST_METHOD: `GET` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:24:04.216302 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e04a8b7-46b7-42e7-8e47-b8de629608a1 admin admin] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:24:04.216575 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e04a8b7-46b7-42e7-8e47-b8de629608a1 admin admin] PATH_INFO: `/v3/roles` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:24:04.217356 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3e04a8b7-46b7-42e7-8e47-b8de629608a1 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:24:04.220019 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3e04a8b7-46b7-42e7-8e47-b8de629608a1 admin admin] RBAC: Authorization granted {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:24:04.235369 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 959/1948] 10.222.0.27 () {58 vars in 1122 bytes} [Thu Oct 13 04:24:04 2022] GET /identity/v3/roles => generated 1420 bytes in 33 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Oct 13 04:24:04.238086 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3e04a8b7-46b7-42e7-8e47-b8de629608a1 admin admin] Authenticating user token {{(pid=81408) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:24:04.249245 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3e04a8b7-46b7-42e7-8e47-b8de629608a1 admin admin] Validating token access rules against request {{(pid=81408) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:24:04.254982 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dd9a3500-d466-46c3-8fb7-c229f18e2dd9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8ba823f313144da3b683bacef8fd62e0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81408) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:24:04.255634 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd9a3500-d466-46c3-8fb7-c229f18e2dd9 admin admin] REQUEST_METHOD: `PUT` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:24:04.255907 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd9a3500-d466-46c3-8fb7-c229f18e2dd9 admin admin] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:24:04.256172 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd9a3500-d466-46c3-8fb7-c229f18e2dd9 admin admin] PATH_INFO: `/v3/projects/f2b13e92741f40bc8044ed4bc999b07f/users/7582f5c0de6b4b5dae18b0320921e4a2/roles/cc56c4ae13a7424ea771c15da4b153c1` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:24:04.268350 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-96b159aa-8394-48c2-be53-98767d2e253e service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:24:04.270883 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-96b159aa-8394-48c2-be53-98767d2e253e service neutron] RBAC: Authorization granted {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:24:04.275044 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dd9a3500-d466-46c3-8fb7-c229f18e2dd9 admin admin] RBAC: Authorizing `identity:create_grant(project_id=f2b13e92741f40bc8044ed4bc999b07f, user_id=7582f5c0de6b4b5dae18b0320921e4a2, role_id=cc56c4ae13a7424ea771c15da4b153c1)` {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:24:04.277786 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dd9a3500-d466-46c3-8fb7-c229f18e2dd9 admin admin] RBAC: Authorization granted {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:24:04.284799 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 990/1949] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Oct 13 04:24:04 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 168 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Oct 13 04:24:04.314641 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 960/1950] 10.222.0.27 () {60 vars in 1483 bytes} [Thu Oct 13 04:24:04 2022] PUT /identity/v3/projects/f2b13e92741f40bc8044ed4bc999b07f/users/7582f5c0de6b4b5dae18b0320921e4a2/roles/cc56c4ae13a7424ea771c15da4b153c1 => generated 0 bytes in 77 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 13 04:24:04.321314 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-404254b4-d423-4e2a-9fc7-a32e172f089d None None] REQUEST_METHOD: `POST` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:24:04.321602 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-404254b4-d423-4e2a-9fc7-a32e172f089d None None] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:24:04.325119 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-404254b4-d423-4e2a-9fc7-a32e172f089d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:24:04.346294 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.auth.core [None req-404254b4-d423-4e2a-9fc7-a32e172f089d None None] MFA Rules not processed for user `0e2d979563d6421995451a0e86bf3022`. Rule list: `[]` (Enabled: `True`). {{(pid=81409) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 13 04:24:04.347625 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.fernet_utils [None req-404254b4-d423-4e2a-9fc7-a32e172f089d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81409) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:24:04.389752 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG dbcounter [-] [81409] Writing DB stats keystone:SELECT=100 {{(pid=81409) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:24:04.411637 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 991/1951] 10.222.0.27 () {58 vars in 958 bytes} [Thu Oct 13 04:24:04 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 92 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 13 04:24:04.423693 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dd9a3500-d466-46c3-8fb7-c229f18e2dd9 admin admin] Authenticating user token {{(pid=81408) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:24:04.425512 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.fernet_utils [None req-dd9a3500-d466-46c3-8fb7-c229f18e2dd9 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81408) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:24:04.493464 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dd9a3500-d466-46c3-8fb7-c229f18e2dd9 admin admin] Validating token access rules against request {{(pid=81408) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:24:04.499263 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8bfca8b3-1bd8-4370-a1c0-193874eee25a service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd8812c3974f84871bb2d761cf0016257', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dcec0431fb6849499d6aec35a234daa6', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81408) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:24:04.506786 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8bfca8b3-1bd8-4370-a1c0-193874eee25a service neutron] REQUEST_METHOD: `GET` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:24:04.507395 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8bfca8b3-1bd8-4370-a1c0-193874eee25a service neutron] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:24:04.508562 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8bfca8b3-1bd8-4370-a1c0-193874eee25a service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:24:04.522711 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8bfca8b3-1bd8-4370-a1c0-193874eee25a service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:24:04.524558 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8bfca8b3-1bd8-4370-a1c0-193874eee25a service neutron] RBAC: Authorization granted {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:24:04.537695 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 961/1952] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Oct 13 04:24:04 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 114 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 13 04:24:06.235316 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-404254b4-d423-4e2a-9fc7-a32e172f089d None None] Authenticating user token {{(pid=81409) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:24:06.246713 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-404254b4-d423-4e2a-9fc7-a32e172f089d None None] Validating token access rules against request {{(pid=81409) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:24:06.254454 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-50620671-a925-479f-8d5b-b6ce05c32494 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd8812c3974f84871bb2d761cf0016257', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dcec0431fb6849499d6aec35a234daa6', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81409) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:24:06.255433 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50620671-a925-479f-8d5b-b6ce05c32494 service neutron] REQUEST_METHOD: `GET` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:24:06.257464 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50620671-a925-479f-8d5b-b6ce05c32494 service neutron] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:24:06.258178 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50620671-a925-479f-8d5b-b6ce05c32494 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:24:06.260351 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.fernet_utils [None req-50620671-a925-479f-8d5b-b6ce05c32494 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81409) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:24:06.338822 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-50620671-a925-479f-8d5b-b6ce05c32494 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:24:06.340275 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-50620671-a925-479f-8d5b-b6ce05c32494 service neutron] RBAC: Authorization granted {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:24:06.354399 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 992/1953] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Oct 13 04:24:06 2022] GET /identity/v3/auth/tokens => generated 3315 bytes in 120 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 13 04:24:09.041160 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8bfca8b3-1bd8-4370-a1c0-193874eee25a service neutron] Authenticating user token {{(pid=81408) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:24:09.042880 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.fernet_utils [None req-8bfca8b3-1bd8-4370-a1c0-193874eee25a service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81408) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:24:09.078719 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8bfca8b3-1bd8-4370-a1c0-193874eee25a service neutron] Validating token access rules against request {{(pid=81408) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:24:09.093299 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d9ae48a7-188b-4558-9c97-3e7b4f2dc9eb admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8ba823f313144da3b683bacef8fd62e0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81408) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:24:09.093963 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9ae48a7-188b-4558-9c97-3e7b4f2dc9eb admin admin] REQUEST_METHOD: `DELETE` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:24:09.094266 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9ae48a7-188b-4558-9c97-3e7b4f2dc9eb admin admin] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:24:09.094547 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9ae48a7-188b-4558-9c97-3e7b4f2dc9eb admin admin] PATH_INFO: `/v3/users/02da85c0dde349ce9ee51943f64f76cf` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:24:09.099324 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d9ae48a7-188b-4558-9c97-3e7b4f2dc9eb admin admin] RBAC: Authorizing `identity:delete_user(user_id=02da85c0dde349ce9ee51943f64f76cf)` {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:24:09.102320 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d9ae48a7-188b-4558-9c97-3e7b4f2dc9eb admin admin] RBAC: Authorization granted {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:24:09.154293 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG dbcounter [-] [81408] Writing DB stats keystone:SELECT=90,keystone:INSERT=5,keystone:DELETE=5 {{(pid=81408) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:24:09.197868 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.notifications [None req-d9ae48a7-188b-4558-9c97-3e7b4f2dc9eb admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '02da85c0dde349ce9ee51943f64f76cf'} {{(pid=81408) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 13 04:24:09.209647 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.notifications [None req-d9ae48a7-188b-4558-9c97-3e7b4f2dc9eb admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '02da85c0dde349ce9ee51943f64f76cf'} {{(pid=81408) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 13 04:24:09.210444 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.notifications [None req-d9ae48a7-188b-4558-9c97-3e7b4f2dc9eb admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '02da85c0dde349ce9ee51943f64f76cf'} {{(pid=81408) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 13 04:24:09.211116 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.notifications [None req-d9ae48a7-188b-4558-9c97-3e7b4f2dc9eb admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '02da85c0dde349ce9ee51943f64f76cf'} {{(pid=81408) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 13 04:24:09.222608 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.notifications [None req-d9ae48a7-188b-4558-9c97-3e7b4f2dc9eb admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '02da85c0dde349ce9ee51943f64f76cf'} {{(pid=81408) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 13 04:24:09.261434 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 962/1954] 10.222.0.27 () {58 vars in 1224 bytes} [Thu Oct 13 04:24:09 2022] DELETE /identity/v3/users/02da85c0dde349ce9ee51943f64f76cf => generated 0 bytes in 221 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 13 04:24:09.265960 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-962b78c0-39e8-4bf8-a2ee-17397779c799 None None] REQUEST_METHOD: `POST` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:24:09.266846 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-962b78c0-39e8-4bf8-a2ee-17397779c799 None None] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:24:09.267268 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-962b78c0-39e8-4bf8-a2ee-17397779c799 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:24:09.290578 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.auth.core [None req-962b78c0-39e8-4bf8-a2ee-17397779c799 None None] MFA Rules not processed for user `0e2d979563d6421995451a0e86bf3022`. Rule list: `[]` (Enabled: `True`). {{(pid=81409) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 13 04:24:09.292053 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.fernet_utils [None req-962b78c0-39e8-4bf8-a2ee-17397779c799 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81409) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:24:09.362166 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 993/1955] 10.222.0.27 () {58 vars in 958 bytes} [Thu Oct 13 04:24:09 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 97 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 13 04:24:09.376318 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d9ae48a7-188b-4558-9c97-3e7b4f2dc9eb admin admin] Authenticating user token {{(pid=81408) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:24:09.378037 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.fernet_utils [None req-d9ae48a7-188b-4558-9c97-3e7b4f2dc9eb admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81408) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:24:09.490785 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d9ae48a7-188b-4558-9c97-3e7b4f2dc9eb admin admin] Validating token access rules against request {{(pid=81408) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:24:09.496037 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-87c7f07a-e02a-4283-bfb1-54aa56ba789e service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd8812c3974f84871bb2d761cf0016257', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dcec0431fb6849499d6aec35a234daa6', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81408) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:24:09.496727 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87c7f07a-e02a-4283-bfb1-54aa56ba789e service neutron] REQUEST_METHOD: `GET` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:24:09.497004 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87c7f07a-e02a-4283-bfb1-54aa56ba789e service neutron] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:24:09.497282 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87c7f07a-e02a-4283-bfb1-54aa56ba789e service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:24:09.511492 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-87c7f07a-e02a-4283-bfb1-54aa56ba789e service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:24:09.512938 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-87c7f07a-e02a-4283-bfb1-54aa56ba789e service neutron] RBAC: Authorization granted {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:24:09.526691 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 963/1956] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Oct 13 04:24:09 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 151 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 13 04:24:09.736740 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-962b78c0-39e8-4bf8-a2ee-17397779c799 None None] Authenticating user token {{(pid=81409) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:24:09.737953 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.fernet_utils [None req-962b78c0-39e8-4bf8-a2ee-17397779c799 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81409) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:24:09.869849 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-962b78c0-39e8-4bf8-a2ee-17397779c799 None None] Validating token access rules against request {{(pid=81409) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:24:09.875719 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-567e6d5e-5cfe-44cc-ae7a-fc6e50535d0b service placement] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ba1b6e5868a549a99760ebe6068fd6b9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dcec0431fb6849499d6aec35a234daa6', 'project_domain_id': 'default', 'roles': ['service', 'member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81409) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:24:09.876546 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-567e6d5e-5cfe-44cc-ae7a-fc6e50535d0b service placement] REQUEST_METHOD: `GET` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:24:09.876830 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-567e6d5e-5cfe-44cc-ae7a-fc6e50535d0b service placement] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:24:09.877100 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-567e6d5e-5cfe-44cc-ae7a-fc6e50535d0b service placement] PATH_INFO: `/v3/auth/tokens` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:24:09.882523 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.fernet_utils [None req-567e6d5e-5cfe-44cc-ae7a-fc6e50535d0b service placement] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81409) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:24:09.918405 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-567e6d5e-5cfe-44cc-ae7a-fc6e50535d0b service placement] RBAC: Authorizing `identity:validate_token()` {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:24:09.922708 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-567e6d5e-5cfe-44cc-ae7a-fc6e50535d0b service placement] RBAC: Authorization granted {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:24:09.947832 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 994/1957] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Oct 13 04:24:09 2022] GET /identity/v3/auth/tokens => generated 3308 bytes in 212 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 13 04:24:10.351770 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-87c7f07a-e02a-4283-bfb1-54aa56ba789e service neutron] Authenticating user token {{(pid=81408) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:24:10.353332 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.fernet_utils [None req-87c7f07a-e02a-4283-bfb1-54aa56ba789e service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81408) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:24:10.376199 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-87c7f07a-e02a-4283-bfb1-54aa56ba789e service neutron] Validating token access rules against request {{(pid=81408) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:24:10.383875 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e51cc94a-e379-4b14-8521-1cf6fea74809 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8ba823f313144da3b683bacef8fd62e0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81408) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:24:10.385803 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e51cc94a-e379-4b14-8521-1cf6fea74809 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:24:10.386331 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e51cc94a-e379-4b14-8521-1cf6fea74809 admin admin] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:24:10.386831 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e51cc94a-e379-4b14-8521-1cf6fea74809 admin admin] PATH_INFO: `/v3/projects/bc0632ce0f1648bb81b10c1c8147c5e3` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:24:10.388413 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e51cc94a-e379-4b14-8521-1cf6fea74809 admin admin] RBAC: Authorizing `identity:delete_project(project_id=bc0632ce0f1648bb81b10c1c8147c5e3)` {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:24:10.391197 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e51cc94a-e379-4b14-8521-1cf6fea74809 admin admin] RBAC: Authorization granted {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:24:10.431249 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d36909d-f7ed-4805-85b4-bc01fd0d4a9c None None] REQUEST_METHOD: `POST` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:24:10.431728 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d36909d-f7ed-4805-85b4-bc01fd0d4a9c None None] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:24:10.431836 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d36909d-f7ed-4805-85b4-bc01fd0d4a9c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:24:10.476677 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.notifications [None req-e51cc94a-e379-4b14-8521-1cf6fea74809 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'bc0632ce0f1648bb81b10c1c8147c5e3', 'request_id': 'req-e51cc94a-e379-4b14-8521-1cf6fea74809'} {{(pid=81408) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 13 04:24:10.477401 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG dbcounter [-] [81409] Writing DB stats keystone:SELECT=100 {{(pid=81409) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:24:10.503579 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.notifications [None req-e51cc94a-e379-4b14-8521-1cf6fea74809 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'bc0632ce0f1648bb81b10c1c8147c5e3', 'request_id': 'req-e51cc94a-e379-4b14-8521-1cf6fea74809'} {{(pid=81408) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 13 04:24:10.522548 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.notifications [None req-e51cc94a-e379-4b14-8521-1cf6fea74809 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'bc0632ce0f1648bb81b10c1c8147c5e3', 'request_id': 'req-e51cc94a-e379-4b14-8521-1cf6fea74809'} {{(pid=81408) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 13 04:24:10.529986 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.auth.core [None req-0d36909d-f7ed-4805-85b4-bc01fd0d4a9c None None] MFA Rules not processed for user `7582f5c0de6b4b5dae18b0320921e4a2`. Rule list: `[]` (Enabled: `True`). {{(pid=81409) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 13 04:24:10.531690 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.fernet_utils [None req-0d36909d-f7ed-4805-85b4-bc01fd0d4a9c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81409) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:24:10.552974 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.notifications [None req-e51cc94a-e379-4b14-8521-1cf6fea74809 admin admin] The token cache is being invalidate because project bc0632ce0f1648bb81b10c1c8147c5e3 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81408) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 13 04:24:10.554318 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.notifications [None req-e51cc94a-e379-4b14-8521-1cf6fea74809 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81408) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 13 04:24:10.557198 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.notifications [None req-e51cc94a-e379-4b14-8521-1cf6fea74809 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81408) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 13 04:24:10.561894 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 964/1958] 10.222.0.27 () {58 vars in 1233 bytes} [Thu Oct 13 04:24:10 2022] DELETE /identity/v3/projects/bc0632ce0f1648bb81b10c1c8147c5e3 => generated 0 bytes in 210 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 13 04:24:10.698957 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 995/1959] 10.222.0.27 () {58 vars in 958 bytes} [Thu Oct 13 04:24:10 2022] POST /identity/v3/auth/tokens => generated 3262 bytes in 269 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 13 04:24:10.703777 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e51cc94a-e379-4b14-8521-1cf6fea74809 admin admin] Authenticating user token {{(pid=81408) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:24:10.710388 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.fernet_utils [None req-e51cc94a-e379-4b14-8521-1cf6fea74809 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81408) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:24:10.833097 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG dbcounter [-] [81408] Writing DB stats keystone:SELECT=86,keystone:DELETE=12,keystone:INSERT=2 {{(pid=81408) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:24:10.892210 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e51cc94a-e379-4b14-8521-1cf6fea74809 admin admin] Validating token access rules against request {{(pid=81408) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:24:10.899142 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8dcf618e-11d2-4248-94c8-8cb8cf24d3ac admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8ba823f313144da3b683bacef8fd62e0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81408) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:24:10.899789 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8dcf618e-11d2-4248-94c8-8cb8cf24d3ac admin admin] REQUEST_METHOD: `POST` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:24:10.900072 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8dcf618e-11d2-4248-94c8-8cb8cf24d3ac admin admin] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:24:10.900416 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8dcf618e-11d2-4248-94c8-8cb8cf24d3ac admin admin] PATH_INFO: `/v3/projects` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:24:10.900986 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8dcf618e-11d2-4248-94c8-8cb8cf24d3ac admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:24:10.903056 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8dcf618e-11d2-4248-94c8-8cb8cf24d3ac admin admin] RBAC: Authorization granted {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:24:10.952756 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 965/1960] 10.222.0.27 () {60 vars in 1153 bytes} [Thu Oct 13 04:24:10 2022] POST /identity/v3/projects => generated 385 bytes in 247 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 13 04:24:10.954797 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0d36909d-f7ed-4805-85b4-bc01fd0d4a9c None None] Authenticating user token {{(pid=81409) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:24:10.974328 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0d36909d-f7ed-4805-85b4-bc01fd0d4a9c None None] Validating token access rules against request {{(pid=81409) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:24:11.055688 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-34f6804f-0576-4013-8fd2-00cffb1600b4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8ba823f313144da3b683bacef8fd62e0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81409) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:24:11.056592 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34f6804f-0576-4013-8fd2-00cffb1600b4 admin admin] REQUEST_METHOD: `POST` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:24:11.056890 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34f6804f-0576-4013-8fd2-00cffb1600b4 admin admin] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:24:11.057162 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34f6804f-0576-4013-8fd2-00cffb1600b4 admin admin] PATH_INFO: `/v3/users` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:24:11.057803 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-34f6804f-0576-4013-8fd2-00cffb1600b4 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:24:11.059786 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-34f6804f-0576-4013-8fd2-00cffb1600b4 admin admin] RBAC: Authorization granted {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:24:11.084065 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 996/1961] 10.222.0.27 () {60 vars in 1144 bytes} [Thu Oct 13 04:24:10 2022] POST /identity/v3/users => generated 299 bytes in 130 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 13 04:24:11.088170 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8dcf618e-11d2-4248-94c8-8cb8cf24d3ac admin admin] Authenticating user token {{(pid=81408) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:24:11.104644 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8dcf618e-11d2-4248-94c8-8cb8cf24d3ac admin admin] Validating token access rules against request {{(pid=81408) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:24:11.113561 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f388e092-1b8a-479b-9987-7beb043e3214 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8ba823f313144da3b683bacef8fd62e0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81408) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:24:11.114244 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f388e092-1b8a-479b-9987-7beb043e3214 admin admin] REQUEST_METHOD: `GET` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:24:11.114619 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f388e092-1b8a-479b-9987-7beb043e3214 admin admin] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:24:11.114955 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f388e092-1b8a-479b-9987-7beb043e3214 admin admin] PATH_INFO: `/v3/roles` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:24:11.115732 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f388e092-1b8a-479b-9987-7beb043e3214 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:24:11.117738 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f388e092-1b8a-479b-9987-7beb043e3214 admin admin] RBAC: Authorization granted {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:24:11.130273 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 966/1962] 10.222.0.27 () {58 vars in 1122 bytes} [Thu Oct 13 04:24:11 2022] GET /identity/v3/roles => generated 1420 bytes in 42 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Oct 13 04:24:11.134565 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-34f6804f-0576-4013-8fd2-00cffb1600b4 admin admin] Authenticating user token {{(pid=81409) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:24:11.145977 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-34f6804f-0576-4013-8fd2-00cffb1600b4 admin admin] Validating token access rules against request {{(pid=81409) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:24:11.151435 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2150f74c-17b0-4194-b31f-ab53d21a11fe admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8ba823f313144da3b683bacef8fd62e0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81409) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:24:11.152677 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2150f74c-17b0-4194-b31f-ab53d21a11fe admin admin] REQUEST_METHOD: `PUT` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:24:11.153265 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2150f74c-17b0-4194-b31f-ab53d21a11fe admin admin] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:24:11.153445 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2150f74c-17b0-4194-b31f-ab53d21a11fe admin admin] PATH_INFO: `/v3/projects/f3cec5d37e134dc395624f81a9f77fc1/users/7b4787618a464777b59ec623551de8dc/roles/ef55fbd94e1040be83695754075a7ed5` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:24:11.177036 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2150f74c-17b0-4194-b31f-ab53d21a11fe admin admin] RBAC: Authorizing `identity:create_grant(project_id=f3cec5d37e134dc395624f81a9f77fc1, user_id=7b4787618a464777b59ec623551de8dc, role_id=ef55fbd94e1040be83695754075a7ed5)` {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:24:11.181069 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2150f74c-17b0-4194-b31f-ab53d21a11fe admin admin] RBAC: Authorization granted {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:24:11.198564 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 997/1963] 10.222.0.27 () {60 vars in 1483 bytes} [Thu Oct 13 04:24:11 2022] PUT /identity/v3/projects/f3cec5d37e134dc395624f81a9f77fc1/users/7b4787618a464777b59ec623551de8dc/roles/ef55fbd94e1040be83695754075a7ed5 => generated 0 bytes in 64 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 13 04:24:11.202935 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f388e092-1b8a-479b-9987-7beb043e3214 admin admin] Authenticating user token {{(pid=81408) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:24:11.220846 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f388e092-1b8a-479b-9987-7beb043e3214 admin admin] Validating token access rules against request {{(pid=81408) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:24:11.316222 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a379d8a7-9ac0-495f-bb23-39b6d9eb9cc2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8ba823f313144da3b683bacef8fd62e0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81408) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:24:11.316968 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a379d8a7-9ac0-495f-bb23-39b6d9eb9cc2 admin admin] REQUEST_METHOD: `GET` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:24:11.317430 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a379d8a7-9ac0-495f-bb23-39b6d9eb9cc2 admin admin] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:24:11.317771 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a379d8a7-9ac0-495f-bb23-39b6d9eb9cc2 admin admin] PATH_INFO: `/v3/roles` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:24:11.318557 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a379d8a7-9ac0-495f-bb23-39b6d9eb9cc2 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:24:11.320804 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a379d8a7-9ac0-495f-bb23-39b6d9eb9cc2 admin admin] RBAC: Authorization granted {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:24:11.336970 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 967/1964] 10.222.0.27 () {58 vars in 1122 bytes} [Thu Oct 13 04:24:11 2022] GET /identity/v3/roles => generated 1420 bytes in 135 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 13 04:24:11.341107 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2150f74c-17b0-4194-b31f-ab53d21a11fe admin admin] Authenticating user token {{(pid=81409) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:24:11.358174 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2150f74c-17b0-4194-b31f-ab53d21a11fe admin admin] Validating token access rules against request {{(pid=81409) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:24:11.370321 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8f0837c5-d791-415d-b90e-2ca29cc02b4e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8ba823f313144da3b683bacef8fd62e0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81409) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:24:11.371439 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f0837c5-d791-415d-b90e-2ca29cc02b4e admin admin] REQUEST_METHOD: `PUT` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:24:11.371896 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f0837c5-d791-415d-b90e-2ca29cc02b4e admin admin] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:24:11.372333 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f0837c5-d791-415d-b90e-2ca29cc02b4e admin admin] PATH_INFO: `/v3/projects/f3cec5d37e134dc395624f81a9f77fc1/users/7b4787618a464777b59ec623551de8dc/roles/ef55fbd94e1040be83695754075a7ed5` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:24:11.375058 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8f0837c5-d791-415d-b90e-2ca29cc02b4e admin admin] RBAC: Authorizing `identity:create_grant(project_id=f3cec5d37e134dc395624f81a9f77fc1, user_id=7b4787618a464777b59ec623551de8dc, role_id=ef55fbd94e1040be83695754075a7ed5)` {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:24:11.378935 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8f0837c5-d791-415d-b90e-2ca29cc02b4e admin admin] RBAC: Authorization granted {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:24:11.393411 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 998/1965] 10.222.0.27 () {60 vars in 1483 bytes} [Thu Oct 13 04:24:11 2022] PUT /identity/v3/projects/f3cec5d37e134dc395624f81a9f77fc1/users/7b4787618a464777b59ec623551de8dc/roles/ef55fbd94e1040be83695754075a7ed5 => generated 0 bytes in 53 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 13 04:24:12.430088 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87aceb9c-5af8-4c98-a6b2-25ab7099d8dc None None] REQUEST_METHOD: `POST` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:24:12.431023 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87aceb9c-5af8-4c98-a6b2-25ab7099d8dc None None] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:24:12.431535 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87aceb9c-5af8-4c98-a6b2-25ab7099d8dc None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:24:12.458528 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.auth.core [None req-87aceb9c-5af8-4c98-a6b2-25ab7099d8dc None None] MFA Rules not processed for user `487b1605fd4143cfaa422a3a7a0344c7`. Rule list: `[]` (Enabled: `True`). {{(pid=81408) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 13 04:24:12.461847 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.fernet_utils [None req-87aceb9c-5af8-4c98-a6b2-25ab7099d8dc None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81408) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:24:12.611353 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 968/1966] 10.222.0.27 () {58 vars in 1009 bytes} [Thu Oct 13 04:24:12 2022] POST /identity/v3/auth/tokens => generated 3195 bytes in 182 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 13 04:24:12.625367 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8f0837c5-d791-415d-b90e-2ca29cc02b4e admin admin] Authenticating user token {{(pid=81409) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:24:12.627252 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.fernet_utils [None req-8f0837c5-d791-415d-b90e-2ca29cc02b4e admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81409) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:24:12.745241 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG dbcounter [-] [81409] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=81409) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:24:12.804366 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8f0837c5-d791-415d-b90e-2ca29cc02b4e admin admin] Validating token access rules against request {{(pid=81409) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:24:12.824961 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c9fb7306-c811-4bae-9b3a-e886f2a6bcd0 service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90a0f466b664458898cf9dc4112ae651', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dcec0431fb6849499d6aec35a234daa6', 'project_domain_id': 'default', 'roles': ['service', 'member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81409) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:24:12.825670 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9fb7306-c811-4bae-9b3a-e886f2a6bcd0 service swift] REQUEST_METHOD: `GET` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:24:12.825948 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9fb7306-c811-4bae-9b3a-e886f2a6bcd0 service swift] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:24:12.826301 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9fb7306-c811-4bae-9b3a-e886f2a6bcd0 service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:24:12.839173 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c9fb7306-c811-4bae-9b3a-e886f2a6bcd0 service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:24:12.841754 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c9fb7306-c811-4bae-9b3a-e886f2a6bcd0 service swift] RBAC: Authorization granted {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:24:12.863359 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 999/1967] 10.222.0.27 () {60 vars in 1384 bytes} [Thu Oct 13 04:24:12 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 230 msecs (HTTP/1.1 200) 6 headers in 379 bytes (2 switches on core 0) Oct 13 04:24:18.792932 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c493ff40-fa07-4ce0-b0d1-55a8226d064a None None] REQUEST_METHOD: `POST` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:24:18.793336 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c493ff40-fa07-4ce0-b0d1-55a8226d064a None None] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:24:18.793336 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c493ff40-fa07-4ce0-b0d1-55a8226d064a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:24:18.857921 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.auth.core [None req-c493ff40-fa07-4ce0-b0d1-55a8226d064a None None] MFA Rules not processed for user `7b4787618a464777b59ec623551de8dc`. Rule list: `[]` (Enabled: `True`). {{(pid=81408) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 13 04:24:18.859829 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.fernet_utils [None req-c493ff40-fa07-4ce0-b0d1-55a8226d064a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81408) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:24:18.911885 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG dbcounter [-] [81408] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81408) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:24:18.965445 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 969/1968] 10.222.0.27 () {58 vars in 958 bytes} [Thu Oct 13 04:24:18 2022] POST /identity/v3/auth/tokens => generated 3329 bytes in 174 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 13 04:24:18.982626 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c9fb7306-c811-4bae-9b3a-e886f2a6bcd0 service swift] Authenticating user token {{(pid=81409) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:24:18.984820 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.fernet_utils [None req-c9fb7306-c811-4bae-9b3a-e886f2a6bcd0 service swift] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81409) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:24:19.098804 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c9fb7306-c811-4bae-9b3a-e886f2a6bcd0 service swift] Validating token access rules against request {{(pid=81409) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:24:19.103663 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-03a53dee-e429-4407-99e9-3199df733565 service glance] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a6700f49b86543f6a5b92cad8faaba42', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dcec0431fb6849499d6aec35a234daa6', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81409) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:24:19.104945 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03a53dee-e429-4407-99e9-3199df733565 service glance] REQUEST_METHOD: `GET` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:24:19.105223 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03a53dee-e429-4407-99e9-3199df733565 service glance] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:24:19.105656 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03a53dee-e429-4407-99e9-3199df733565 service glance] PATH_INFO: `/v3/auth/tokens` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:24:19.194053 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-03a53dee-e429-4407-99e9-3199df733565 service glance] RBAC: Authorizing `identity:validate_token()` {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:24:19.195553 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-03a53dee-e429-4407-99e9-3199df733565 service glance] RBAC: Authorization granted {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:24:19.211604 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 1000/1969] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Oct 13 04:24:18 2022] GET /identity/v3/auth/tokens => generated 3262 bytes in 229 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 13 04:24:19.417046 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8be34da-029a-4cc4-889a-6cd58fa98658 None None] REQUEST_METHOD: `POST` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:24:19.417644 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8be34da-029a-4cc4-889a-6cd58fa98658 None None] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:24:19.418140 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8be34da-029a-4cc4-889a-6cd58fa98658 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:24:19.474075 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.auth.core [None req-b8be34da-029a-4cc4-889a-6cd58fa98658 None None] MFA Rules not processed for user `487b1605fd4143cfaa422a3a7a0344c7`. Rule list: `[]` (Enabled: `True`). {{(pid=81408) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 13 04:24:19.476353 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.fernet_utils [None req-b8be34da-029a-4cc4-889a-6cd58fa98658 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81408) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:24:19.485160 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 970/1970] 10.222.0.27 () {58 vars in 1009 bytes} [Thu Oct 13 04:24:19 2022] POST /identity/v3/auth/tokens => generated 3195 bytes in 69 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 13 04:24:19.507368 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-03a53dee-e429-4407-99e9-3199df733565 service glance] Authenticating user token {{(pid=81409) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:24:19.536760 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-03a53dee-e429-4407-99e9-3199df733565 service glance] Validating token access rules against request {{(pid=81409) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:24:19.543670 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1ee4136c-7347-487e-8af2-0585d47fff76 service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90a0f466b664458898cf9dc4112ae651', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dcec0431fb6849499d6aec35a234daa6', 'project_domain_id': 'default', 'roles': ['service', 'member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81409) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:24:19.544314 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ee4136c-7347-487e-8af2-0585d47fff76 service swift] REQUEST_METHOD: `GET` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:24:19.544670 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ee4136c-7347-487e-8af2-0585d47fff76 service swift] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:24:19.545018 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ee4136c-7347-487e-8af2-0585d47fff76 service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:24:19.556035 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1ee4136c-7347-487e-8af2-0585d47fff76 service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:24:19.568182 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1ee4136c-7347-487e-8af2-0585d47fff76 service swift] RBAC: Authorization granted {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:24:19.584591 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 1001/1971] 10.222.0.27 () {60 vars in 1384 bytes} [Thu Oct 13 04:24:19 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 78 msecs (HTTP/1.1 200) 6 headers in 379 bytes (1 switches on core 0) Oct 13 04:24:28.871141 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9f4cada-2537-4ad3-ad41-731d22e98745 None None] REQUEST_METHOD: `POST` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:24:28.871705 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9f4cada-2537-4ad3-ad41-731d22e98745 None None] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:24:28.871705 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9f4cada-2537-4ad3-ad41-731d22e98745 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:24:28.890805 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.auth.core [None req-e9f4cada-2537-4ad3-ad41-731d22e98745 None None] MFA Rules not processed for user `7582f5c0de6b4b5dae18b0320921e4a2`. Rule list: `[]` (Enabled: `True`). {{(pid=81408) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 13 04:24:28.892000 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.fernet_utils [None req-e9f4cada-2537-4ad3-ad41-731d22e98745 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81408) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:24:28.898858 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 971/1972] 10.222.0.27 () {58 vars in 958 bytes} [Thu Oct 13 04:24:28 2022] POST /identity/v3/auth/tokens => generated 3262 bytes in 29 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 13 04:24:29.576347 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG dbcounter [-] [81409] Writing DB stats keystone:SELECT=63 {{(pid=81409) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:24:32.268463 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bba1084c-7cbc-4642-8fa1-36497f8a4c15 None None] REQUEST_METHOD: `POST` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:24:32.269089 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bba1084c-7cbc-4642-8fa1-36497f8a4c15 None None] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:24:32.269089 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bba1084c-7cbc-4642-8fa1-36497f8a4c15 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:24:32.289300 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.auth.core [None req-bba1084c-7cbc-4642-8fa1-36497f8a4c15 None None] MFA Rules not processed for user `487b1605fd4143cfaa422a3a7a0344c7`. Rule list: `[]` (Enabled: `True`). {{(pid=81409) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 13 04:24:32.290541 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.fernet_utils [None req-bba1084c-7cbc-4642-8fa1-36497f8a4c15 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81409) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:24:32.297701 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 1002/1973] 10.222.0.27 () {58 vars in 1009 bytes} [Thu Oct 13 04:24:32 2022] POST /identity/v3/auth/tokens => generated 3195 bytes in 30 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 13 04:24:32.309932 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e9f4cada-2537-4ad3-ad41-731d22e98745 None None] Authenticating user token {{(pid=81408) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:24:32.321800 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e9f4cada-2537-4ad3-ad41-731d22e98745 None None] Validating token access rules against request {{(pid=81408) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:24:32.327668 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bc797803-6eb7-4992-8020-ffa84a925044 service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90a0f466b664458898cf9dc4112ae651', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dcec0431fb6849499d6aec35a234daa6', 'project_domain_id': 'default', 'roles': ['service', 'member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81408) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:24:32.328368 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc797803-6eb7-4992-8020-ffa84a925044 service swift] REQUEST_METHOD: `GET` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:24:32.328711 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc797803-6eb7-4992-8020-ffa84a925044 service swift] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:24:32.329070 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc797803-6eb7-4992-8020-ffa84a925044 service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:24:32.337427 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bc797803-6eb7-4992-8020-ffa84a925044 service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:24:32.339139 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bc797803-6eb7-4992-8020-ffa84a925044 service swift] RBAC: Authorization granted {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:24:32.348442 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 972/1974] 10.222.0.27 () {60 vars in 1384 bytes} [Thu Oct 13 04:24:32 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 39 msecs (HTTP/1.1 200) 6 headers in 379 bytes (1 switches on core 0) Oct 13 04:24:42.283623 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG dbcounter [-] [81409] Writing DB stats keystone:SELECT=3 {{(pid=81409) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:24:42.345433 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG dbcounter [-] [81408] Writing DB stats keystone:SELECT=25 {{(pid=81408) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:24:47.488762 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be772ae5-f926-415e-9c1c-c21e64a178b7 None None] REQUEST_METHOD: `POST` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:24:47.489792 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be772ae5-f926-415e-9c1c-c21e64a178b7 None None] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:24:47.490164 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be772ae5-f926-415e-9c1c-c21e64a178b7 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:24:47.518497 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.auth.core [None req-be772ae5-f926-415e-9c1c-c21e64a178b7 None None] MFA Rules not processed for user `487b1605fd4143cfaa422a3a7a0344c7`. Rule list: `[]` (Enabled: `True`). {{(pid=81409) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 13 04:24:47.519887 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.fernet_utils [None req-be772ae5-f926-415e-9c1c-c21e64a178b7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81409) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:24:47.538886 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 1003/1975] 10.222.0.27 () {58 vars in 967 bytes} [Thu Oct 13 04:24:47 2022] POST /identity/v3/auth/tokens => generated 3195 bytes in 52 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 13 04:24:47.561157 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bc797803-6eb7-4992-8020-ffa84a925044 service swift] Authenticating user token {{(pid=81408) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:24:47.612070 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bc797803-6eb7-4992-8020-ffa84a925044 service swift] Validating token access rules against request {{(pid=81408) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:24:47.646093 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8a0e5e5e-edbf-49bd-9f62-a4cea5e84443 service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90a0f466b664458898cf9dc4112ae651', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dcec0431fb6849499d6aec35a234daa6', 'project_domain_id': 'default', 'roles': ['service', 'member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81408) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:24:47.646093 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a0e5e5e-edbf-49bd-9f62-a4cea5e84443 service swift] REQUEST_METHOD: `GET` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:24:47.646093 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a0e5e5e-edbf-49bd-9f62-a4cea5e84443 service swift] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:24:47.646093 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a0e5e5e-edbf-49bd-9f62-a4cea5e84443 service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:24:47.659848 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8a0e5e5e-edbf-49bd-9f62-a4cea5e84443 service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:24:47.666324 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8a0e5e5e-edbf-49bd-9f62-a4cea5e84443 service swift] RBAC: Authorization granted {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:24:47.703068 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 973/1976] 10.222.0.27 () {60 vars in 1384 bytes} [Thu Oct 13 04:24:47 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 142 msecs (HTTP/1.1 200) 6 headers in 379 bytes (1 switches on core 0) Oct 13 04:24:57.509371 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG dbcounter [-] [81409] Writing DB stats keystone:SELECT=3 {{(pid=81409) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:24:57.677792 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG dbcounter [-] [81408] Writing DB stats keystone:SELECT=6 {{(pid=81408) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:25:02.211746 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5dab7d13-5543-4e9a-96a4-628412dd128c None None] REQUEST_METHOD: `POST` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:25:02.212267 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5dab7d13-5543-4e9a-96a4-628412dd128c None None] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:25:02.212267 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5dab7d13-5543-4e9a-96a4-628412dd128c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:25:02.231845 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.auth.core [None req-5dab7d13-5543-4e9a-96a4-628412dd128c None None] MFA Rules not processed for user `999c89e1bc7a426886fc802e28d38596`. Rule list: `[]` (Enabled: `True`). {{(pid=81409) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 13 04:25:02.233039 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.fernet_utils [None req-5dab7d13-5543-4e9a-96a4-628412dd128c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81409) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:25:02.287286 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 1004/1977] 10.222.0.27 () {58 vars in 958 bytes} [Thu Oct 13 04:25:02 2022] POST /identity/v3/auth/tokens => generated 3260 bytes in 77 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 13 04:25:08.457914 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8a0e5e5e-edbf-49bd-9f62-a4cea5e84443 service swift] Authenticating user token {{(pid=81408) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:25:08.459467 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.fernet_utils [None req-8a0e5e5e-edbf-49bd-9f62-a4cea5e84443 service swift] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81408) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:25:08.518833 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8a0e5e5e-edbf-49bd-9f62-a4cea5e84443 service swift] Validating token access rules against request {{(pid=81408) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:25:08.522379 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6790b937-2b0c-4a1c-bc5d-baf1a6bf10bc service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd8812c3974f84871bb2d761cf0016257', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dcec0431fb6849499d6aec35a234daa6', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81408) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:25:08.522981 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6790b937-2b0c-4a1c-bc5d-baf1a6bf10bc service neutron] REQUEST_METHOD: `GET` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:25:08.523276 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6790b937-2b0c-4a1c-bc5d-baf1a6bf10bc service neutron] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:25:08.523550 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6790b937-2b0c-4a1c-bc5d-baf1a6bf10bc service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:25:08.532984 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6790b937-2b0c-4a1c-bc5d-baf1a6bf10bc service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:25:08.534449 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6790b937-2b0c-4a1c-bc5d-baf1a6bf10bc service neutron] RBAC: Authorization granted {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:25:08.544506 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 974/1978] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Oct 13 04:25:08 2022] GET /identity/v3/auth/tokens => generated 3329 bytes in 87 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 13 04:25:12.279898 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG dbcounter [-] [81409] Writing DB stats keystone:SELECT=25 {{(pid=81409) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:25:14.299032 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49fba6cd-cbc6-4194-8c1b-93435d7c27d8 None None] REQUEST_METHOD: `POST` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:25:14.299668 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49fba6cd-cbc6-4194-8c1b-93435d7c27d8 None None] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:25:14.299668 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49fba6cd-cbc6-4194-8c1b-93435d7c27d8 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:25:14.337771 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.auth.core [None req-49fba6cd-cbc6-4194-8c1b-93435d7c27d8 None None] MFA Rules not processed for user `da9c75191a3b4ef49438aae9e35fcf13`. Rule list: `[]` (Enabled: `True`). {{(pid=81409) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 13 04:25:14.339546 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.fernet_utils [None req-49fba6cd-cbc6-4194-8c1b-93435d7c27d8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81409) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:25:14.407958 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 1005/1979] 10.222.0.27 () {58 vars in 958 bytes} [Thu Oct 13 04:25:14 2022] POST /identity/v3/auth/tokens => generated 3270 bytes in 110 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 13 04:25:18.540241 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG dbcounter [-] [81408] Writing DB stats keystone:SELECT=26 {{(pid=81408) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:25:24.400809 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG dbcounter [-] [81409] Writing DB stats keystone:SELECT=25 {{(pid=81409) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:25:40.126734 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20baad7a-5bf7-4921-bbfc-6f242d279549 None None] REQUEST_METHOD: `POST` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:25:40.127227 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20baad7a-5bf7-4921-bbfc-6f242d279549 None None] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:25:40.127333 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20baad7a-5bf7-4921-bbfc-6f242d279549 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:25:40.146964 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.auth.core [None req-20baad7a-5bf7-4921-bbfc-6f242d279549 None None] MFA Rules not processed for user `35a058c849104a9699ecfcd0c81d398e`. Rule list: `[]` (Enabled: `True`). {{(pid=81408) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 13 04:25:40.148148 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.fernet_utils [None req-20baad7a-5bf7-4921-bbfc-6f242d279549 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81408) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:25:40.201878 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 975/1980] 10.222.0.27 () {58 vars in 958 bytes} [Thu Oct 13 04:25:40 2022] POST /identity/v3/auth/tokens => generated 3266 bytes in 76 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 13 04:25:50.195237 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG dbcounter [-] [81408] Writing DB stats keystone:SELECT=25 {{(pid=81408) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:25:51.524009 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-49fba6cd-cbc6-4194-8c1b-93435d7c27d8 None None] Authenticating user token {{(pid=81409) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:25:51.533681 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-49fba6cd-cbc6-4194-8c1b-93435d7c27d8 None None] Validating token access rules against request {{(pid=81409) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:25:51.537208 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ba47753b-d50c-42da-b3d1-9e3efdb9e08b service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd8812c3974f84871bb2d761cf0016257', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dcec0431fb6849499d6aec35a234daa6', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81409) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:25:51.537807 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba47753b-d50c-42da-b3d1-9e3efdb9e08b service neutron] REQUEST_METHOD: `GET` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:25:51.538082 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba47753b-d50c-42da-b3d1-9e3efdb9e08b service neutron] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:25:51.538395 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba47753b-d50c-42da-b3d1-9e3efdb9e08b service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:25:51.540103 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.fernet_utils [None req-ba47753b-d50c-42da-b3d1-9e3efdb9e08b service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81409) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:25:51.610281 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ba47753b-d50c-42da-b3d1-9e3efdb9e08b service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:25:51.611644 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ba47753b-d50c-42da-b3d1-9e3efdb9e08b service neutron] RBAC: Authorization granted {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:25:51.622553 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 1006/1981] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Oct 13 04:25:51 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 99 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 13 04:25:51.750145 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-20baad7a-5bf7-4921-bbfc-6f242d279549 None None] Authenticating user token {{(pid=81408) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:25:51.751645 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.fernet_utils [None req-20baad7a-5bf7-4921-bbfc-6f242d279549 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81408) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:25:51.763530 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-20baad7a-5bf7-4921-bbfc-6f242d279549 None None] Validating token access rules against request {{(pid=81408) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:25:51.767222 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-25b449d0-c690-4e83-b2eb-ea82d956c30a service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd8812c3974f84871bb2d761cf0016257', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dcec0431fb6849499d6aec35a234daa6', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81408) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:25:51.767815 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25b449d0-c690-4e83-b2eb-ea82d956c30a service neutron] REQUEST_METHOD: `GET` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:25:51.768092 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25b449d0-c690-4e83-b2eb-ea82d956c30a service neutron] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:25:51.768358 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25b449d0-c690-4e83-b2eb-ea82d956c30a service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:25:51.770015 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.fernet_utils [None req-25b449d0-c690-4e83-b2eb-ea82d956c30a service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81408) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:25:51.783374 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-25b449d0-c690-4e83-b2eb-ea82d956c30a service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:25:51.784960 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-25b449d0-c690-4e83-b2eb-ea82d956c30a service neutron] RBAC: Authorization granted {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:25:51.796809 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 976/1982] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Oct 13 04:25:51 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 48 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 13 04:26:01.617678 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG dbcounter [-] [81409] Writing DB stats keystone:SELECT=34 {{(pid=81409) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:26:01.791235 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG dbcounter [-] [81408] Writing DB stats keystone:SELECT=6 {{(pid=81408) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:26:06.066703 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ba47753b-d50c-42da-b3d1-9e3efdb9e08b service neutron] Authenticating user token {{(pid=81409) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:26:06.068093 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.fernet_utils [None req-ba47753b-d50c-42da-b3d1-9e3efdb9e08b service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81409) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:26:06.081695 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ba47753b-d50c-42da-b3d1-9e3efdb9e08b service neutron] Validating token access rules against request {{(pid=81409) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:26:06.087121 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7bb1b3ae-4b8f-49eb-a646-8f689fdfdd3e service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cb740ed7f7684b079e45f887be83f795', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dcec0431fb6849499d6aec35a234daa6', 'project_domain_id': 'default', 'roles': ['ResellerAdmin', 'admin', 'reader', 'service', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81409) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:26:06.087720 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7bb1b3ae-4b8f-49eb-a646-8f689fdfdd3e service nova] REQUEST_METHOD: `GET` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:26:06.087993 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7bb1b3ae-4b8f-49eb-a646-8f689fdfdd3e service nova] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:26:06.088262 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7bb1b3ae-4b8f-49eb-a646-8f689fdfdd3e service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:26:06.089900 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.fernet_utils [None req-7bb1b3ae-4b8f-49eb-a646-8f689fdfdd3e service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81409) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:26:06.100280 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7bb1b3ae-4b8f-49eb-a646-8f689fdfdd3e service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:26:06.101835 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7bb1b3ae-4b8f-49eb-a646-8f689fdfdd3e service nova] RBAC: Authorization granted {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:26:06.111455 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 1007/1983] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Oct 13 04:26:06 2022] GET /identity/v3/auth/tokens => generated 3260 bytes in 45 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 13 04:26:13.584298 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-25b449d0-c690-4e83-b2eb-ea82d956c30a service neutron] Authenticating user token {{(pid=81408) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:26:13.585634 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.fernet_utils [None req-25b449d0-c690-4e83-b2eb-ea82d956c30a service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81408) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:26:13.599798 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-25b449d0-c690-4e83-b2eb-ea82d956c30a service neutron] Validating token access rules against request {{(pid=81408) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:26:13.605348 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-01f84bfa-67d0-4176-812b-2fa23a81f69c service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cb740ed7f7684b079e45f887be83f795', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dcec0431fb6849499d6aec35a234daa6', 'project_domain_id': 'default', 'roles': ['ResellerAdmin', 'admin', 'reader', 'service', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81408) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:26:13.605944 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01f84bfa-67d0-4176-812b-2fa23a81f69c service nova] REQUEST_METHOD: `GET` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:26:13.606531 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01f84bfa-67d0-4176-812b-2fa23a81f69c service nova] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:26:13.606806 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01f84bfa-67d0-4176-812b-2fa23a81f69c service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:26:13.608514 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.fernet_utils [None req-01f84bfa-67d0-4176-812b-2fa23a81f69c service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81408) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:26:13.618743 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-01f84bfa-67d0-4176-812b-2fa23a81f69c service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:26:13.620310 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-01f84bfa-67d0-4176-812b-2fa23a81f69c service nova] RBAC: Authorization granted {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:26:13.629837 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 977/1984] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Oct 13 04:26:13 2022] GET /identity/v3/auth/tokens => generated 3270 bytes in 46 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 13 04:26:16.106852 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG dbcounter [-] [81409] Writing DB stats keystone:SELECT=6 {{(pid=81409) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:26:23.624929 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG dbcounter [-] [81408] Writing DB stats keystone:SELECT=6 {{(pid=81408) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:26:28.359663 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7bb1b3ae-4b8f-49eb-a646-8f689fdfdd3e service nova] Authenticating user token {{(pid=81409) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:26:28.362294 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.fernet_utils [None req-7bb1b3ae-4b8f-49eb-a646-8f689fdfdd3e service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81409) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:26:28.427114 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7bb1b3ae-4b8f-49eb-a646-8f689fdfdd3e service nova] Validating token access rules against request {{(pid=81409) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:26:28.430996 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-06756588-b482-4e5c-bb9e-d4267201f522 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6141b8a6c3d54e179d6879bb019f4d12', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dcec0431fb6849499d6aec35a234daa6', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81409) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:26:28.431649 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06756588-b482-4e5c-bb9e-d4267201f522 service cinder] REQUEST_METHOD: `GET` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:26:28.431945 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06756588-b482-4e5c-bb9e-d4267201f522 service cinder] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:26:28.432217 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06756588-b482-4e5c-bb9e-d4267201f522 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:26:28.433971 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.fernet_utils [None req-06756588-b482-4e5c-bb9e-d4267201f522 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81409) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:26:28.496320 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-06756588-b482-4e5c-bb9e-d4267201f522 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:26:28.497772 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-06756588-b482-4e5c-bb9e-d4267201f522 service cinder] RBAC: Authorization granted {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:26:28.508126 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 1008/1985] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Oct 13 04:26:28 2022] GET /identity/v3/auth/tokens => generated 3254 bytes in 149 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 13 04:26:29.845595 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-01f84bfa-67d0-4176-812b-2fa23a81f69c service nova] Authenticating user token {{(pid=81408) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:26:29.855211 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-01f84bfa-67d0-4176-812b-2fa23a81f69c service nova] Validating token access rules against request {{(pid=81408) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:26:29.859236 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ab4f3342-72ec-47b2-a54f-695e1932b78a service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd8812c3974f84871bb2d761cf0016257', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dcec0431fb6849499d6aec35a234daa6', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81408) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:26:29.859913 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab4f3342-72ec-47b2-a54f-695e1932b78a service neutron] REQUEST_METHOD: `GET` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:26:29.860194 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab4f3342-72ec-47b2-a54f-695e1932b78a service neutron] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:26:29.860469 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab4f3342-72ec-47b2-a54f-695e1932b78a service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:26:29.862286 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.fernet_utils [None req-ab4f3342-72ec-47b2-a54f-695e1932b78a service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81408) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:26:29.926992 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ab4f3342-72ec-47b2-a54f-695e1932b78a service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:26:29.928478 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ab4f3342-72ec-47b2-a54f-695e1932b78a service neutron] RBAC: Authorization granted {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:26:29.938621 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 978/1986] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Oct 13 04:26:29 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 94 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 13 04:26:31.530081 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-06756588-b482-4e5c-bb9e-d4267201f522 service cinder] Authenticating user token {{(pid=81409) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:26:31.546891 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-06756588-b482-4e5c-bb9e-d4267201f522 service cinder] Validating token access rules against request {{(pid=81409) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:26:31.552233 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eba03d22-83a8-4f9c-916b-cc53bd6fa9f0 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd8812c3974f84871bb2d761cf0016257', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dcec0431fb6849499d6aec35a234daa6', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81409) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:26:31.553483 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eba03d22-83a8-4f9c-916b-cc53bd6fa9f0 service neutron] REQUEST_METHOD: `GET` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:26:31.554419 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eba03d22-83a8-4f9c-916b-cc53bd6fa9f0 service neutron] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:26:31.554951 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eba03d22-83a8-4f9c-916b-cc53bd6fa9f0 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:26:31.556975 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.fernet_utils [None req-eba03d22-83a8-4f9c-916b-cc53bd6fa9f0 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81409) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:26:31.568884 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eba03d22-83a8-4f9c-916b-cc53bd6fa9f0 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:26:31.570914 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eba03d22-83a8-4f9c-916b-cc53bd6fa9f0 service neutron] RBAC: Authorization granted {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:26:31.583267 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 1009/1987] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Oct 13 04:26:31 2022] GET /identity/v3/auth/tokens => generated 3121 bytes in 54 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Oct 13 04:26:33.787246 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ab4f3342-72ec-47b2-a54f-695e1932b78a service neutron] Authenticating user token {{(pid=81408) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:26:33.810139 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ab4f3342-72ec-47b2-a54f-695e1932b78a service neutron] Validating token access rules against request {{(pid=81408) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:26:33.821769 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-64bfba68-c09e-4ae0-a49b-a116188bad05 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cb740ed7f7684b079e45f887be83f795', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dcec0431fb6849499d6aec35a234daa6', 'project_domain_id': 'default', 'roles': ['ResellerAdmin', 'admin', 'reader', 'service', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81408) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:26:33.822665 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64bfba68-c09e-4ae0-a49b-a116188bad05 service nova] REQUEST_METHOD: `GET` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:26:33.823116 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64bfba68-c09e-4ae0-a49b-a116188bad05 service nova] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:26:33.823675 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64bfba68-c09e-4ae0-a49b-a116188bad05 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:26:33.828813 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.fernet_utils [None req-64bfba68-c09e-4ae0-a49b-a116188bad05 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81408) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:26:33.856833 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-64bfba68-c09e-4ae0-a49b-a116188bad05 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:26:33.858420 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-64bfba68-c09e-4ae0-a49b-a116188bad05 service nova] RBAC: Authorization granted {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:26:33.890016 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 979/1988] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Oct 13 04:26:33 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 103 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 13 04:26:34.539708 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2851fb15-9bb8-4561-ae4e-a9888920a9e6 None None] REQUEST_METHOD: `POST` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:26:34.540727 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2851fb15-9bb8-4561-ae4e-a9888920a9e6 None None] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:26:34.541375 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2851fb15-9bb8-4561-ae4e-a9888920a9e6 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:26:34.564466 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.auth.core [None req-2851fb15-9bb8-4561-ae4e-a9888920a9e6 None None] MFA Rules not processed for user `999c89e1bc7a426886fc802e28d38596`. Rule list: `[]` (Enabled: `True`). {{(pid=81409) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 13 04:26:34.566472 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.fernet_utils [None req-2851fb15-9bb8-4561-ae4e-a9888920a9e6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81409) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:26:34.580476 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 1010/1989] 10.222.0.27 () {58 vars in 958 bytes} [Thu Oct 13 04:26:34 2022] POST /identity/v3/auth/tokens => generated 3260 bytes in 42 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 13 04:26:35.217595 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-64bfba68-c09e-4ae0-a49b-a116188bad05 service nova] Authenticating user token {{(pid=81408) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:26:35.220454 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.fernet_utils [None req-64bfba68-c09e-4ae0-a49b-a116188bad05 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81408) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:26:35.326454 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-64bfba68-c09e-4ae0-a49b-a116188bad05 service nova] Validating token access rules against request {{(pid=81408) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:26:35.334199 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-15317be3-b6a4-4330-9337-ee68f7c8806a service placement] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ba1b6e5868a549a99760ebe6068fd6b9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dcec0431fb6849499d6aec35a234daa6', 'project_domain_id': 'default', 'roles': ['service', 'member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81408) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:26:35.335196 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15317be3-b6a4-4330-9337-ee68f7c8806a service placement] REQUEST_METHOD: `GET` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:26:35.335412 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15317be3-b6a4-4330-9337-ee68f7c8806a service placement] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:26:35.335864 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15317be3-b6a4-4330-9337-ee68f7c8806a service placement] PATH_INFO: `/v3/auth/tokens` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:26:35.338923 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.fernet_utils [None req-15317be3-b6a4-4330-9337-ee68f7c8806a service placement] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81408) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:26:35.342854 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2851fb15-9bb8-4561-ae4e-a9888920a9e6 None None] Authenticating user token {{(pid=81409) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:26:35.344995 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.fernet_utils [None req-2851fb15-9bb8-4561-ae4e-a9888920a9e6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81409) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:26:35.361937 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-15317be3-b6a4-4330-9337-ee68f7c8806a service placement] RBAC: Authorizing `identity:validate_token()` {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:26:35.363983 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-15317be3-b6a4-4330-9337-ee68f7c8806a service placement] RBAC: Authorization granted {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:26:35.373460 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2851fb15-9bb8-4561-ae4e-a9888920a9e6 None None] Validating token access rules against request {{(pid=81409) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:26:35.382012 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8c4cc84f-901e-4635-9a38-7cf63f3bf8cd admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8ba823f313144da3b683bacef8fd62e0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81409) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:26:35.383231 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c4cc84f-901e-4635-9a38-7cf63f3bf8cd admin admin] REQUEST_METHOD: `DELETE` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:26:35.383667 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c4cc84f-901e-4635-9a38-7cf63f3bf8cd admin admin] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:26:35.384096 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c4cc84f-901e-4635-9a38-7cf63f3bf8cd admin admin] PATH_INFO: `/v3/users/da9c75191a3b4ef49438aae9e35fcf13` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:26:35.385481 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8c4cc84f-901e-4635-9a38-7cf63f3bf8cd admin admin] RBAC: Authorizing `identity:delete_user(user_id=da9c75191a3b4ef49438aae9e35fcf13)` {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:26:35.385892 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 980/1990] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Oct 13 04:26:35 2022] GET /identity/v3/auth/tokens => generated 3308 bytes in 168 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 13 04:26:35.386933 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-15317be3-b6a4-4330-9337-ee68f7c8806a service placement] Authenticating user token {{(pid=81408) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:26:35.388611 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8c4cc84f-901e-4635-9a38-7cf63f3bf8cd admin admin] RBAC: Authorization granted {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:26:35.408163 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-15317be3-b6a4-4330-9337-ee68f7c8806a service placement] Validating token access rules against request {{(pid=81408) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:26:35.414111 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2f7af14f-8e42-42d1-89ef-b3019fa998f3 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd8812c3974f84871bb2d761cf0016257', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dcec0431fb6849499d6aec35a234daa6', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81408) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:26:35.414990 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f7af14f-8e42-42d1-89ef-b3019fa998f3 service neutron] REQUEST_METHOD: `GET` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:26:35.415559 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f7af14f-8e42-42d1-89ef-b3019fa998f3 service neutron] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:26:35.415997 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f7af14f-8e42-42d1-89ef-b3019fa998f3 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:26:35.417971 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.fernet_utils [None req-2f7af14f-8e42-42d1-89ef-b3019fa998f3 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81408) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:26:35.432171 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2f7af14f-8e42-42d1-89ef-b3019fa998f3 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:26:35.434127 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2f7af14f-8e42-42d1-89ef-b3019fa998f3 service neutron] RBAC: Authorization granted {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:26:35.448148 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 981/1991] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Oct 13 04:26:35 2022] GET /identity/v3/auth/tokens => generated 3121 bytes in 62 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 13 04:26:35.449010 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2f7af14f-8e42-42d1-89ef-b3019fa998f3 service neutron] Authenticating user token {{(pid=81408) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:26:35.466069 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2f7af14f-8e42-42d1-89ef-b3019fa998f3 service neutron] Validating token access rules against request {{(pid=81408) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:26:35.474431 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-248660de-913f-4197-a603-ef4fd5d00bd5 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cb740ed7f7684b079e45f887be83f795', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dcec0431fb6849499d6aec35a234daa6', 'project_domain_id': 'default', 'roles': ['ResellerAdmin', 'admin', 'reader', 'service', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81408) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:26:35.475102 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.notifications [None req-8c4cc84f-901e-4635-9a38-7cf63f3bf8cd admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'da9c75191a3b4ef49438aae9e35fcf13'} {{(pid=81409) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 13 04:26:35.475461 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-248660de-913f-4197-a603-ef4fd5d00bd5 service nova] REQUEST_METHOD: `GET` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:26:35.475899 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.notifications [None req-8c4cc84f-901e-4635-9a38-7cf63f3bf8cd admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'da9c75191a3b4ef49438aae9e35fcf13'} {{(pid=81409) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 13 04:26:35.476049 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-248660de-913f-4197-a603-ef4fd5d00bd5 service nova] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:26:35.476455 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-248660de-913f-4197-a603-ef4fd5d00bd5 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:26:35.478576 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.fernet_utils [None req-248660de-913f-4197-a603-ef4fd5d00bd5 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81408) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:26:35.485359 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.notifications [None req-8c4cc84f-901e-4635-9a38-7cf63f3bf8cd admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'da9c75191a3b4ef49438aae9e35fcf13'} {{(pid=81409) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 13 04:26:35.486123 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.notifications [None req-8c4cc84f-901e-4635-9a38-7cf63f3bf8cd admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'da9c75191a3b4ef49438aae9e35fcf13'} {{(pid=81409) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 13 04:26:35.493549 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-248660de-913f-4197-a603-ef4fd5d00bd5 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:26:35.495231 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-248660de-913f-4197-a603-ef4fd5d00bd5 service nova] RBAC: Authorization granted {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:26:35.502834 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.notifications [None req-8c4cc84f-901e-4635-9a38-7cf63f3bf8cd admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'da9c75191a3b4ef49438aae9e35fcf13'} {{(pid=81409) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 13 04:26:35.507689 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 982/1992] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Oct 13 04:26:35 2022] GET /identity/v3/auth/tokens => generated 3266 bytes in 59 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 13 04:26:35.548276 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 1011/1993] 10.222.0.27 () {58 vars in 1224 bytes} [Thu Oct 13 04:26:35 2022] DELETE /identity/v3/users/da9c75191a3b4ef49438aae9e35fcf13 => generated 0 bytes in 206 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 13 04:26:35.907003 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-248660de-913f-4197-a603-ef4fd5d00bd5 service nova] Authenticating user token {{(pid=81408) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:26:35.908556 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.fernet_utils [None req-248660de-913f-4197-a603-ef4fd5d00bd5 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81408) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:26:35.930458 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-248660de-913f-4197-a603-ef4fd5d00bd5 service nova] Validating token access rules against request {{(pid=81408) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:26:35.985583 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG dbcounter [-] [81408] Writing DB stats keystone:SELECT=100 {{(pid=81408) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:26:35.995988 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8c4cc84f-901e-4635-9a38-7cf63f3bf8cd admin admin] Authenticating user token {{(pid=81409) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:26:35.997393 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.fernet_utils [None req-8c4cc84f-901e-4635-9a38-7cf63f3bf8cd admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81409) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:26:36.031465 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d1c0075e-da38-47d0-9eed-23cfaadbf597 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8ba823f313144da3b683bacef8fd62e0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81408) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:26:36.032438 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1c0075e-da38-47d0-9eed-23cfaadbf597 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:26:36.032973 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1c0075e-da38-47d0-9eed-23cfaadbf597 admin admin] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:26:36.033495 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1c0075e-da38-47d0-9eed-23cfaadbf597 admin admin] PATH_INFO: `/v3/projects/0913f8eb06164012b366d62c553d261c` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:26:36.033620 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG dbcounter [-] [81409] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81409) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:26:36.036725 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d1c0075e-da38-47d0-9eed-23cfaadbf597 admin admin] RBAC: Authorizing `identity:delete_project(project_id=0913f8eb06164012b366d62c553d261c)` {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:26:36.039729 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d1c0075e-da38-47d0-9eed-23cfaadbf597 admin admin] RBAC: Authorization granted {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:26:36.154381 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8c4cc84f-901e-4635-9a38-7cf63f3bf8cd admin admin] Validating token access rules against request {{(pid=81409) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:26:36.167038 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.notifications [None req-d1c0075e-da38-47d0-9eed-23cfaadbf597 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '0913f8eb06164012b366d62c553d261c', 'request_id': 'req-d1c0075e-da38-47d0-9eed-23cfaadbf597'} {{(pid=81408) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 13 04:26:36.177041 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.notifications [None req-d1c0075e-da38-47d0-9eed-23cfaadbf597 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '0913f8eb06164012b366d62c553d261c', 'request_id': 'req-d1c0075e-da38-47d0-9eed-23cfaadbf597'} {{(pid=81408) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 13 04:26:36.186106 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.notifications [None req-d1c0075e-da38-47d0-9eed-23cfaadbf597 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '0913f8eb06164012b366d62c553d261c', 'request_id': 'req-d1c0075e-da38-47d0-9eed-23cfaadbf597'} {{(pid=81408) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 13 04:26:36.202105 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.notifications [None req-d1c0075e-da38-47d0-9eed-23cfaadbf597 admin admin] The token cache is being invalidate because project 0913f8eb06164012b366d62c553d261c was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81408) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 13 04:26:36.202590 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.notifications [None req-d1c0075e-da38-47d0-9eed-23cfaadbf597 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81408) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 13 04:26:36.203264 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.notifications [None req-d1c0075e-da38-47d0-9eed-23cfaadbf597 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81408) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 13 04:26:36.205457 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 983/1994] 10.222.0.27 () {58 vars in 1233 bytes} [Thu Oct 13 04:26:35 2022] DELETE /identity/v3/projects/0913f8eb06164012b366d62c553d261c => generated 0 bytes in 299 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 13 04:26:36.211228 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15243c86-f635-4400-803c-53194ac48b36 None None] REQUEST_METHOD: `POST` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:26:36.211519 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15243c86-f635-4400-803c-53194ac48b36 None None] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:26:36.211809 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15243c86-f635-4400-803c-53194ac48b36 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:26:36.238750 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-49a51bf5-97e0-42fb-bc68-f46996981a2c service placement] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ba1b6e5868a549a99760ebe6068fd6b9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dcec0431fb6849499d6aec35a234daa6', 'project_domain_id': 'default', 'roles': ['service', 'member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81409) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:26:36.239456 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49a51bf5-97e0-42fb-bc68-f46996981a2c service placement] REQUEST_METHOD: `GET` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:26:36.239723 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49a51bf5-97e0-42fb-bc68-f46996981a2c service placement] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:26:36.239995 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49a51bf5-97e0-42fb-bc68-f46996981a2c service placement] PATH_INFO: `/v3/auth/tokens` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:26:36.242035 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.fernet_utils [None req-49a51bf5-97e0-42fb-bc68-f46996981a2c service placement] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81409) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:26:36.244346 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.auth.core [None req-15243c86-f635-4400-803c-53194ac48b36 None None] MFA Rules not processed for user `0e2d979563d6421995451a0e86bf3022`. Rule list: `[]` (Enabled: `True`). {{(pid=81408) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 13 04:26:36.245740 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.fernet_utils [None req-15243c86-f635-4400-803c-53194ac48b36 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81408) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:26:36.255594 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-49a51bf5-97e0-42fb-bc68-f46996981a2c service placement] RBAC: Authorizing `identity:validate_token()` {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:26:36.257313 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-49a51bf5-97e0-42fb-bc68-f46996981a2c service placement] RBAC: Authorization granted {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:26:36.271172 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 1012/1995] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Oct 13 04:26:35 2022] GET /identity/v3/auth/tokens => generated 3308 bytes in 275 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 13 04:26:36.319420 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 984/1996] 10.222.0.27 () {58 vars in 958 bytes} [Thu Oct 13 04:26:36 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 109 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 13 04:26:36.327414 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d36cbbd7-8407-4044-99e4-5b6d48a7571c None None] REQUEST_METHOD: `POST` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:26:36.327698 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d36cbbd7-8407-4044-99e4-5b6d48a7571c None None] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:26:36.327967 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d36cbbd7-8407-4044-99e4-5b6d48a7571c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:26:36.351278 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.auth.core [None req-d36cbbd7-8407-4044-99e4-5b6d48a7571c None None] MFA Rules not processed for user `0e2d979563d6421995451a0e86bf3022`. Rule list: `[]` (Enabled: `True`). {{(pid=81409) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 13 04:26:36.352580 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.fernet_utils [None req-d36cbbd7-8407-4044-99e4-5b6d48a7571c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81409) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:26:36.363464 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 1013/1997] 10.222.0.27 () {58 vars in 958 bytes} [Thu Oct 13 04:26:36 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 36 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 13 04:26:36.367830 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-15243c86-f635-4400-803c-53194ac48b36 None None] Authenticating user token {{(pid=81408) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:26:36.377693 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-15243c86-f635-4400-803c-53194ac48b36 None None] Validating token access rules against request {{(pid=81408) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:26:36.382785 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5e13d35a-f75b-4914-9acd-ffd0590d42d6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8ba823f313144da3b683bacef8fd62e0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81408) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:26:36.383386 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e13d35a-f75b-4914-9acd-ffd0590d42d6 admin admin] REQUEST_METHOD: `GET` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:26:36.383711 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e13d35a-f75b-4914-9acd-ffd0590d42d6 admin admin] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:26:36.383916 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e13d35a-f75b-4914-9acd-ffd0590d42d6 admin admin] PATH_INFO: `/v3/domains` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:26:36.384586 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5e13d35a-f75b-4914-9acd-ffd0590d42d6 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81408) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 13 04:26:36.384886 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5e13d35a-f75b-4914-9acd-ffd0590d42d6 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:26:36.386543 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5e13d35a-f75b-4914-9acd-ffd0590d42d6 admin admin] RBAC: Authorization granted {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:26:36.399609 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 985/1998] 10.222.0.27 () {58 vars in 1153 bytes} [Thu Oct 13 04:26:36 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 13 04:26:36.403847 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d36cbbd7-8407-4044-99e4-5b6d48a7571c None None] Authenticating user token {{(pid=81409) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:26:36.414504 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d36cbbd7-8407-4044-99e4-5b6d48a7571c None None] Validating token access rules against request {{(pid=81409) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:26:36.422483 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9bd63811-5981-431d-9401-dce7a0367884 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8ba823f313144da3b683bacef8fd62e0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81409) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:26:36.423096 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9bd63811-5981-431d-9401-dce7a0367884 admin admin] REQUEST_METHOD: `POST` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:26:36.423370 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9bd63811-5981-431d-9401-dce7a0367884 admin admin] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:26:36.423661 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9bd63811-5981-431d-9401-dce7a0367884 admin admin] PATH_INFO: `/v3/projects` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:26:36.424418 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9bd63811-5981-431d-9401-dce7a0367884 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:26:36.427906 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9bd63811-5981-431d-9401-dce7a0367884 admin admin] RBAC: Authorization granted {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:26:36.458964 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 1014/1999] 10.222.0.27 () {60 vars in 1153 bytes} [Thu Oct 13 04:26:36 2022] POST /identity/v3/projects => generated 405 bytes in 56 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 13 04:26:36.463341 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5e13d35a-f75b-4914-9acd-ffd0590d42d6 admin admin] Authenticating user token {{(pid=81408) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:26:36.476083 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5e13d35a-f75b-4914-9acd-ffd0590d42d6 admin admin] Validating token access rules against request {{(pid=81408) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:26:36.554307 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-11c498e3-7acd-4459-9847-42b9fd120fb6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8ba823f313144da3b683bacef8fd62e0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81408) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:26:36.555223 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11c498e3-7acd-4459-9847-42b9fd120fb6 admin admin] REQUEST_METHOD: `POST` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:26:36.555879 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11c498e3-7acd-4459-9847-42b9fd120fb6 admin admin] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:26:36.556308 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11c498e3-7acd-4459-9847-42b9fd120fb6 admin admin] PATH_INFO: `/v3/users` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:26:36.557208 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-11c498e3-7acd-4459-9847-42b9fd120fb6 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:26:36.559878 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-11c498e3-7acd-4459-9847-42b9fd120fb6 admin admin] RBAC: Authorization granted {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:26:36.572173 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG dbcounter [-] [81408] Writing DB stats keystone:SELECT=91,keystone:DELETE=7,keystone:INSERT=2 {{(pid=81408) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:26:36.588179 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 986/2000] 10.222.0.27 () {60 vars in 1144 bytes} [Thu Oct 13 04:26:36 2022] POST /identity/v3/users => generated 303 bytes in 125 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 13 04:26:36.592209 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9bd63811-5981-431d-9401-dce7a0367884 admin admin] Authenticating user token {{(pid=81409) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:26:36.606488 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9bd63811-5981-431d-9401-dce7a0367884 admin admin] Validating token access rules against request {{(pid=81409) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:26:36.614168 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9994d656-6acc-4b0d-b6b8-2ae0d3bcc252 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8ba823f313144da3b683bacef8fd62e0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81409) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:26:36.615186 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9994d656-6acc-4b0d-b6b8-2ae0d3bcc252 admin admin] REQUEST_METHOD: `GET` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:26:36.616262 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9994d656-6acc-4b0d-b6b8-2ae0d3bcc252 admin admin] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:26:36.616676 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9994d656-6acc-4b0d-b6b8-2ae0d3bcc252 admin admin] PATH_INFO: `/v3/roles` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:26:36.617324 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9994d656-6acc-4b0d-b6b8-2ae0d3bcc252 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:26:36.619162 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9994d656-6acc-4b0d-b6b8-2ae0d3bcc252 admin admin] RBAC: Authorization granted {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:26:36.634830 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 1015/2001] 10.222.0.27 () {58 vars in 1122 bytes} [Thu Oct 13 04:26:36 2022] GET /identity/v3/roles => generated 1420 bytes in 43 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 13 04:26:36.635778 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-11c498e3-7acd-4459-9847-42b9fd120fb6 admin admin] Authenticating user token {{(pid=81408) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:26:36.647168 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-11c498e3-7acd-4459-9847-42b9fd120fb6 admin admin] Validating token access rules against request {{(pid=81408) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:26:36.653355 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-93f4ac07-8ab0-4ddc-be4e-127cd4bf7e94 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8ba823f313144da3b683bacef8fd62e0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81408) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:26:36.654124 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93f4ac07-8ab0-4ddc-be4e-127cd4bf7e94 admin admin] REQUEST_METHOD: `GET` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:26:36.654533 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93f4ac07-8ab0-4ddc-be4e-127cd4bf7e94 admin admin] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:26:36.655181 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93f4ac07-8ab0-4ddc-be4e-127cd4bf7e94 admin admin] PATH_INFO: `/v3/roles` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:26:36.656067 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-93f4ac07-8ab0-4ddc-be4e-127cd4bf7e94 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:26:36.658301 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-93f4ac07-8ab0-4ddc-be4e-127cd4bf7e94 admin admin] RBAC: Authorization granted {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:26:36.673399 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 987/2002] 10.222.0.27 () {58 vars in 1122 bytes} [Thu Oct 13 04:26:36 2022] GET /identity/v3/roles => generated 1420 bytes in 38 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Oct 13 04:26:36.680200 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9994d656-6acc-4b0d-b6b8-2ae0d3bcc252 admin admin] Authenticating user token {{(pid=81409) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:26:36.694506 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9994d656-6acc-4b0d-b6b8-2ae0d3bcc252 admin admin] Validating token access rules against request {{(pid=81409) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:26:36.701729 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c8cf5634-d8d3-4ee9-af54-733b6def1963 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8ba823f313144da3b683bacef8fd62e0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81409) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:26:36.702671 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8cf5634-d8d3-4ee9-af54-733b6def1963 admin admin] REQUEST_METHOD: `PUT` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:26:36.703132 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8cf5634-d8d3-4ee9-af54-733b6def1963 admin admin] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:26:36.703610 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8cf5634-d8d3-4ee9-af54-733b6def1963 admin admin] PATH_INFO: `/v3/projects/d5f35c396a4c4d6fb8980583f91f80e5/users/9fdfe96132ff4c50b12930134543ccbc/roles/cc56c4ae13a7424ea771c15da4b153c1` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:26:36.721014 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c8cf5634-d8d3-4ee9-af54-733b6def1963 admin admin] RBAC: Authorizing `identity:create_grant(project_id=d5f35c396a4c4d6fb8980583f91f80e5, user_id=9fdfe96132ff4c50b12930134543ccbc, role_id=cc56c4ae13a7424ea771c15da4b153c1)` {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:26:36.724477 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c8cf5634-d8d3-4ee9-af54-733b6def1963 admin admin] RBAC: Authorization granted {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:26:36.741494 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 1016/2003] 10.222.0.27 () {60 vars in 1483 bytes} [Thu Oct 13 04:26:36 2022] PUT /identity/v3/projects/d5f35c396a4c4d6fb8980583f91f80e5/users/9fdfe96132ff4c50b12930134543ccbc/roles/cc56c4ae13a7424ea771c15da4b153c1 => generated 0 bytes in 62 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 13 04:26:36.756025 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0c115ea-7055-4ba1-afbb-e1b89ba5a70a None None] REQUEST_METHOD: `POST` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:26:36.756380 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0c115ea-7055-4ba1-afbb-e1b89ba5a70a None None] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:26:36.756632 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0c115ea-7055-4ba1-afbb-e1b89ba5a70a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:26:36.790782 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.auth.core [None req-a0c115ea-7055-4ba1-afbb-e1b89ba5a70a None None] MFA Rules not processed for user `9fdfe96132ff4c50b12930134543ccbc`. Rule list: `[]` (Enabled: `True`). {{(pid=81408) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 13 04:26:36.792070 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.fernet_utils [None req-a0c115ea-7055-4ba1-afbb-e1b89ba5a70a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81408) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:26:36.827727 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e5afa9b2-57cd-403c-8326-2388a1846736 None None] REQUEST_METHOD: `GET` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:26:36.828145 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e5afa9b2-57cd-403c-8326-2388a1846736 None None] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:26:36.828449 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e5afa9b2-57cd-403c-8326-2388a1846736 None None] PATH_INFO: `/` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:26:36.830110 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 1017/2004] 10.222.0.27 () {54 vars in 921 bytes} [Thu Oct 13 04:26:36 2022] GET /identity => generated 270 bytes in 6 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 13 04:26:36.843203 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7c6f741-7420-4f26-9571-dc34949d26c6 None None] REQUEST_METHOD: `POST` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:26:36.843546 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7c6f741-7420-4f26-9571-dc34949d26c6 None None] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:26:36.844199 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7c6f741-7420-4f26-9571-dc34949d26c6 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:26:36.876661 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.auth.core [None req-e7c6f741-7420-4f26-9571-dc34949d26c6 None None] MFA Rules not processed for user `d8812c3974f84871bb2d761cf0016257`. Rule list: `[]` (Enabled: `True`). {{(pid=81409) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 13 04:26:36.883506 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.fernet_utils [None req-e7c6f741-7420-4f26-9571-dc34949d26c6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81409) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:26:36.906383 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 988/2005] 10.222.0.27 () {58 vars in 958 bytes} [Thu Oct 13 04:26:36 2022] POST /identity/v3/auth/tokens => generated 3282 bytes in 152 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 13 04:26:36.911013 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a0c115ea-7055-4ba1-afbb-e1b89ba5a70a None None] Authenticating user token {{(pid=81408) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:26:36.934587 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG dbcounter [-] [81409] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=81409) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:26:36.961720 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 1018/2006] 10.222.0.27 () {58 vars in 1018 bytes} [Thu Oct 13 04:26:36 2022] POST /identity/v3/auth/tokens => generated 3121 bytes in 119 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 13 04:26:36.976517 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e7c6f741-7420-4f26-9571-dc34949d26c6 None None] Authenticating user token {{(pid=81409) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:26:36.978652 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.fernet_utils [None req-e7c6f741-7420-4f26-9571-dc34949d26c6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81409) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:26:36.992334 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e7c6f741-7420-4f26-9571-dc34949d26c6 None None] Validating token access rules against request {{(pid=81409) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:26:36.994534 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a0c115ea-7055-4ba1-afbb-e1b89ba5a70a None None] Validating token access rules against request {{(pid=81408) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:26:36.997077 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-543b79aa-cca6-409d-a775-c7bd7794933c service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd8812c3974f84871bb2d761cf0016257', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dcec0431fb6849499d6aec35a234daa6', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81409) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:26:36.998149 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-543b79aa-cca6-409d-a775-c7bd7794933c service neutron] REQUEST_METHOD: `GET` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:26:36.998636 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-543b79aa-cca6-409d-a775-c7bd7794933c service neutron] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:26:36.998929 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-543b79aa-cca6-409d-a775-c7bd7794933c service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:26:37.001121 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6fc4cf5f-4d66-4213-ab41-45345c4a7676 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8ba823f313144da3b683bacef8fd62e0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81408) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:26:37.002052 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6fc4cf5f-4d66-4213-ab41-45345c4a7676 admin admin] REQUEST_METHOD: `POST` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:26:37.002503 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6fc4cf5f-4d66-4213-ab41-45345c4a7676 admin admin] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:26:37.002917 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6fc4cf5f-4d66-4213-ab41-45345c4a7676 admin admin] PATH_INFO: `/v3/projects` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:26:37.004860 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6fc4cf5f-4d66-4213-ab41-45345c4a7676 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:26:37.008286 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-543b79aa-cca6-409d-a775-c7bd7794933c service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:26:37.009125 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6fc4cf5f-4d66-4213-ab41-45345c4a7676 admin admin] RBAC: Authorization granted {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:26:37.010082 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-543b79aa-cca6-409d-a775-c7bd7794933c service neutron] RBAC: Authorization granted {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:26:37.021561 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 1019/2007] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Oct 13 04:26:36 2022] GET /identity/v3/auth/tokens => generated 3121 bytes in 45 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Oct 13 04:26:37.072651 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 989/2008] 10.222.0.27 () {60 vars in 1153 bytes} [Thu Oct 13 04:26:36 2022] POST /identity/v3/projects => generated 405 bytes in 163 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 13 04:26:37.075670 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-543b79aa-cca6-409d-a775-c7bd7794933c service neutron] Authenticating user token {{(pid=81409) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:26:37.093677 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-543b79aa-cca6-409d-a775-c7bd7794933c service neutron] Validating token access rules against request {{(pid=81409) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:26:37.162462 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf9a021a-4618-4c4a-b643-d1a8e30547a5 None None] REQUEST_METHOD: `GET` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:26:37.163097 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf9a021a-4618-4c4a-b643-d1a8e30547a5 None None] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:26:37.163488 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf9a021a-4618-4c4a-b643-d1a8e30547a5 None None] PATH_INFO: `/` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:26:37.165449 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 990/2009] 10.222.0.27 () {54 vars in 921 bytes} [Thu Oct 13 04:26:37 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Oct 13 04:26:37.182416 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2bc7722b-4133-4236-a790-20903510e973 None None] REQUEST_METHOD: `POST` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:26:37.182696 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2bc7722b-4133-4236-a790-20903510e973 None None] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:26:37.182971 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2bc7722b-4133-4236-a790-20903510e973 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:26:37.213383 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.auth.core [None req-2bc7722b-4133-4236-a790-20903510e973 None None] MFA Rules not processed for user `cb740ed7f7684b079e45f887be83f795`. Rule list: `[]` (Enabled: `True`). {{(pid=81408) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 13 04:26:37.214790 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.fernet_utils [None req-2bc7722b-4133-4236-a790-20903510e973 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81408) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:26:37.246992 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4701d1da-3084-44f2-868a-39da05a8d482 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8ba823f313144da3b683bacef8fd62e0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81409) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:26:37.247625 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4701d1da-3084-44f2-868a-39da05a8d482 admin admin] REQUEST_METHOD: `POST` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:26:37.247919 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4701d1da-3084-44f2-868a-39da05a8d482 admin admin] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:26:37.248174 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4701d1da-3084-44f2-868a-39da05a8d482 admin admin] PATH_INFO: `/v3/users` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:26:37.250949 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4701d1da-3084-44f2-868a-39da05a8d482 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:26:37.252664 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4701d1da-3084-44f2-868a-39da05a8d482 admin admin] RBAC: Authorization granted {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:26:37.279364 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG dbcounter [-] [81408] Writing DB stats keystone:INSERT=3,keystone:SELECT=97 {{(pid=81408) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:26:37.288076 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 1020/2010] 10.222.0.27 () {60 vars in 1144 bytes} [Thu Oct 13 04:26:37 2022] POST /identity/v3/users => generated 309 bytes in 213 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 13 04:26:37.291898 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4701d1da-3084-44f2-868a-39da05a8d482 admin admin] Authenticating user token {{(pid=81409) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:26:37.303825 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4701d1da-3084-44f2-868a-39da05a8d482 admin admin] Validating token access rules against request {{(pid=81409) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:26:37.305637 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 991/2011] 10.222.0.27 () {58 vars in 1018 bytes} [Thu Oct 13 04:26:37 2022] POST /identity/v3/auth/tokens => generated 3372 bytes in 124 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 13 04:26:37.309227 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-378e9d62-4306-46c6-b4a4-a2681ca75cab admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8ba823f313144da3b683bacef8fd62e0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81409) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:26:37.310037 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-378e9d62-4306-46c6-b4a4-a2681ca75cab admin admin] REQUEST_METHOD: `GET` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:26:37.310356 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-378e9d62-4306-46c6-b4a4-a2681ca75cab admin admin] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:26:37.310631 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-378e9d62-4306-46c6-b4a4-a2681ca75cab admin admin] PATH_INFO: `/v3/roles` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:26:37.311426 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-378e9d62-4306-46c6-b4a4-a2681ca75cab admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:26:37.312997 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-378e9d62-4306-46c6-b4a4-a2681ca75cab admin admin] RBAC: Authorization granted {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:26:37.317903 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2bc7722b-4133-4236-a790-20903510e973 None None] Authenticating user token {{(pid=81408) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:26:37.321173 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.fernet_utils [None req-2bc7722b-4133-4236-a790-20903510e973 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81408) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:26:37.326188 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 1021/2012] 10.222.0.27 () {58 vars in 1122 bytes} [Thu Oct 13 04:26:37 2022] GET /identity/v3/roles => generated 1420 bytes in 35 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Oct 13 04:26:37.330118 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-378e9d62-4306-46c6-b4a4-a2681ca75cab admin admin] Authenticating user token {{(pid=81409) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:26:37.342134 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-378e9d62-4306-46c6-b4a4-a2681ca75cab admin admin] Validating token access rules against request {{(pid=81409) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:26:37.347371 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3eaf8a75-6e3e-4748-b754-62e7924a51c9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8ba823f313144da3b683bacef8fd62e0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81409) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:26:37.348236 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3eaf8a75-6e3e-4748-b754-62e7924a51c9 admin admin] REQUEST_METHOD: `PUT` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:26:37.348505 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3eaf8a75-6e3e-4748-b754-62e7924a51c9 admin admin] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:26:37.348931 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3eaf8a75-6e3e-4748-b754-62e7924a51c9 admin admin] PATH_INFO: `/v3/projects/9c71282799414a6ca819211d84bbd8b4/users/91739ce8029347bdae0fa504818ce0e3/roles/ef55fbd94e1040be83695754075a7ed5` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:26:37.368500 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3eaf8a75-6e3e-4748-b754-62e7924a51c9 admin admin] RBAC: Authorizing `identity:create_grant(project_id=9c71282799414a6ca819211d84bbd8b4, user_id=91739ce8029347bdae0fa504818ce0e3, role_id=ef55fbd94e1040be83695754075a7ed5)` {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:26:37.373655 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3eaf8a75-6e3e-4748-b754-62e7924a51c9 admin admin] RBAC: Authorization granted {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:26:37.388289 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2bc7722b-4133-4236-a790-20903510e973 None None] Validating token access rules against request {{(pid=81408) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:26:37.390557 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 1022/2013] 10.222.0.27 () {60 vars in 1483 bytes} [Thu Oct 13 04:26:37 2022] PUT /identity/v3/projects/9c71282799414a6ca819211d84bbd8b4/users/91739ce8029347bdae0fa504818ce0e3/roles/ef55fbd94e1040be83695754075a7ed5 => generated 0 bytes in 61 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 13 04:26:37.395770 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3eaf8a75-6e3e-4748-b754-62e7924a51c9 admin admin] Authenticating user token {{(pid=81409) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:26:37.447318 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eee75c83-f9b5-4b02-8ed3-7d8e34765d58 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd8812c3974f84871bb2d761cf0016257', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dcec0431fb6849499d6aec35a234daa6', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81408) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:26:37.447935 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eee75c83-f9b5-4b02-8ed3-7d8e34765d58 service neutron] REQUEST_METHOD: `GET` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:26:37.448348 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eee75c83-f9b5-4b02-8ed3-7d8e34765d58 service neutron] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:26:37.448628 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eee75c83-f9b5-4b02-8ed3-7d8e34765d58 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:26:37.483290 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3eaf8a75-6e3e-4748-b754-62e7924a51c9 admin admin] Validating token access rules against request {{(pid=81409) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:26:37.489731 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ae3e9830-cdf4-4ba9-9495-cc1603b8b4ef admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8ba823f313144da3b683bacef8fd62e0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81409) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:26:37.490426 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae3e9830-cdf4-4ba9-9495-cc1603b8b4ef admin admin] REQUEST_METHOD: `GET` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:26:37.490705 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae3e9830-cdf4-4ba9-9495-cc1603b8b4ef admin admin] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:26:37.490975 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae3e9830-cdf4-4ba9-9495-cc1603b8b4ef admin admin] PATH_INFO: `/v3/roles` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:26:37.491670 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ae3e9830-cdf4-4ba9-9495-cc1603b8b4ef admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:26:37.493341 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ae3e9830-cdf4-4ba9-9495-cc1603b8b4ef admin admin] RBAC: Authorization granted {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:26:37.506290 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 1023/2014] 10.222.0.27 () {58 vars in 1122 bytes} [Thu Oct 13 04:26:37 2022] GET /identity/v3/roles => generated 1420 bytes in 110 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Oct 13 04:26:37.508489 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ae3e9830-cdf4-4ba9-9495-cc1603b8b4ef admin admin] Authenticating user token {{(pid=81409) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:26:37.519928 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ae3e9830-cdf4-4ba9-9495-cc1603b8b4ef admin admin] Validating token access rules against request {{(pid=81409) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:26:37.526120 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8572416c-d23b-4000-97b4-eac60e30a72f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8ba823f313144da3b683bacef8fd62e0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81409) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:26:37.527012 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8572416c-d23b-4000-97b4-eac60e30a72f admin admin] REQUEST_METHOD: `PUT` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:26:37.527492 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8572416c-d23b-4000-97b4-eac60e30a72f admin admin] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:26:37.527927 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8572416c-d23b-4000-97b4-eac60e30a72f admin admin] PATH_INFO: `/v3/projects/9c71282799414a6ca819211d84bbd8b4/users/91739ce8029347bdae0fa504818ce0e3/roles/ef55fbd94e1040be83695754075a7ed5` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:26:37.530363 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8572416c-d23b-4000-97b4-eac60e30a72f admin admin] RBAC: Authorizing `identity:create_grant(project_id=9c71282799414a6ca819211d84bbd8b4, user_id=91739ce8029347bdae0fa504818ce0e3, role_id=ef55fbd94e1040be83695754075a7ed5)` {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:26:37.533259 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8572416c-d23b-4000-97b4-eac60e30a72f admin admin] RBAC: Authorization granted {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:26:37.536281 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eee75c83-f9b5-4b02-8ed3-7d8e34765d58 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:26:37.537760 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eee75c83-f9b5-4b02-8ed3-7d8e34765d58 service neutron] RBAC: Authorization granted {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:26:37.545629 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 1024/2015] 10.222.0.27 () {60 vars in 1483 bytes} [Thu Oct 13 04:26:37 2022] PUT /identity/v3/projects/9c71282799414a6ca819211d84bbd8b4/users/91739ce8029347bdae0fa504818ce0e3/roles/ef55fbd94e1040be83695754075a7ed5 => generated 0 bytes in 38 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 13 04:26:37.555871 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 992/2016] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Oct 13 04:26:37 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 233 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Oct 13 04:26:37.556537 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84681335-d5b1-470b-99d2-12ef2fca5db5 None None] REQUEST_METHOD: `POST` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:26:37.556824 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84681335-d5b1-470b-99d2-12ef2fca5db5 None None] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:26:37.557099 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84681335-d5b1-470b-99d2-12ef2fca5db5 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:26:37.580835 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG dbcounter [-] [81409] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=81409) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:26:37.594160 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.auth.core [None req-84681335-d5b1-470b-99d2-12ef2fca5db5 None None] MFA Rules not processed for user `91739ce8029347bdae0fa504818ce0e3`. Rule list: `[]` (Enabled: `True`). {{(pid=81409) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 13 04:26:37.599210 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.fernet_utils [None req-84681335-d5b1-470b-99d2-12ef2fca5db5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81409) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:26:37.719603 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 1025/2017] 10.222.0.27 () {58 vars in 958 bytes} [Thu Oct 13 04:26:37 2022] POST /identity/v3/auth/tokens => generated 3349 bytes in 164 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 13 04:26:37.732702 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eee75c83-f9b5-4b02-8ed3-7d8e34765d58 service neutron] Authenticating user token {{(pid=81408) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:26:37.734257 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.fernet_utils [None req-eee75c83-f9b5-4b02-8ed3-7d8e34765d58 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81408) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:26:37.861470 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG dbcounter [-] [81408] Writing DB stats keystone:SELECT=100 {{(pid=81408) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:26:37.877615 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eee75c83-f9b5-4b02-8ed3-7d8e34765d58 service neutron] Validating token access rules against request {{(pid=81408) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:26:37.946670 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7932af7a-fd35-4f72-8718-100d77afce10 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6141b8a6c3d54e179d6879bb019f4d12', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dcec0431fb6849499d6aec35a234daa6', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81408) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:26:37.948253 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7932af7a-fd35-4f72-8718-100d77afce10 service cinder] REQUEST_METHOD: `GET` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:26:37.948705 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7932af7a-fd35-4f72-8718-100d77afce10 service cinder] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:26:37.949108 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7932af7a-fd35-4f72-8718-100d77afce10 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:26:37.965847 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7932af7a-fd35-4f72-8718-100d77afce10 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:26:37.967802 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7932af7a-fd35-4f72-8718-100d77afce10 service cinder] RBAC: Authorization granted {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:26:37.993757 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 993/2018] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Oct 13 04:26:37 2022] GET /identity/v3/auth/tokens => generated 3349 bytes in 260 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 13 04:26:38.221634 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-84681335-d5b1-470b-99d2-12ef2fca5db5 None None] Authenticating user token {{(pid=81409) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:26:38.322482 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-84681335-d5b1-470b-99d2-12ef2fca5db5 None None] Validating token access rules against request {{(pid=81409) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:26:38.332907 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a2dc5292-7149-46b2-a19c-05bf1e1c92e3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8ba823f313144da3b683bacef8fd62e0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81409) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:26:38.335447 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2dc5292-7149-46b2-a19c-05bf1e1c92e3 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:26:38.335748 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2dc5292-7149-46b2-a19c-05bf1e1c92e3 admin admin] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:26:38.340432 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2dc5292-7149-46b2-a19c-05bf1e1c92e3 admin admin] PATH_INFO: `/v3/users/9fdfe96132ff4c50b12930134543ccbc` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:26:38.342184 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a2dc5292-7149-46b2-a19c-05bf1e1c92e3 admin admin] RBAC: Authorizing `identity:delete_user(user_id=9fdfe96132ff4c50b12930134543ccbc)` {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:26:38.344916 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a2dc5292-7149-46b2-a19c-05bf1e1c92e3 admin admin] RBAC: Authorization granted {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:26:38.446055 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.notifications [None req-a2dc5292-7149-46b2-a19c-05bf1e1c92e3 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '9fdfe96132ff4c50b12930134543ccbc'} {{(pid=81409) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 13 04:26:38.448021 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.notifications [None req-a2dc5292-7149-46b2-a19c-05bf1e1c92e3 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '9fdfe96132ff4c50b12930134543ccbc'} {{(pid=81409) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 13 04:26:38.467522 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.notifications [None req-a2dc5292-7149-46b2-a19c-05bf1e1c92e3 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '9fdfe96132ff4c50b12930134543ccbc'} {{(pid=81409) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 13 04:26:38.469737 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.notifications [None req-a2dc5292-7149-46b2-a19c-05bf1e1c92e3 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '9fdfe96132ff4c50b12930134543ccbc'} {{(pid=81409) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 13 04:26:38.488964 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.notifications [None req-a2dc5292-7149-46b2-a19c-05bf1e1c92e3 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '9fdfe96132ff4c50b12930134543ccbc'} {{(pid=81409) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 13 04:26:38.528266 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 1026/2019] 10.222.0.27 () {58 vars in 1224 bytes} [Thu Oct 13 04:26:38 2022] DELETE /identity/v3/users/9fdfe96132ff4c50b12930134543ccbc => generated 0 bytes in 306 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 13 04:26:38.532837 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d2b75a2-4e17-4f15-90e2-06995d0a71a1 None None] REQUEST_METHOD: `POST` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:26:38.533355 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d2b75a2-4e17-4f15-90e2-06995d0a71a1 None None] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:26:38.535987 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d2b75a2-4e17-4f15-90e2-06995d0a71a1 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:26:38.594010 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.auth.core [None req-8d2b75a2-4e17-4f15-90e2-06995d0a71a1 None None] MFA Rules not processed for user `0e2d979563d6421995451a0e86bf3022`. Rule list: `[]` (Enabled: `True`). {{(pid=81408) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 13 04:26:38.595312 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.fernet_utils [None req-8d2b75a2-4e17-4f15-90e2-06995d0a71a1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81408) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:26:38.713797 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 994/2020] 10.222.0.27 () {58 vars in 958 bytes} [Thu Oct 13 04:26:38 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 182 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 13 04:26:38.732163 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a2dc5292-7149-46b2-a19c-05bf1e1c92e3 admin admin] Authenticating user token {{(pid=81409) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:26:38.735458 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.fernet_utils [None req-a2dc5292-7149-46b2-a19c-05bf1e1c92e3 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81409) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:26:38.758138 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG dbcounter [-] [81409] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81409) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:26:38.824144 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a2dc5292-7149-46b2-a19c-05bf1e1c92e3 admin admin] Validating token access rules against request {{(pid=81409) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:26:38.828121 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-18c62da3-83e9-4d98-8437-a9d6bd8d998e service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd8812c3974f84871bb2d761cf0016257', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dcec0431fb6849499d6aec35a234daa6', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81409) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:26:38.828722 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18c62da3-83e9-4d98-8437-a9d6bd8d998e service neutron] REQUEST_METHOD: `GET` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:26:38.828992 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18c62da3-83e9-4d98-8437-a9d6bd8d998e service neutron] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:26:38.829254 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18c62da3-83e9-4d98-8437-a9d6bd8d998e service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:26:38.840455 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-18c62da3-83e9-4d98-8437-a9d6bd8d998e service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:26:38.842015 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-18c62da3-83e9-4d98-8437-a9d6bd8d998e service neutron] RBAC: Authorization granted {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:26:38.853647 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 1027/2021] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Oct 13 04:26:38 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 125 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 13 04:26:39.402259 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8d2b75a2-4e17-4f15-90e2-06995d0a71a1 None None] Authenticating user token {{(pid=81408) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:26:39.404787 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.fernet_utils [None req-8d2b75a2-4e17-4f15-90e2-06995d0a71a1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81408) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:26:39.462019 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8d2b75a2-4e17-4f15-90e2-06995d0a71a1 None None] Validating token access rules against request {{(pid=81408) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:26:39.474287 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-071d96ed-0f23-4c81-9d57-1e42b18be3b8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8ba823f313144da3b683bacef8fd62e0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81408) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:26:39.474948 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-071d96ed-0f23-4c81-9d57-1e42b18be3b8 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:26:39.475296 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-071d96ed-0f23-4c81-9d57-1e42b18be3b8 admin admin] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:26:39.475572 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-071d96ed-0f23-4c81-9d57-1e42b18be3b8 admin admin] PATH_INFO: `/v3/projects/d5f35c396a4c4d6fb8980583f91f80e5` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:26:39.476953 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-071d96ed-0f23-4c81-9d57-1e42b18be3b8 admin admin] RBAC: Authorizing `identity:delete_project(project_id=d5f35c396a4c4d6fb8980583f91f80e5)` {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:26:39.480009 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-071d96ed-0f23-4c81-9d57-1e42b18be3b8 admin admin] RBAC: Authorization granted {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:26:39.601640 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.notifications [None req-071d96ed-0f23-4c81-9d57-1e42b18be3b8 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'd5f35c396a4c4d6fb8980583f91f80e5', 'request_id': 'req-071d96ed-0f23-4c81-9d57-1e42b18be3b8'} {{(pid=81408) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 13 04:26:39.617726 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.notifications [None req-071d96ed-0f23-4c81-9d57-1e42b18be3b8 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'd5f35c396a4c4d6fb8980583f91f80e5', 'request_id': 'req-071d96ed-0f23-4c81-9d57-1e42b18be3b8'} {{(pid=81408) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 13 04:26:39.629845 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.notifications [None req-071d96ed-0f23-4c81-9d57-1e42b18be3b8 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'd5f35c396a4c4d6fb8980583f91f80e5', 'request_id': 'req-071d96ed-0f23-4c81-9d57-1e42b18be3b8'} {{(pid=81408) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 13 04:26:39.649234 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.notifications [None req-071d96ed-0f23-4c81-9d57-1e42b18be3b8 admin admin] The token cache is being invalidate because project d5f35c396a4c4d6fb8980583f91f80e5 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81408) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 13 04:26:39.651169 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.notifications [None req-071d96ed-0f23-4c81-9d57-1e42b18be3b8 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81408) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 13 04:26:39.652758 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.notifications [None req-071d96ed-0f23-4c81-9d57-1e42b18be3b8 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81408) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 13 04:26:39.657189 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 995/2022] 10.222.0.27 () {58 vars in 1233 bytes} [Thu Oct 13 04:26:39 2022] DELETE /identity/v3/projects/d5f35c396a4c4d6fb8980583f91f80e5 => generated 0 bytes in 255 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 13 04:26:39.662407 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-18c62da3-83e9-4d98-8437-a9d6bd8d998e service neutron] Authenticating user token {{(pid=81409) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:26:39.678970 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-18c62da3-83e9-4d98-8437-a9d6bd8d998e service neutron] Validating token access rules against request {{(pid=81409) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:26:39.751583 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-42933393-19ea-4252-93e2-7ceb4e0f1575 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8ba823f313144da3b683bacef8fd62e0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81409) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:26:39.752224 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42933393-19ea-4252-93e2-7ceb4e0f1575 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:26:39.752695 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42933393-19ea-4252-93e2-7ceb4e0f1575 admin admin] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:26:39.753756 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42933393-19ea-4252-93e2-7ceb4e0f1575 admin admin] PATH_INFO: `/v3/users/91739ce8029347bdae0fa504818ce0e3` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:26:39.755581 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-42933393-19ea-4252-93e2-7ceb4e0f1575 admin admin] RBAC: Authorizing `identity:delete_user(user_id=91739ce8029347bdae0fa504818ce0e3)` {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:26:39.764163 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-42933393-19ea-4252-93e2-7ceb4e0f1575 admin admin] RBAC: Authorization granted {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:26:39.884927 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.notifications [None req-42933393-19ea-4252-93e2-7ceb4e0f1575 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '91739ce8029347bdae0fa504818ce0e3'} {{(pid=81409) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 13 04:26:39.885702 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.notifications [None req-42933393-19ea-4252-93e2-7ceb4e0f1575 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '91739ce8029347bdae0fa504818ce0e3'} {{(pid=81409) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 13 04:26:39.895067 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.notifications [None req-42933393-19ea-4252-93e2-7ceb4e0f1575 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '91739ce8029347bdae0fa504818ce0e3'} {{(pid=81409) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 13 04:26:39.896423 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.notifications [None req-42933393-19ea-4252-93e2-7ceb4e0f1575 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '91739ce8029347bdae0fa504818ce0e3'} {{(pid=81409) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 13 04:26:39.905045 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.notifications [None req-42933393-19ea-4252-93e2-7ceb4e0f1575 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '91739ce8029347bdae0fa504818ce0e3'} {{(pid=81409) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 13 04:26:39.929443 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 1028/2023] 10.222.0.27 () {58 vars in 1224 bytes} [Thu Oct 13 04:26:39 2022] DELETE /identity/v3/users/91739ce8029347bdae0fa504818ce0e3 => generated 0 bytes in 268 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 13 04:26:40.680321 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a7c60f3-f071-4194-8c1f-6d74ac5244e8 None None] REQUEST_METHOD: `POST` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:26:40.680928 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a7c60f3-f071-4194-8c1f-6d74ac5244e8 None None] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:26:40.680928 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a7c60f3-f071-4194-8c1f-6d74ac5244e8 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:26:40.700181 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.auth.core [None req-1a7c60f3-f071-4194-8c1f-6d74ac5244e8 None None] MFA Rules not processed for user `1f6cf725908041ccaceb1ffa694ad161`. Rule list: `[]` (Enabled: `True`). {{(pid=81408) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 13 04:26:40.701393 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.fernet_utils [None req-1a7c60f3-f071-4194-8c1f-6d74ac5244e8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81408) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:26:40.729213 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG dbcounter [-] [81408] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=81408) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:26:40.758561 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-42933393-19ea-4252-93e2-7ceb4e0f1575 admin admin] Authenticating user token {{(pid=81409) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:26:40.761306 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.fernet_utils [None req-42933393-19ea-4252-93e2-7ceb4e0f1575 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81409) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:26:40.802326 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 996/2024] 10.222.0.27 () {58 vars in 958 bytes} [Thu Oct 13 04:26:40 2022] POST /identity/v3/auth/tokens => generated 3254 bytes in 123 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 13 04:26:40.830120 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG dbcounter [-] [81409] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81409) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:26:40.865036 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-42933393-19ea-4252-93e2-7ceb4e0f1575 admin admin] Validating token access rules against request {{(pid=81409) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:26:40.870171 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3b0e5d61-ef62-4a7a-be03-81c0d49a0a3d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8ba823f313144da3b683bacef8fd62e0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81409) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:26:40.871296 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b0e5d61-ef62-4a7a-be03-81c0d49a0a3d admin admin] REQUEST_METHOD: `DELETE` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:26:40.871786 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b0e5d61-ef62-4a7a-be03-81c0d49a0a3d admin admin] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:26:40.872350 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b0e5d61-ef62-4a7a-be03-81c0d49a0a3d admin admin] PATH_INFO: `/v3/projects/9c71282799414a6ca819211d84bbd8b4` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:26:40.873868 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3b0e5d61-ef62-4a7a-be03-81c0d49a0a3d admin admin] RBAC: Authorizing `identity:delete_project(project_id=9c71282799414a6ca819211d84bbd8b4)` {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:26:40.876634 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3b0e5d61-ef62-4a7a-be03-81c0d49a0a3d admin admin] RBAC: Authorization granted {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:26:40.936126 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.notifications [None req-3b0e5d61-ef62-4a7a-be03-81c0d49a0a3d admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '9c71282799414a6ca819211d84bbd8b4', 'request_id': 'req-3b0e5d61-ef62-4a7a-be03-81c0d49a0a3d'} {{(pid=81409) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 13 04:26:40.947228 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.notifications [None req-3b0e5d61-ef62-4a7a-be03-81c0d49a0a3d admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '9c71282799414a6ca819211d84bbd8b4', 'request_id': 'req-3b0e5d61-ef62-4a7a-be03-81c0d49a0a3d'} {{(pid=81409) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 13 04:26:40.964089 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.notifications [None req-3b0e5d61-ef62-4a7a-be03-81c0d49a0a3d admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '9c71282799414a6ca819211d84bbd8b4', 'request_id': 'req-3b0e5d61-ef62-4a7a-be03-81c0d49a0a3d'} {{(pid=81409) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 13 04:26:40.976738 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.notifications [None req-3b0e5d61-ef62-4a7a-be03-81c0d49a0a3d admin admin] The token cache is being invalidate because project 9c71282799414a6ca819211d84bbd8b4 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81409) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 13 04:26:40.977025 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.notifications [None req-3b0e5d61-ef62-4a7a-be03-81c0d49a0a3d admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81409) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 13 04:26:40.977684 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.notifications [None req-3b0e5d61-ef62-4a7a-be03-81c0d49a0a3d admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81409) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 13 04:26:40.979324 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 1029/2025] 10.222.0.27 () {58 vars in 1233 bytes} [Thu Oct 13 04:26:40 2022] DELETE /identity/v3/projects/9c71282799414a6ca819211d84bbd8b4 => generated 0 bytes in 221 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 13 04:26:40.984533 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b08bccb-ad1e-4a18-9d06-d1bed3b4edb6 None None] REQUEST_METHOD: `POST` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:26:40.984823 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b08bccb-ad1e-4a18-9d06-d1bed3b4edb6 None None] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:26:40.985096 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b08bccb-ad1e-4a18-9d06-d1bed3b4edb6 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:26:41.006226 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.auth.core [None req-4b08bccb-ad1e-4a18-9d06-d1bed3b4edb6 None None] MFA Rules not processed for user `0e2d979563d6421995451a0e86bf3022`. Rule list: `[]` (Enabled: `True`). {{(pid=81408) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 13 04:26:41.007502 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.fernet_utils [None req-4b08bccb-ad1e-4a18-9d06-d1bed3b4edb6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81408) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:26:41.086361 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 997/2026] 10.222.0.27 () {58 vars in 958 bytes} [Thu Oct 13 04:26:40 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 103 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 13 04:26:41.094167 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44bffdb6-a1f5-45bd-bcc2-069bb4c57044 None None] REQUEST_METHOD: `POST` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:26:41.094704 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44bffdb6-a1f5-45bd-bcc2-069bb4c57044 None None] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:26:41.095756 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44bffdb6-a1f5-45bd-bcc2-069bb4c57044 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:26:41.119922 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.auth.core [None req-44bffdb6-a1f5-45bd-bcc2-069bb4c57044 None None] MFA Rules not processed for user `0e2d979563d6421995451a0e86bf3022`. Rule list: `[]` (Enabled: `True`). {{(pid=81409) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 13 04:26:41.123442 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.fernet_utils [None req-44bffdb6-a1f5-45bd-bcc2-069bb4c57044 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81409) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:26:41.133662 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 1030/2027] 10.222.0.27 () {58 vars in 958 bytes} [Thu Oct 13 04:26:41 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 40 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 13 04:26:41.137740 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4b08bccb-ad1e-4a18-9d06-d1bed3b4edb6 None None] Authenticating user token {{(pid=81408) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:26:41.150345 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4b08bccb-ad1e-4a18-9d06-d1bed3b4edb6 None None] Validating token access rules against request {{(pid=81408) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:26:41.158100 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b9e68090-abfd-452f-b07d-1ce1dd7c0474 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8ba823f313144da3b683bacef8fd62e0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81408) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:26:41.158776 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9e68090-abfd-452f-b07d-1ce1dd7c0474 admin admin] REQUEST_METHOD: `GET` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:26:41.159051 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9e68090-abfd-452f-b07d-1ce1dd7c0474 admin admin] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:26:41.162955 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9e68090-abfd-452f-b07d-1ce1dd7c0474 admin admin] PATH_INFO: `/v3/domains` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:26:41.164077 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b9e68090-abfd-452f-b07d-1ce1dd7c0474 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81408) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 13 04:26:41.164543 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b9e68090-abfd-452f-b07d-1ce1dd7c0474 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:26:41.166367 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b9e68090-abfd-452f-b07d-1ce1dd7c0474 admin admin] RBAC: Authorization granted {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:26:41.186227 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 998/2028] 10.222.0.27 () {58 vars in 1153 bytes} [Thu Oct 13 04:26:41 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 49 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 13 04:26:41.187225 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-44bffdb6-a1f5-45bd-bcc2-069bb4c57044 None None] Authenticating user token {{(pid=81409) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:26:41.199081 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-44bffdb6-a1f5-45bd-bcc2-069bb4c57044 None None] Validating token access rules against request {{(pid=81409) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:26:41.214247 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0f8f2751-e019-4f45-9ea2-f176b1d87c71 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8ba823f313144da3b683bacef8fd62e0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81409) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:26:41.214247 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f8f2751-e019-4f45-9ea2-f176b1d87c71 admin admin] REQUEST_METHOD: `POST` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:26:41.214247 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f8f2751-e019-4f45-9ea2-f176b1d87c71 admin admin] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:26:41.214247 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f8f2751-e019-4f45-9ea2-f176b1d87c71 admin admin] PATH_INFO: `/v3/projects` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:26:41.214716 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0f8f2751-e019-4f45-9ea2-f176b1d87c71 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:26:41.214716 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0f8f2751-e019-4f45-9ea2-f176b1d87c71 admin admin] RBAC: Authorization granted {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:26:41.249228 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 1031/2029] 10.222.0.27 () {60 vars in 1153 bytes} [Thu Oct 13 04:26:41 2022] POST /identity/v3/projects => generated 355 bytes in 63 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 13 04:26:41.254177 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b9e68090-abfd-452f-b07d-1ce1dd7c0474 admin admin] Authenticating user token {{(pid=81408) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:26:41.267222 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b9e68090-abfd-452f-b07d-1ce1dd7c0474 admin admin] Validating token access rules against request {{(pid=81408) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:26:41.332985 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f0b8a8dc-058c-41b9-8547-172bb97af9fd admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8ba823f313144da3b683bacef8fd62e0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81408) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:26:41.333858 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0b8a8dc-058c-41b9-8547-172bb97af9fd admin admin] REQUEST_METHOD: `POST` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:26:41.334325 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0b8a8dc-058c-41b9-8547-172bb97af9fd admin admin] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:26:41.334808 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0b8a8dc-058c-41b9-8547-172bb97af9fd admin admin] PATH_INFO: `/v3/users` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:26:41.335750 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f0b8a8dc-058c-41b9-8547-172bb97af9fd admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:26:41.337912 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f0b8a8dc-058c-41b9-8547-172bb97af9fd admin admin] RBAC: Authorization granted {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:26:41.364604 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 999/2030] 10.222.0.27 () {60 vars in 1144 bytes} [Thu Oct 13 04:26:41 2022] POST /identity/v3/users => generated 278 bytes in 112 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 13 04:26:41.369072 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0f8f2751-e019-4f45-9ea2-f176b1d87c71 admin admin] Authenticating user token {{(pid=81409) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:26:41.381311 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0f8f2751-e019-4f45-9ea2-f176b1d87c71 admin admin] Validating token access rules against request {{(pid=81409) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:26:41.386057 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c17a042b-30b5-4714-82e9-18a5b029d283 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8ba823f313144da3b683bacef8fd62e0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81409) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:26:41.386704 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c17a042b-30b5-4714-82e9-18a5b029d283 admin admin] REQUEST_METHOD: `GET` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:26:41.386975 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c17a042b-30b5-4714-82e9-18a5b029d283 admin admin] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:26:41.387229 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c17a042b-30b5-4714-82e9-18a5b029d283 admin admin] PATH_INFO: `/v3/roles` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:26:41.387866 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c17a042b-30b5-4714-82e9-18a5b029d283 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:26:41.389692 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c17a042b-30b5-4714-82e9-18a5b029d283 admin admin] RBAC: Authorization granted {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:26:41.408199 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 1032/2031] 10.222.0.27 () {58 vars in 1122 bytes} [Thu Oct 13 04:26:41 2022] GET /identity/v3/roles => generated 1420 bytes in 39 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 13 04:26:41.412742 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f0b8a8dc-058c-41b9-8547-172bb97af9fd admin admin] Authenticating user token {{(pid=81408) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:26:41.435153 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f0b8a8dc-058c-41b9-8547-172bb97af9fd admin admin] Validating token access rules against request {{(pid=81408) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:26:41.440054 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ac7e5b79-a14d-4430-bde4-87a5cc0871ec admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8ba823f313144da3b683bacef8fd62e0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81408) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:26:41.441247 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac7e5b79-a14d-4430-bde4-87a5cc0871ec admin admin] REQUEST_METHOD: `GET` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:26:41.441774 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac7e5b79-a14d-4430-bde4-87a5cc0871ec admin admin] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:26:41.442117 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac7e5b79-a14d-4430-bde4-87a5cc0871ec admin admin] PATH_INFO: `/v3/roles` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:26:41.443382 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ac7e5b79-a14d-4430-bde4-87a5cc0871ec admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:26:41.445298 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ac7e5b79-a14d-4430-bde4-87a5cc0871ec admin admin] RBAC: Authorization granted {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:26:41.458629 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 1000/2032] 10.222.0.27 () {58 vars in 1122 bytes} [Thu Oct 13 04:26:41 2022] GET /identity/v3/roles => generated 1420 bytes in 46 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 13 04:26:41.463491 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c17a042b-30b5-4714-82e9-18a5b029d283 admin admin] Authenticating user token {{(pid=81409) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:26:41.475165 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c17a042b-30b5-4714-82e9-18a5b029d283 admin admin] Validating token access rules against request {{(pid=81409) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:26:41.480298 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-19396a3c-805c-47c2-97b6-0415cfd71cf8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8ba823f313144da3b683bacef8fd62e0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81409) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:26:41.480983 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19396a3c-805c-47c2-97b6-0415cfd71cf8 admin admin] REQUEST_METHOD: `PUT` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:26:41.481566 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19396a3c-805c-47c2-97b6-0415cfd71cf8 admin admin] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:26:41.481922 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19396a3c-805c-47c2-97b6-0415cfd71cf8 admin admin] PATH_INFO: `/v3/projects/372ec495c99743798f45d2532e65ca23/users/28799988fff84a0a89f1fecdb7a3e898/roles/cc56c4ae13a7424ea771c15da4b153c1` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:26:41.501197 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-19396a3c-805c-47c2-97b6-0415cfd71cf8 admin admin] RBAC: Authorizing `identity:create_grant(project_id=372ec495c99743798f45d2532e65ca23, user_id=28799988fff84a0a89f1fecdb7a3e898, role_id=cc56c4ae13a7424ea771c15da4b153c1)` {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:26:41.506506 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-19396a3c-805c-47c2-97b6-0415cfd71cf8 admin admin] RBAC: Authorization granted {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:26:41.521874 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 1033/2033] 10.222.0.27 () {60 vars in 1483 bytes} [Thu Oct 13 04:26:41 2022] PUT /identity/v3/projects/372ec495c99743798f45d2532e65ca23/users/28799988fff84a0a89f1fecdb7a3e898/roles/cc56c4ae13a7424ea771c15da4b153c1 => generated 0 bytes in 59 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 13 04:26:41.534986 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4faa91ed-25ab-4ef3-ac16-27342e60a0ae None None] REQUEST_METHOD: `POST` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:26:41.535598 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4faa91ed-25ab-4ef3-ac16-27342e60a0ae None None] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:26:41.536095 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4faa91ed-25ab-4ef3-ac16-27342e60a0ae None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:26:41.581922 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.auth.core [None req-4faa91ed-25ab-4ef3-ac16-27342e60a0ae None None] MFA Rules not processed for user `28799988fff84a0a89f1fecdb7a3e898`. Rule list: `[]` (Enabled: `True`). {{(pid=81408) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 13 04:26:41.583315 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.fernet_utils [None req-4faa91ed-25ab-4ef3-ac16-27342e60a0ae None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81408) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:26:41.612702 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG dbcounter [-] [81408] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81408) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:26:41.666244 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 1001/2034] 10.222.0.27 () {58 vars in 958 bytes} [Thu Oct 13 04:26:41 2022] POST /identity/v3/auth/tokens => generated 3232 bytes in 130 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 13 04:26:41.668286 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-19396a3c-805c-47c2-97b6-0415cfd71cf8 admin admin] Authenticating user token {{(pid=81409) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:26:41.736234 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-19396a3c-805c-47c2-97b6-0415cfd71cf8 admin admin] Validating token access rules against request {{(pid=81409) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:26:41.741470 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a1513b53-3094-457d-bfe8-a13226245168 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8ba823f313144da3b683bacef8fd62e0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81409) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:26:41.742301 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1513b53-3094-457d-bfe8-a13226245168 admin admin] REQUEST_METHOD: `POST` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:26:41.742466 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1513b53-3094-457d-bfe8-a13226245168 admin admin] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:26:41.742647 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1513b53-3094-457d-bfe8-a13226245168 admin admin] PATH_INFO: `/v3/projects` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:26:41.743326 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a1513b53-3094-457d-bfe8-a13226245168 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:26:41.745218 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a1513b53-3094-457d-bfe8-a13226245168 admin admin] RBAC: Authorization granted {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:26:41.750788 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG dbcounter [-] [81409] Writing DB stats keystone:SELECT=90,keystone:DELETE=7,keystone:INSERT=3 {{(pid=81409) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:26:41.777776 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 1034/2035] 10.222.0.27 () {60 vars in 1153 bytes} [Thu Oct 13 04:26:41 2022] POST /identity/v3/projects => generated 357 bytes in 110 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 13 04:26:41.783746 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4faa91ed-25ab-4ef3-ac16-27342e60a0ae None None] Authenticating user token {{(pid=81408) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:26:41.798447 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4faa91ed-25ab-4ef3-ac16-27342e60a0ae None None] Validating token access rules against request {{(pid=81408) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:26:41.868438 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ecbcb927-9d6d-4453-ab9e-9a4b67fa2bc2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8ba823f313144da3b683bacef8fd62e0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81408) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:26:41.869545 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ecbcb927-9d6d-4453-ab9e-9a4b67fa2bc2 admin admin] REQUEST_METHOD: `POST` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:26:41.870003 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ecbcb927-9d6d-4453-ab9e-9a4b67fa2bc2 admin admin] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:26:41.870615 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ecbcb927-9d6d-4453-ab9e-9a4b67fa2bc2 admin admin] PATH_INFO: `/v3/users` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:26:41.871277 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ecbcb927-9d6d-4453-ab9e-9a4b67fa2bc2 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:26:41.873335 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ecbcb927-9d6d-4453-ab9e-9a4b67fa2bc2 admin admin] RBAC: Authorization granted {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:26:41.898724 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 1002/2036] 10.222.0.27 () {60 vars in 1144 bytes} [Thu Oct 13 04:26:41 2022] POST /identity/v3/users => generated 285 bytes in 116 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 13 04:26:41.902860 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a1513b53-3094-457d-bfe8-a13226245168 admin admin] Authenticating user token {{(pid=81409) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:26:41.913855 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a1513b53-3094-457d-bfe8-a13226245168 admin admin] Validating token access rules against request {{(pid=81409) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:26:41.918755 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f1055716-1867-4fe4-8064-dcb54d045083 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8ba823f313144da3b683bacef8fd62e0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81409) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:26:41.919428 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1055716-1867-4fe4-8064-dcb54d045083 admin admin] REQUEST_METHOD: `GET` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:26:41.919775 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1055716-1867-4fe4-8064-dcb54d045083 admin admin] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:26:41.920111 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1055716-1867-4fe4-8064-dcb54d045083 admin admin] PATH_INFO: `/v3/roles` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:26:41.920833 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f1055716-1867-4fe4-8064-dcb54d045083 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:26:41.922526 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f1055716-1867-4fe4-8064-dcb54d045083 admin admin] RBAC: Authorization granted {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:26:41.933211 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 1035/2037] 10.222.0.27 () {58 vars in 1122 bytes} [Thu Oct 13 04:26:41 2022] GET /identity/v3/roles => generated 1420 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 13 04:26:41.937326 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ecbcb927-9d6d-4453-ab9e-9a4b67fa2bc2 admin admin] Authenticating user token {{(pid=81408) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:26:41.947559 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ecbcb927-9d6d-4453-ab9e-9a4b67fa2bc2 admin admin] Validating token access rules against request {{(pid=81408) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:26:41.952240 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f2a1e785-f1ad-4eb6-8f25-fc4358d1912c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8ba823f313144da3b683bacef8fd62e0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81408) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:26:41.953066 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2a1e785-f1ad-4eb6-8f25-fc4358d1912c admin admin] REQUEST_METHOD: `PUT` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:26:41.953349 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2a1e785-f1ad-4eb6-8f25-fc4358d1912c admin admin] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:26:41.953632 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2a1e785-f1ad-4eb6-8f25-fc4358d1912c admin admin] PATH_INFO: `/v3/projects/800f49615f1f4506bfaa79751170a668/users/cf1b18f763e6452fad740651f9a4c396/roles/ef55fbd94e1040be83695754075a7ed5` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:26:41.970025 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f2a1e785-f1ad-4eb6-8f25-fc4358d1912c admin admin] RBAC: Authorizing `identity:create_grant(project_id=800f49615f1f4506bfaa79751170a668, user_id=cf1b18f763e6452fad740651f9a4c396, role_id=ef55fbd94e1040be83695754075a7ed5)` {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:26:41.973563 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f2a1e785-f1ad-4eb6-8f25-fc4358d1912c admin admin] RBAC: Authorization granted {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:26:41.987756 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 1003/2038] 10.222.0.27 () {60 vars in 1483 bytes} [Thu Oct 13 04:26:41 2022] PUT /identity/v3/projects/800f49615f1f4506bfaa79751170a668/users/cf1b18f763e6452fad740651f9a4c396/roles/ef55fbd94e1040be83695754075a7ed5 => generated 0 bytes in 51 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 13 04:26:41.991714 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f1055716-1867-4fe4-8064-dcb54d045083 admin admin] Authenticating user token {{(pid=81409) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:26:42.001731 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f1055716-1867-4fe4-8064-dcb54d045083 admin admin] Validating token access rules against request {{(pid=81409) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:26:42.058025 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cec96b52-4fa1-4398-b9ae-7c00b1f45fdc admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8ba823f313144da3b683bacef8fd62e0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81409) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:26:42.058667 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cec96b52-4fa1-4398-b9ae-7c00b1f45fdc admin admin] REQUEST_METHOD: `GET` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:26:42.058954 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cec96b52-4fa1-4398-b9ae-7c00b1f45fdc admin admin] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:26:42.059217 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cec96b52-4fa1-4398-b9ae-7c00b1f45fdc admin admin] PATH_INFO: `/v3/roles` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:26:42.059884 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cec96b52-4fa1-4398-b9ae-7c00b1f45fdc admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:26:42.061826 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cec96b52-4fa1-4398-b9ae-7c00b1f45fdc admin admin] RBAC: Authorization granted {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:26:42.072448 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 1036/2039] 10.222.0.27 () {58 vars in 1122 bytes} [Thu Oct 13 04:26:41 2022] GET /identity/v3/roles => generated 1420 bytes in 81 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 13 04:26:42.076835 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f2a1e785-f1ad-4eb6-8f25-fc4358d1912c admin admin] Authenticating user token {{(pid=81408) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:26:42.087865 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f2a1e785-f1ad-4eb6-8f25-fc4358d1912c admin admin] Validating token access rules against request {{(pid=81408) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:26:42.092390 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-055519ac-5562-4097-995a-203f40edb6ea admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8ba823f313144da3b683bacef8fd62e0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81408) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:26:42.093033 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-055519ac-5562-4097-995a-203f40edb6ea admin admin] REQUEST_METHOD: `PUT` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:26:42.093305 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-055519ac-5562-4097-995a-203f40edb6ea admin admin] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:26:42.093573 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-055519ac-5562-4097-995a-203f40edb6ea admin admin] PATH_INFO: `/v3/projects/800f49615f1f4506bfaa79751170a668/users/cf1b18f763e6452fad740651f9a4c396/roles/ef55fbd94e1040be83695754075a7ed5` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:26:42.095508 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-055519ac-5562-4097-995a-203f40edb6ea admin admin] RBAC: Authorizing `identity:create_grant(project_id=800f49615f1f4506bfaa79751170a668, user_id=cf1b18f763e6452fad740651f9a4c396, role_id=ef55fbd94e1040be83695754075a7ed5)` {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:26:42.098284 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-055519ac-5562-4097-995a-203f40edb6ea admin admin] RBAC: Authorization granted {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:26:42.108487 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 1004/2040] 10.222.0.27 () {60 vars in 1483 bytes} [Thu Oct 13 04:26:42 2022] PUT /identity/v3/projects/800f49615f1f4506bfaa79751170a668/users/cf1b18f763e6452fad740651f9a4c396/roles/ef55fbd94e1040be83695754075a7ed5 => generated 0 bytes in 32 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 13 04:26:42.118972 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6edd3dde-86aa-45cb-9c80-664ec641e0e5 None None] REQUEST_METHOD: `POST` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:26:42.119360 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6edd3dde-86aa-45cb-9c80-664ec641e0e5 None None] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:26:42.119676 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6edd3dde-86aa-45cb-9c80-664ec641e0e5 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:26:42.149632 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.auth.core [None req-6edd3dde-86aa-45cb-9c80-664ec641e0e5 None None] MFA Rules not processed for user `cf1b18f763e6452fad740651f9a4c396`. Rule list: `[]` (Enabled: `True`). {{(pid=81409) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 13 04:26:42.151004 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.fernet_utils [None req-6edd3dde-86aa-45cb-9c80-664ec641e0e5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81409) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:26:42.219865 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 1037/2041] 10.222.0.27 () {58 vars in 958 bytes} [Thu Oct 13 04:26:42 2022] POST /identity/v3/auth/tokens => generated 3301 bytes in 102 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 13 04:26:42.231450 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-055519ac-5562-4097-995a-203f40edb6ea admin admin] Authenticating user token {{(pid=81408) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:26:42.233272 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.fernet_utils [None req-055519ac-5562-4097-995a-203f40edb6ea admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81408) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:26:42.293234 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-055519ac-5562-4097-995a-203f40edb6ea admin admin] Validating token access rules against request {{(pid=81408) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:26:42.297127 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-74a5479b-d526-428d-b24d-31e6eb5e896b service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6141b8a6c3d54e179d6879bb019f4d12', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dcec0431fb6849499d6aec35a234daa6', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81408) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:26:42.297742 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74a5479b-d526-428d-b24d-31e6eb5e896b service cinder] REQUEST_METHOD: `GET` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:26:42.298017 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74a5479b-d526-428d-b24d-31e6eb5e896b service cinder] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:26:42.298319 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74a5479b-d526-428d-b24d-31e6eb5e896b service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:26:42.307867 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-74a5479b-d526-428d-b24d-31e6eb5e896b service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:26:42.309299 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-74a5479b-d526-428d-b24d-31e6eb5e896b service cinder] RBAC: Authorization granted {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:26:42.319652 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 1005/2042] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Oct 13 04:26:42 2022] GET /identity/v3/auth/tokens => generated 3301 bytes in 89 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 13 04:26:42.397523 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6edd3dde-86aa-45cb-9c80-664ec641e0e5 None None] Authenticating user token {{(pid=81409) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:26:42.407673 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6edd3dde-86aa-45cb-9c80-664ec641e0e5 None None] Validating token access rules against request {{(pid=81409) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:26:42.411407 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-74c123c5-a111-4e61-9655-f47c99faa27c service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6141b8a6c3d54e179d6879bb019f4d12', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dcec0431fb6849499d6aec35a234daa6', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81409) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:26:42.412016 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74c123c5-a111-4e61-9655-f47c99faa27c service cinder] REQUEST_METHOD: `GET` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:26:42.412293 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74c123c5-a111-4e61-9655-f47c99faa27c service cinder] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:26:42.412566 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74c123c5-a111-4e61-9655-f47c99faa27c service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:26:42.467475 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG dbcounter [-] [81409] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81409) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:26:42.470060 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-74c123c5-a111-4e61-9655-f47c99faa27c service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:26:42.471573 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-74c123c5-a111-4e61-9655-f47c99faa27c service cinder] RBAC: Authorization granted {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:26:42.483216 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 1038/2043] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Oct 13 04:26:42 2022] GET /identity/v3/auth/tokens => generated 3232 bytes in 87 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 13 04:26:46.553155 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-74a5479b-d526-428d-b24d-31e6eb5e896b service cinder] Authenticating user token {{(pid=81408) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:26:46.554738 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.fernet_utils [None req-74a5479b-d526-428d-b24d-31e6eb5e896b service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81408) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:26:46.587493 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG dbcounter [-] [81408] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=81408) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:26:46.646677 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-74a5479b-d526-428d-b24d-31e6eb5e896b service cinder] Validating token access rules against request {{(pid=81408) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:26:46.652138 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f1e75f46-12e5-47d6-877c-58b5bd8b0fe9 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cb740ed7f7684b079e45f887be83f795', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dcec0431fb6849499d6aec35a234daa6', 'project_domain_id': 'default', 'roles': ['ResellerAdmin', 'admin', 'reader', 'service', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81408) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:26:46.652730 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1e75f46-12e5-47d6-877c-58b5bd8b0fe9 service nova] REQUEST_METHOD: `GET` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:26:46.653012 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1e75f46-12e5-47d6-877c-58b5bd8b0fe9 service nova] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:26:46.653287 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1e75f46-12e5-47d6-877c-58b5bd8b0fe9 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:26:46.655047 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.fernet_utils [None req-f1e75f46-12e5-47d6-877c-58b5bd8b0fe9 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81408) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:26:46.667831 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f1e75f46-12e5-47d6-877c-58b5bd8b0fe9 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:26:46.669407 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f1e75f46-12e5-47d6-877c-58b5bd8b0fe9 service nova] RBAC: Authorization granted {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:26:46.680534 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 1006/2044] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Oct 13 04:26:46 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 128 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Oct 13 04:26:51.805515 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-74c123c5-a111-4e61-9655-f47c99faa27c service cinder] Authenticating user token {{(pid=81409) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:26:51.818154 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-74c123c5-a111-4e61-9655-f47c99faa27c service cinder] Validating token access rules against request {{(pid=81409) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:26:51.823869 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9a67640b-b8b7-40da-a0b5-034a7c0cd962 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cb740ed7f7684b079e45f887be83f795', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dcec0431fb6849499d6aec35a234daa6', 'project_domain_id': 'default', 'roles': ['ResellerAdmin', 'admin', 'reader', 'service', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81409) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:26:51.824487 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a67640b-b8b7-40da-a0b5-034a7c0cd962 service nova] REQUEST_METHOD: `GET` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:26:51.824777 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a67640b-b8b7-40da-a0b5-034a7c0cd962 service nova] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:26:51.825019 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a67640b-b8b7-40da-a0b5-034a7c0cd962 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:26:51.826944 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.fernet_utils [None req-9a67640b-b8b7-40da-a0b5-034a7c0cd962 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81409) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:26:51.839942 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9a67640b-b8b7-40da-a0b5-034a7c0cd962 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:26:51.841531 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9a67640b-b8b7-40da-a0b5-034a7c0cd962 service nova] RBAC: Authorization granted {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:26:51.853854 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 1039/2045] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Oct 13 04:26:51 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 49 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 13 04:26:56.676105 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG dbcounter [-] [81408] Writing DB stats keystone:SELECT=24 {{(pid=81408) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:26:58.118279 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f1e75f46-12e5-47d6-877c-58b5bd8b0fe9 service nova] Authenticating user token {{(pid=81408) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:26:58.183431 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f1e75f46-12e5-47d6-877c-58b5bd8b0fe9 service nova] Validating token access rules against request {{(pid=81408) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:26:58.188129 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6d17ab6f-ae28-4af6-818b-bfaabd219dbf admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8ba823f313144da3b683bacef8fd62e0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81408) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:26:58.188853 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d17ab6f-ae28-4af6-818b-bfaabd219dbf admin admin] REQUEST_METHOD: `DELETE` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:26:58.189202 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d17ab6f-ae28-4af6-818b-bfaabd219dbf admin admin] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:26:58.189475 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d17ab6f-ae28-4af6-818b-bfaabd219dbf admin admin] PATH_INFO: `/v3/users/28799988fff84a0a89f1fecdb7a3e898` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:26:58.190757 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6d17ab6f-ae28-4af6-818b-bfaabd219dbf admin admin] RBAC: Authorizing `identity:delete_user(user_id=28799988fff84a0a89f1fecdb7a3e898)` {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:26:58.192703 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6d17ab6f-ae28-4af6-818b-bfaabd219dbf admin admin] RBAC: Authorization granted {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:26:58.241864 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.notifications [None req-6d17ab6f-ae28-4af6-818b-bfaabd219dbf admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '28799988fff84a0a89f1fecdb7a3e898'} {{(pid=81408) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 13 04:26:58.249376 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.notifications [None req-6d17ab6f-ae28-4af6-818b-bfaabd219dbf admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '28799988fff84a0a89f1fecdb7a3e898'} {{(pid=81408) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 13 04:26:58.250076 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.notifications [None req-6d17ab6f-ae28-4af6-818b-bfaabd219dbf admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '28799988fff84a0a89f1fecdb7a3e898'} {{(pid=81408) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 13 04:26:58.250787 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.notifications [None req-6d17ab6f-ae28-4af6-818b-bfaabd219dbf admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '28799988fff84a0a89f1fecdb7a3e898'} {{(pid=81408) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 13 04:26:58.259380 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.notifications [None req-6d17ab6f-ae28-4af6-818b-bfaabd219dbf admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '28799988fff84a0a89f1fecdb7a3e898'} {{(pid=81408) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 13 04:26:58.278404 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 1007/2046] 10.222.0.27 () {58 vars in 1224 bytes} [Thu Oct 13 04:26:58 2022] DELETE /identity/v3/users/28799988fff84a0a89f1fecdb7a3e898 => generated 0 bytes in 161 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 13 04:26:58.282877 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e7a088c-6bba-464a-a560-26688f126d25 None None] REQUEST_METHOD: `POST` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:26:58.283366 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e7a088c-6bba-464a-a560-26688f126d25 None None] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:26:58.283526 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e7a088c-6bba-464a-a560-26688f126d25 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:26:58.302899 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.auth.core [None req-4e7a088c-6bba-464a-a560-26688f126d25 None None] MFA Rules not processed for user `0e2d979563d6421995451a0e86bf3022`. Rule list: `[]` (Enabled: `True`). {{(pid=81409) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 13 04:26:58.304097 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.fernet_utils [None req-4e7a088c-6bba-464a-a560-26688f126d25 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81409) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:26:58.361877 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 1040/2047] 10.222.0.27 () {58 vars in 958 bytes} [Thu Oct 13 04:26:58 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 80 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 13 04:26:58.371578 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6d17ab6f-ae28-4af6-818b-bfaabd219dbf admin admin] Authenticating user token {{(pid=81408) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:26:58.372960 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.fernet_utils [None req-6d17ab6f-ae28-4af6-818b-bfaabd219dbf admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81408) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:26:58.430954 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6d17ab6f-ae28-4af6-818b-bfaabd219dbf admin admin] Validating token access rules against request {{(pid=81408) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:26:58.434723 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3db69276-5d7b-4270-97fa-dde30bbe0eee service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd8812c3974f84871bb2d761cf0016257', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dcec0431fb6849499d6aec35a234daa6', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81408) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:26:58.435267 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3db69276-5d7b-4270-97fa-dde30bbe0eee service neutron] REQUEST_METHOD: `GET` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:26:58.435545 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3db69276-5d7b-4270-97fa-dde30bbe0eee service neutron] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:26:58.435816 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3db69276-5d7b-4270-97fa-dde30bbe0eee service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:26:58.445045 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3db69276-5d7b-4270-97fa-dde30bbe0eee service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:26:58.446507 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3db69276-5d7b-4270-97fa-dde30bbe0eee service neutron] RBAC: Authorization granted {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:26:58.456600 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 1008/2048] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Oct 13 04:26:58 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 85 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 13 04:26:58.724927 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4e7a088c-6bba-464a-a560-26688f126d25 None None] Authenticating user token {{(pid=81409) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:26:58.726419 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.fernet_utils [None req-4e7a088c-6bba-464a-a560-26688f126d25 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81409) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:26:58.739432 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4e7a088c-6bba-464a-a560-26688f126d25 None None] Validating token access rules against request {{(pid=81409) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:26:58.744331 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0a555653-c22f-4c3e-9756-535775d139ee admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8ba823f313144da3b683bacef8fd62e0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81409) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:26:58.745170 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a555653-c22f-4c3e-9756-535775d139ee admin admin] REQUEST_METHOD: `DELETE` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:26:58.745454 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a555653-c22f-4c3e-9756-535775d139ee admin admin] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:26:58.745717 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a555653-c22f-4c3e-9756-535775d139ee admin admin] PATH_INFO: `/v3/projects/372ec495c99743798f45d2532e65ca23` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:26:58.746899 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0a555653-c22f-4c3e-9756-535775d139ee admin admin] RBAC: Authorizing `identity:delete_project(project_id=372ec495c99743798f45d2532e65ca23)` {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:26:58.748897 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0a555653-c22f-4c3e-9756-535775d139ee admin admin] RBAC: Authorization granted {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:26:58.796841 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.notifications [None req-0a555653-c22f-4c3e-9756-535775d139ee admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '372ec495c99743798f45d2532e65ca23', 'request_id': 'req-0a555653-c22f-4c3e-9756-535775d139ee'} {{(pid=81409) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 13 04:26:58.804878 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.notifications [None req-0a555653-c22f-4c3e-9756-535775d139ee admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '372ec495c99743798f45d2532e65ca23', 'request_id': 'req-0a555653-c22f-4c3e-9756-535775d139ee'} {{(pid=81409) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 13 04:26:58.811363 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.notifications [None req-0a555653-c22f-4c3e-9756-535775d139ee admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '372ec495c99743798f45d2532e65ca23', 'request_id': 'req-0a555653-c22f-4c3e-9756-535775d139ee'} {{(pid=81409) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 13 04:26:58.823825 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.notifications [None req-0a555653-c22f-4c3e-9756-535775d139ee admin admin] The token cache is being invalidate because project 372ec495c99743798f45d2532e65ca23 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81409) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 13 04:26:58.824113 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.notifications [None req-0a555653-c22f-4c3e-9756-535775d139ee admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81409) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 13 04:26:58.825519 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.notifications [None req-0a555653-c22f-4c3e-9756-535775d139ee admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81409) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 13 04:26:58.827779 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 1041/2049] 10.222.0.27 () {58 vars in 1233 bytes} [Thu Oct 13 04:26:58 2022] DELETE /identity/v3/projects/372ec495c99743798f45d2532e65ca23 => generated 0 bytes in 103 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 13 04:26:58.832351 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3db69276-5d7b-4270-97fa-dde30bbe0eee service neutron] Authenticating user token {{(pid=81408) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:26:58.843925 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3db69276-5d7b-4270-97fa-dde30bbe0eee service neutron] Validating token access rules against request {{(pid=81408) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:26:58.875800 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG dbcounter [-] [81408] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81408) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:26:58.925323 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d2434a76-7f51-4d74-9acb-c7b7766a6c53 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8ba823f313144da3b683bacef8fd62e0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81408) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:26:58.925966 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2434a76-7f51-4d74-9acb-c7b7766a6c53 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:26:58.926332 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2434a76-7f51-4d74-9acb-c7b7766a6c53 admin admin] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:26:58.926597 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2434a76-7f51-4d74-9acb-c7b7766a6c53 admin admin] PATH_INFO: `/v3/users/cf1b18f763e6452fad740651f9a4c396` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:26:58.927771 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d2434a76-7f51-4d74-9acb-c7b7766a6c53 admin admin] RBAC: Authorizing `identity:delete_user(user_id=cf1b18f763e6452fad740651f9a4c396)` {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:26:58.929727 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d2434a76-7f51-4d74-9acb-c7b7766a6c53 admin admin] RBAC: Authorization granted {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:26:58.980464 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.notifications [None req-d2434a76-7f51-4d74-9acb-c7b7766a6c53 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'cf1b18f763e6452fad740651f9a4c396'} {{(pid=81408) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 13 04:26:58.987825 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.notifications [None req-d2434a76-7f51-4d74-9acb-c7b7766a6c53 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'cf1b18f763e6452fad740651f9a4c396'} {{(pid=81408) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 13 04:26:58.988506 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.notifications [None req-d2434a76-7f51-4d74-9acb-c7b7766a6c53 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'cf1b18f763e6452fad740651f9a4c396'} {{(pid=81408) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 13 04:26:58.989107 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.notifications [None req-d2434a76-7f51-4d74-9acb-c7b7766a6c53 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'cf1b18f763e6452fad740651f9a4c396'} {{(pid=81408) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 13 04:26:58.996687 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.notifications [None req-d2434a76-7f51-4d74-9acb-c7b7766a6c53 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'cf1b18f763e6452fad740651f9a4c396'} {{(pid=81408) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 13 04:26:59.015652 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 1009/2050] 10.222.0.27 () {58 vars in 1224 bytes} [Thu Oct 13 04:26:58 2022] DELETE /identity/v3/users/cf1b18f763e6452fad740651f9a4c396 => generated 0 bytes in 184 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 13 04:26:59.383443 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-826a42c3-03d3-4639-aea3-d14b8f3a179f None None] REQUEST_METHOD: `POST` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:26:59.383953 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-826a42c3-03d3-4639-aea3-d14b8f3a179f None None] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:26:59.384065 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-826a42c3-03d3-4639-aea3-d14b8f3a179f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:26:59.405609 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.auth.core [None req-826a42c3-03d3-4639-aea3-d14b8f3a179f None None] MFA Rules not processed for user `487b1605fd4143cfaa422a3a7a0344c7`. Rule list: `[]` (Enabled: `True`). {{(pid=81409) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 13 04:26:59.410992 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.fernet_utils [None req-826a42c3-03d3-4639-aea3-d14b8f3a179f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81409) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:26:59.459023 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d2434a76-7f51-4d74-9acb-c7b7766a6c53 admin admin] Authenticating user token {{(pid=81408) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:26:59.460487 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.fernet_utils [None req-d2434a76-7f51-4d74-9acb-c7b7766a6c53 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81408) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:26:59.490234 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 1042/2051] 10.222.0.27 () {58 vars in 1009 bytes} [Thu Oct 13 04:26:59 2022] POST /identity/v3/auth/tokens => generated 3195 bytes in 108 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 13 04:26:59.544769 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d2434a76-7f51-4d74-9acb-c7b7766a6c53 admin admin] Validating token access rules against request {{(pid=81408) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:26:59.552611 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9eb00f14-421e-4191-a77d-48a73db2aa1d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8ba823f313144da3b683bacef8fd62e0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81408) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:26:59.553360 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9eb00f14-421e-4191-a77d-48a73db2aa1d admin admin] REQUEST_METHOD: `DELETE` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:26:59.553705 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9eb00f14-421e-4191-a77d-48a73db2aa1d admin admin] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:26:59.554053 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9eb00f14-421e-4191-a77d-48a73db2aa1d admin admin] PATH_INFO: `/v3/projects/800f49615f1f4506bfaa79751170a668` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:26:59.555425 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9eb00f14-421e-4191-a77d-48a73db2aa1d admin admin] RBAC: Authorizing `identity:delete_project(project_id=800f49615f1f4506bfaa79751170a668)` {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:26:59.557429 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9eb00f14-421e-4191-a77d-48a73db2aa1d admin admin] RBAC: Authorization granted {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:26:59.574989 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-826a42c3-03d3-4639-aea3-d14b8f3a179f None None] Authenticating user token {{(pid=81409) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:26:59.577145 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.fernet_utils [None req-826a42c3-03d3-4639-aea3-d14b8f3a179f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81409) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:26:59.603858 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG dbcounter [-] [81409] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=81409) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:26:59.624045 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG dbcounter [-] [81408] Writing DB stats keystone:SELECT=86,keystone:DELETE=13,keystone:INSERT=1 {{(pid=81408) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:26:59.631241 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.notifications [None req-9eb00f14-421e-4191-a77d-48a73db2aa1d admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '800f49615f1f4506bfaa79751170a668', 'request_id': 'req-9eb00f14-421e-4191-a77d-48a73db2aa1d'} {{(pid=81408) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 13 04:26:59.646130 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.notifications [None req-9eb00f14-421e-4191-a77d-48a73db2aa1d admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '800f49615f1f4506bfaa79751170a668', 'request_id': 'req-9eb00f14-421e-4191-a77d-48a73db2aa1d'} {{(pid=81408) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 13 04:26:59.654373 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.notifications [None req-9eb00f14-421e-4191-a77d-48a73db2aa1d admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '800f49615f1f4506bfaa79751170a668', 'request_id': 'req-9eb00f14-421e-4191-a77d-48a73db2aa1d'} {{(pid=81408) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 13 04:26:59.667715 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.notifications [None req-9eb00f14-421e-4191-a77d-48a73db2aa1d admin admin] The token cache is being invalidate because project 800f49615f1f4506bfaa79751170a668 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81408) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 13 04:26:59.668071 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.notifications [None req-9eb00f14-421e-4191-a77d-48a73db2aa1d admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81408) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 13 04:26:59.668719 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.notifications [None req-9eb00f14-421e-4191-a77d-48a73db2aa1d admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81408) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 13 04:26:59.670302 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 1010/2052] 10.222.0.27 () {58 vars in 1233 bytes} [Thu Oct 13 04:26:59 2022] DELETE /identity/v3/projects/800f49615f1f4506bfaa79751170a668 => generated 0 bytes in 212 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 13 04:26:59.677262 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-166588e8-eefb-4623-b57a-d4c58f074b59 None None] REQUEST_METHOD: `POST` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:26:59.677902 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-166588e8-eefb-4623-b57a-d4c58f074b59 None None] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:26:59.678275 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-166588e8-eefb-4623-b57a-d4c58f074b59 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:26:59.685006 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-826a42c3-03d3-4639-aea3-d14b8f3a179f None None] Validating token access rules against request {{(pid=81409) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:26:59.705405 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.auth.core [None req-166588e8-eefb-4623-b57a-d4c58f074b59 None None] MFA Rules not processed for user `0e2d979563d6421995451a0e86bf3022`. Rule list: `[]` (Enabled: `True`). {{(pid=81408) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 13 04:26:59.707347 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.fernet_utils [None req-166588e8-eefb-4623-b57a-d4c58f074b59 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81408) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:26:59.766533 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eea5c71a-1341-4098-bdb7-e75285ffc3c3 service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90a0f466b664458898cf9dc4112ae651', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dcec0431fb6849499d6aec35a234daa6', 'project_domain_id': 'default', 'roles': ['service', 'member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81409) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:26:59.767195 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eea5c71a-1341-4098-bdb7-e75285ffc3c3 service swift] REQUEST_METHOD: `GET` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:26:59.767474 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eea5c71a-1341-4098-bdb7-e75285ffc3c3 service swift] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:26:59.767743 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eea5c71a-1341-4098-bdb7-e75285ffc3c3 service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:26:59.770057 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.fernet_utils [None req-eea5c71a-1341-4098-bdb7-e75285ffc3c3 service swift] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81409) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:26:59.782188 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 1011/2053] 10.222.0.27 () {58 vars in 958 bytes} [Thu Oct 13 04:26:59 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 108 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 13 04:26:59.791396 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f5bcb8b-a255-4fe2-b70e-93045f33877f None None] REQUEST_METHOD: `POST` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:26:59.791684 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f5bcb8b-a255-4fe2-b70e-93045f33877f None None] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:26:59.791961 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f5bcb8b-a255-4fe2-b70e-93045f33877f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:26:59.812914 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.auth.core [None req-9f5bcb8b-a255-4fe2-b70e-93045f33877f None None] MFA Rules not processed for user `0e2d979563d6421995451a0e86bf3022`. Rule list: `[]` (Enabled: `True`). {{(pid=81408) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 13 04:26:59.814171 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.fernet_utils [None req-9f5bcb8b-a255-4fe2-b70e-93045f33877f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81408) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:26:59.828416 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 1012/2054] 10.222.0.27 () {58 vars in 958 bytes} [Thu Oct 13 04:26:59 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 38 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 13 04:26:59.833585 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9f5bcb8b-a255-4fe2-b70e-93045f33877f None None] Authenticating user token {{(pid=81408) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:26:59.845006 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9f5bcb8b-a255-4fe2-b70e-93045f33877f None None] Validating token access rules against request {{(pid=81408) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:26:59.850296 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0cb474ce-f4ac-4526-b7d1-b66f6405cfcc admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8ba823f313144da3b683bacef8fd62e0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81408) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:26:59.851035 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0cb474ce-f4ac-4526-b7d1-b66f6405cfcc admin admin] REQUEST_METHOD: `GET` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:26:59.851413 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0cb474ce-f4ac-4526-b7d1-b66f6405cfcc admin admin] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:26:59.851781 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0cb474ce-f4ac-4526-b7d1-b66f6405cfcc admin admin] PATH_INFO: `/v3/domains` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:26:59.852573 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0cb474ce-f4ac-4526-b7d1-b66f6405cfcc admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81408) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 13 04:26:59.852974 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0cb474ce-f4ac-4526-b7d1-b66f6405cfcc admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:26:59.854719 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0cb474ce-f4ac-4526-b7d1-b66f6405cfcc admin admin] RBAC: Authorization granted {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:26:59.863016 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eea5c71a-1341-4098-bdb7-e75285ffc3c3 service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:26:59.867993 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eea5c71a-1341-4098-bdb7-e75285ffc3c3 service swift] RBAC: Authorization granted {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:26:59.870305 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 1013/2055] 10.222.0.27 () {58 vars in 1153 bytes} [Thu Oct 13 04:26:59 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 38 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 13 04:26:59.874890 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0cb474ce-f4ac-4526-b7d1-b66f6405cfcc admin admin] Authenticating user token {{(pid=81408) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:26:59.896531 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0cb474ce-f4ac-4526-b7d1-b66f6405cfcc admin admin] Validating token access rules against request {{(pid=81408) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:26:59.899718 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 1043/2056] 10.222.0.27 () {60 vars in 1384 bytes} [Thu Oct 13 04:26:59 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 325 msecs (HTTP/1.1 200) 6 headers in 379 bytes (2 switches on core 0) Oct 13 04:26:59.902943 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-36c1a70c-b095-4f81-b9dc-8c6e7e159ce7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8ba823f313144da3b683bacef8fd62e0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81408) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:26:59.903500 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36c1a70c-b095-4f81-b9dc-8c6e7e159ce7 admin admin] REQUEST_METHOD: `POST` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:26:59.903796 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36c1a70c-b095-4f81-b9dc-8c6e7e159ce7 admin admin] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:26:59.904122 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36c1a70c-b095-4f81-b9dc-8c6e7e159ce7 admin admin] PATH_INFO: `/v3/projects` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:26:59.904769 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-36c1a70c-b095-4f81-b9dc-8c6e7e159ce7 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:26:59.913008 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-36c1a70c-b095-4f81-b9dc-8c6e7e159ce7 admin admin] RBAC: Authorization granted {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:26:59.980821 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 1014/2057] 10.222.0.27 () {60 vars in 1153 bytes} [Thu Oct 13 04:26:59 2022] POST /identity/v3/projects => generated 387 bytes in 106 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 13 04:26:59.985857 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eea5c71a-1341-4098-bdb7-e75285ffc3c3 service swift] Authenticating user token {{(pid=81409) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:27:00.015980 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eea5c71a-1341-4098-bdb7-e75285ffc3c3 service swift] Validating token access rules against request {{(pid=81409) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:27:00.136217 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG dbcounter [-] [81409] Writing DB stats keystone:SELECT=100 {{(pid=81409) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:27:00.154494 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-301faba6-aeae-4439-87c5-8279a6e3b29d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8ba823f313144da3b683bacef8fd62e0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81409) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:27:00.158133 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-301faba6-aeae-4439-87c5-8279a6e3b29d admin admin] REQUEST_METHOD: `POST` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:27:00.161465 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-301faba6-aeae-4439-87c5-8279a6e3b29d admin admin] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:27:00.161757 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-301faba6-aeae-4439-87c5-8279a6e3b29d admin admin] PATH_INFO: `/v3/users` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:27:00.162488 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-301faba6-aeae-4439-87c5-8279a6e3b29d admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:27:00.164482 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-301faba6-aeae-4439-87c5-8279a6e3b29d admin admin] RBAC: Authorization granted {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:27:00.207194 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 1044/2058] 10.222.0.27 () {60 vars in 1144 bytes} [Thu Oct 13 04:26:59 2022] POST /identity/v3/users => generated 294 bytes in 222 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 13 04:27:00.208363 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-36c1a70c-b095-4f81-b9dc-8c6e7e159ce7 admin admin] Authenticating user token {{(pid=81408) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:27:00.235988 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-36c1a70c-b095-4f81-b9dc-8c6e7e159ce7 admin admin] Validating token access rules against request {{(pid=81408) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:27:00.249273 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2205953a-209a-4174-9f5f-b2167a81ecf3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8ba823f313144da3b683bacef8fd62e0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81408) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:27:00.250013 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2205953a-209a-4174-9f5f-b2167a81ecf3 admin admin] REQUEST_METHOD: `GET` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:27:00.250337 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2205953a-209a-4174-9f5f-b2167a81ecf3 admin admin] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:27:00.250739 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2205953a-209a-4174-9f5f-b2167a81ecf3 admin admin] PATH_INFO: `/v3/roles` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:27:00.264709 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2205953a-209a-4174-9f5f-b2167a81ecf3 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:27:00.266465 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2205953a-209a-4174-9f5f-b2167a81ecf3 admin admin] RBAC: Authorization granted {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:27:00.292262 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 1015/2059] 10.222.0.27 () {58 vars in 1122 bytes} [Thu Oct 13 04:27:00 2022] GET /identity/v3/roles => generated 1420 bytes in 84 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 13 04:27:00.294447 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-301faba6-aeae-4439-87c5-8279a6e3b29d admin admin] Authenticating user token {{(pid=81409) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:27:00.331306 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-301faba6-aeae-4439-87c5-8279a6e3b29d admin admin] Validating token access rules against request {{(pid=81409) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:27:00.336329 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0345728a-931e-405c-8bd0-6d3b6a3006e8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8ba823f313144da3b683bacef8fd62e0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81409) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:27:00.337353 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0345728a-931e-405c-8bd0-6d3b6a3006e8 admin admin] REQUEST_METHOD: `GET` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:27:00.337638 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0345728a-931e-405c-8bd0-6d3b6a3006e8 admin admin] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:27:00.337912 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0345728a-931e-405c-8bd0-6d3b6a3006e8 admin admin] PATH_INFO: `/v3/roles` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:27:00.338773 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0345728a-931e-405c-8bd0-6d3b6a3006e8 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:27:00.340596 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0345728a-931e-405c-8bd0-6d3b6a3006e8 admin admin] RBAC: Authorization granted {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:27:00.356276 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 1045/2060] 10.222.0.27 () {58 vars in 1122 bytes} [Thu Oct 13 04:27:00 2022] GET /identity/v3/roles => generated 1420 bytes in 61 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Oct 13 04:27:00.358367 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2205953a-209a-4174-9f5f-b2167a81ecf3 admin admin] Authenticating user token {{(pid=81408) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:27:00.376156 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2205953a-209a-4174-9f5f-b2167a81ecf3 admin admin] Validating token access rules against request {{(pid=81408) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:27:00.388245 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5456993c-c6df-418f-a245-57baa441a46b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8ba823f313144da3b683bacef8fd62e0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81408) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:27:00.388944 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5456993c-c6df-418f-a245-57baa441a46b admin admin] REQUEST_METHOD: `PUT` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:27:00.389222 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5456993c-c6df-418f-a245-57baa441a46b admin admin] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:27:00.389611 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5456993c-c6df-418f-a245-57baa441a46b admin admin] PATH_INFO: `/v3/projects/764ba7ae552148ceac4327624ff06db5/users/e8697d895a7d453dbe38994e7d161f9c/roles/cc56c4ae13a7424ea771c15da4b153c1` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:27:00.418601 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5456993c-c6df-418f-a245-57baa441a46b admin admin] RBAC: Authorizing `identity:create_grant(project_id=764ba7ae552148ceac4327624ff06db5, user_id=e8697d895a7d453dbe38994e7d161f9c, role_id=cc56c4ae13a7424ea771c15da4b153c1)` {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:27:00.421687 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5456993c-c6df-418f-a245-57baa441a46b admin admin] RBAC: Authorization granted {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:27:00.451011 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 1016/2061] 10.222.0.27 () {60 vars in 1483 bytes} [Thu Oct 13 04:27:00 2022] PUT /identity/v3/projects/764ba7ae552148ceac4327624ff06db5/users/e8697d895a7d453dbe38994e7d161f9c/roles/cc56c4ae13a7424ea771c15da4b153c1 => generated 0 bytes in 94 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 13 04:27:00.462897 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95103ad4-58c7-4169-bfc4-4641e092264b None None] REQUEST_METHOD: `POST` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:27:00.464260 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95103ad4-58c7-4169-bfc4-4641e092264b None None] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:27:00.464694 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95103ad4-58c7-4169-bfc4-4641e092264b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:27:00.509813 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.auth.core [None req-95103ad4-58c7-4169-bfc4-4641e092264b None None] MFA Rules not processed for user `e8697d895a7d453dbe38994e7d161f9c`. Rule list: `[]` (Enabled: `True`). {{(pid=81409) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 13 04:27:00.511087 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.fernet_utils [None req-95103ad4-58c7-4169-bfc4-4641e092264b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81409) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:27:00.602962 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 1046/2062] 10.222.0.27 () {58 vars in 958 bytes} [Thu Oct 13 04:27:00 2022] POST /identity/v3/auth/tokens => generated 3264 bytes in 138 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 13 04:27:00.604150 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5456993c-c6df-418f-a245-57baa441a46b admin admin] Authenticating user token {{(pid=81408) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:27:00.734787 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG dbcounter [-] [81408] Writing DB stats keystone:SELECT=93,keystone:DELETE=4,keystone:INSERT=3 {{(pid=81408) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:27:00.748322 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5456993c-c6df-418f-a245-57baa441a46b admin admin] Validating token access rules against request {{(pid=81408) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:27:00.764707 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bb9831ac-6d66-425b-b68f-63e5c3b1cebc admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8ba823f313144da3b683bacef8fd62e0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81408) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:27:00.765495 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb9831ac-6d66-425b-b68f-63e5c3b1cebc admin admin] REQUEST_METHOD: `POST` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:27:00.765921 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb9831ac-6d66-425b-b68f-63e5c3b1cebc admin admin] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:27:00.766291 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb9831ac-6d66-425b-b68f-63e5c3b1cebc admin admin] PATH_INFO: `/v3/projects` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:27:00.767048 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bb9831ac-6d66-425b-b68f-63e5c3b1cebc admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:27:00.769156 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bb9831ac-6d66-425b-b68f-63e5c3b1cebc admin admin] RBAC: Authorization granted {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:27:00.811242 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 1017/2063] 10.222.0.27 () {60 vars in 1153 bytes} [Thu Oct 13 04:27:00 2022] POST /identity/v3/projects => generated 385 bytes in 207 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 13 04:27:00.815763 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-95103ad4-58c7-4169-bfc4-4641e092264b None None] Authenticating user token {{(pid=81409) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:27:00.833970 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-95103ad4-58c7-4169-bfc4-4641e092264b None None] Validating token access rules against request {{(pid=81409) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:27:00.925210 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3e7ad03b-c46c-487c-a0b7-620c88b6f4d2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8ba823f313144da3b683bacef8fd62e0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81409) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:27:00.926306 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e7ad03b-c46c-487c-a0b7-620c88b6f4d2 admin admin] REQUEST_METHOD: `POST` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:27:00.926845 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e7ad03b-c46c-487c-a0b7-620c88b6f4d2 admin admin] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:27:00.927257 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e7ad03b-c46c-487c-a0b7-620c88b6f4d2 admin admin] PATH_INFO: `/v3/users` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:27:00.928407 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3e7ad03b-c46c-487c-a0b7-620c88b6f4d2 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:27:00.930632 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3e7ad03b-c46c-487c-a0b7-620c88b6f4d2 admin admin] RBAC: Authorization granted {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:27:00.960185 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 1047/2064] 10.222.0.27 () {60 vars in 1144 bytes} [Thu Oct 13 04:27:00 2022] POST /identity/v3/users => generated 299 bytes in 145 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 13 04:27:00.961236 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bb9831ac-6d66-425b-b68f-63e5c3b1cebc admin admin] Authenticating user token {{(pid=81408) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:27:00.983549 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bb9831ac-6d66-425b-b68f-63e5c3b1cebc admin admin] Validating token access rules against request {{(pid=81408) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:27:00.998574 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-91f8f261-016c-4130-bf6e-879d5f1b1e16 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8ba823f313144da3b683bacef8fd62e0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81408) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:27:00.999210 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91f8f261-016c-4130-bf6e-879d5f1b1e16 admin admin] REQUEST_METHOD: `GET` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:27:00.999487 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91f8f261-016c-4130-bf6e-879d5f1b1e16 admin admin] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:27:00.999757 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91f8f261-016c-4130-bf6e-879d5f1b1e16 admin admin] PATH_INFO: `/v3/roles` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:27:01.000444 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-91f8f261-016c-4130-bf6e-879d5f1b1e16 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:27:01.002062 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-91f8f261-016c-4130-bf6e-879d5f1b1e16 admin admin] RBAC: Authorization granted {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:27:01.015515 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 1018/2065] 10.222.0.27 () {58 vars in 1122 bytes} [Thu Oct 13 04:27:00 2022] GET /identity/v3/roles => generated 1420 bytes in 55 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Oct 13 04:27:01.020384 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3e7ad03b-c46c-487c-a0b7-620c88b6f4d2 admin admin] Authenticating user token {{(pid=81409) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:27:01.037207 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3e7ad03b-c46c-487c-a0b7-620c88b6f4d2 admin admin] Validating token access rules against request {{(pid=81409) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:27:01.042081 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-547f4678-463d-4a0f-93ac-0af289e90c52 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8ba823f313144da3b683bacef8fd62e0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81409) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:27:01.042850 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-547f4678-463d-4a0f-93ac-0af289e90c52 admin admin] REQUEST_METHOD: `PUT` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:27:01.043183 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-547f4678-463d-4a0f-93ac-0af289e90c52 admin admin] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:27:01.043520 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-547f4678-463d-4a0f-93ac-0af289e90c52 admin admin] PATH_INFO: `/v3/projects/7506c4653ed64bfc9a5321bb780ea714/users/b3d06402d9744fc895025d6ebc1130db/roles/ef55fbd94e1040be83695754075a7ed5` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:27:01.068688 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-547f4678-463d-4a0f-93ac-0af289e90c52 admin admin] RBAC: Authorizing `identity:create_grant(project_id=7506c4653ed64bfc9a5321bb780ea714, user_id=b3d06402d9744fc895025d6ebc1130db, role_id=ef55fbd94e1040be83695754075a7ed5)` {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:27:01.073300 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-547f4678-463d-4a0f-93ac-0af289e90c52 admin admin] RBAC: Authorization granted {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:27:01.090464 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 1048/2066] 10.222.0.27 () {60 vars in 1483 bytes} [Thu Oct 13 04:27:01 2022] PUT /identity/v3/projects/7506c4653ed64bfc9a5321bb780ea714/users/b3d06402d9744fc895025d6ebc1130db/roles/ef55fbd94e1040be83695754075a7ed5 => generated 0 bytes in 71 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 13 04:27:01.094750 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-91f8f261-016c-4130-bf6e-879d5f1b1e16 admin admin] Authenticating user token {{(pid=81408) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:27:01.107394 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-91f8f261-016c-4130-bf6e-879d5f1b1e16 admin admin] Validating token access rules against request {{(pid=81408) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:27:01.196560 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9d39ab5a-de15-4277-a17a-ee3eb8e9f8e9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8ba823f313144da3b683bacef8fd62e0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81408) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:27:01.197099 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d39ab5a-de15-4277-a17a-ee3eb8e9f8e9 admin admin] REQUEST_METHOD: `GET` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:27:01.197378 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d39ab5a-de15-4277-a17a-ee3eb8e9f8e9 admin admin] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:27:01.197651 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d39ab5a-de15-4277-a17a-ee3eb8e9f8e9 admin admin] PATH_INFO: `/v3/roles` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:27:01.198373 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9d39ab5a-de15-4277-a17a-ee3eb8e9f8e9 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:27:01.200143 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9d39ab5a-de15-4277-a17a-ee3eb8e9f8e9 admin admin] RBAC: Authorization granted {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:27:01.214263 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 1019/2067] 10.222.0.27 () {58 vars in 1122 bytes} [Thu Oct 13 04:27:01 2022] GET /identity/v3/roles => generated 1420 bytes in 119 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Oct 13 04:27:01.217112 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-547f4678-463d-4a0f-93ac-0af289e90c52 admin admin] Authenticating user token {{(pid=81409) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:27:01.230091 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-547f4678-463d-4a0f-93ac-0af289e90c52 admin admin] Validating token access rules against request {{(pid=81409) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:27:01.239906 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fc186968-b953-4138-af88-caa6abe7d5ea admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8ba823f313144da3b683bacef8fd62e0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81409) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:27:01.240587 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc186968-b953-4138-af88-caa6abe7d5ea admin admin] REQUEST_METHOD: `PUT` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:27:01.240863 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc186968-b953-4138-af88-caa6abe7d5ea admin admin] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:27:01.241133 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc186968-b953-4138-af88-caa6abe7d5ea admin admin] PATH_INFO: `/v3/projects/7506c4653ed64bfc9a5321bb780ea714/users/b3d06402d9744fc895025d6ebc1130db/roles/ef55fbd94e1040be83695754075a7ed5` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:27:01.243529 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fc186968-b953-4138-af88-caa6abe7d5ea admin admin] RBAC: Authorizing `identity:create_grant(project_id=7506c4653ed64bfc9a5321bb780ea714, user_id=b3d06402d9744fc895025d6ebc1130db, role_id=ef55fbd94e1040be83695754075a7ed5)` {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:27:01.246521 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fc186968-b953-4138-af88-caa6abe7d5ea admin admin] RBAC: Authorization granted {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:27:01.259349 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 1049/2068] 10.222.0.27 () {60 vars in 1483 bytes} [Thu Oct 13 04:27:01 2022] PUT /identity/v3/projects/7506c4653ed64bfc9a5321bb780ea714/users/b3d06402d9744fc895025d6ebc1130db/roles/ef55fbd94e1040be83695754075a7ed5 => generated 0 bytes in 42 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 13 04:27:01.271689 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de50f480-dea4-41f8-880f-277ef730c819 None None] REQUEST_METHOD: `POST` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:27:01.272115 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de50f480-dea4-41f8-880f-277ef730c819 None None] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:27:01.272496 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de50f480-dea4-41f8-880f-277ef730c819 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:27:01.310953 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.auth.core [None req-de50f480-dea4-41f8-880f-277ef730c819 None None] MFA Rules not processed for user `b3d06402d9744fc895025d6ebc1130db`. Rule list: `[]` (Enabled: `True`). {{(pid=81408) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 13 04:27:01.312175 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.fernet_utils [None req-de50f480-dea4-41f8-880f-277ef730c819 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81408) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:27:01.428845 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 1020/2069] 10.222.0.27 () {58 vars in 958 bytes} [Thu Oct 13 04:27:01 2022] POST /identity/v3/auth/tokens => generated 3329 bytes in 158 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 13 04:27:01.440268 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fc186968-b953-4138-af88-caa6abe7d5ea admin admin] Authenticating user token {{(pid=81409) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:27:01.443051 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.fernet_utils [None req-fc186968-b953-4138-af88-caa6abe7d5ea admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81409) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:27:01.466104 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG dbcounter [-] [81409] Writing DB stats keystone:SELECT=92,keystone:INSERT=8 {{(pid=81409) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:27:01.531792 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fc186968-b953-4138-af88-caa6abe7d5ea admin admin] Validating token access rules against request {{(pid=81409) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:27:01.536484 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8375cd59-b467-479c-b73e-14cf00293828 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6141b8a6c3d54e179d6879bb019f4d12', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dcec0431fb6849499d6aec35a234daa6', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81409) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:27:01.537160 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8375cd59-b467-479c-b73e-14cf00293828 service cinder] REQUEST_METHOD: `GET` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:27:01.537435 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8375cd59-b467-479c-b73e-14cf00293828 service cinder] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:27:01.537701 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8375cd59-b467-479c-b73e-14cf00293828 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:27:01.552618 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8375cd59-b467-479c-b73e-14cf00293828 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:27:01.554070 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8375cd59-b467-479c-b73e-14cf00293828 service cinder] RBAC: Authorization granted {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:27:01.565381 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 1050/2070] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Oct 13 04:27:01 2022] GET /identity/v3/auth/tokens => generated 3329 bytes in 125 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 13 04:27:01.651240 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-de50f480-dea4-41f8-880f-277ef730c819 None None] Authenticating user token {{(pid=81408) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:27:01.711582 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG dbcounter [-] [81408] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81408) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:27:01.733455 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-de50f480-dea4-41f8-880f-277ef730c819 None None] Validating token access rules against request {{(pid=81408) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:27:01.738868 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e16d6546-45fd-47d4-9037-05851820198f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8ba823f313144da3b683bacef8fd62e0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81408) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:27:01.739493 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e16d6546-45fd-47d4-9037-05851820198f admin admin] REQUEST_METHOD: `DELETE` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:27:01.739775 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e16d6546-45fd-47d4-9037-05851820198f admin admin] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:27:01.740042 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e16d6546-45fd-47d4-9037-05851820198f admin admin] PATH_INFO: `/v3/users/e8697d895a7d453dbe38994e7d161f9c` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:27:01.741155 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e16d6546-45fd-47d4-9037-05851820198f admin admin] RBAC: Authorizing `identity:delete_user(user_id=e8697d895a7d453dbe38994e7d161f9c)` {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:27:01.743537 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e16d6546-45fd-47d4-9037-05851820198f admin admin] RBAC: Authorization granted {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:27:01.832778 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.notifications [None req-e16d6546-45fd-47d4-9037-05851820198f admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'e8697d895a7d453dbe38994e7d161f9c'} {{(pid=81408) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 13 04:27:01.840601 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.notifications [None req-e16d6546-45fd-47d4-9037-05851820198f admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'e8697d895a7d453dbe38994e7d161f9c'} {{(pid=81408) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 13 04:27:01.841433 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.notifications [None req-e16d6546-45fd-47d4-9037-05851820198f admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'e8697d895a7d453dbe38994e7d161f9c'} {{(pid=81408) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 13 04:27:01.842040 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.notifications [None req-e16d6546-45fd-47d4-9037-05851820198f admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'e8697d895a7d453dbe38994e7d161f9c'} {{(pid=81408) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 13 04:27:01.855582 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.notifications [None req-e16d6546-45fd-47d4-9037-05851820198f admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'e8697d895a7d453dbe38994e7d161f9c'} {{(pid=81408) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 13 04:27:01.881639 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 1021/2071] 10.222.0.27 () {58 vars in 1224 bytes} [Thu Oct 13 04:27:01 2022] DELETE /identity/v3/users/e8697d895a7d453dbe38994e7d161f9c => generated 0 bytes in 231 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 13 04:27:01.886150 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2a094bf-db92-48bf-8773-2e8de71be978 None None] REQUEST_METHOD: `POST` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:27:01.886455 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2a094bf-db92-48bf-8773-2e8de71be978 None None] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:27:01.886728 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2a094bf-db92-48bf-8773-2e8de71be978 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:27:01.896625 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-312a07ce-3c7a-4e34-808b-b92eed06f841 None None] REQUEST_METHOD: `POST` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:27:01.896916 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-312a07ce-3c7a-4e34-808b-b92eed06f841 None None] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:27:01.897189 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-312a07ce-3c7a-4e34-808b-b92eed06f841 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:27:01.912733 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.auth.core [None req-b2a094bf-db92-48bf-8773-2e8de71be978 None None] MFA Rules not processed for user `0e2d979563d6421995451a0e86bf3022`. Rule list: `[]` (Enabled: `True`). {{(pid=81409) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 13 04:27:01.914023 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.fernet_utils [None req-b2a094bf-db92-48bf-8773-2e8de71be978 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81409) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:27:01.929435 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.auth.core [None req-312a07ce-3c7a-4e34-808b-b92eed06f841 None None] MFA Rules not processed for user `487b1605fd4143cfaa422a3a7a0344c7`. Rule list: `[]` (Enabled: `True`). {{(pid=81408) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 13 04:27:01.930677 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.fernet_utils [None req-312a07ce-3c7a-4e34-808b-b92eed06f841 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81408) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:27:02.004725 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 1051/2072] 10.222.0.27 () {58 vars in 958 bytes} [Thu Oct 13 04:27:01 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 120 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 13 04:27:02.010351 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 1022/2073] 10.222.0.27 () {58 vars in 967 bytes} [Thu Oct 13 04:27:01 2022] POST /identity/v3/auth/tokens => generated 3195 bytes in 115 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 13 04:27:02.015747 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b2a094bf-db92-48bf-8773-2e8de71be978 None None] Authenticating user token {{(pid=81409) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:27:02.021029 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.fernet_utils [None req-b2a094bf-db92-48bf-8773-2e8de71be978 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81409) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:27:02.028194 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-312a07ce-3c7a-4e34-808b-b92eed06f841 None None] Authenticating user token {{(pid=81408) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:27:02.029614 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.fernet_utils [None req-312a07ce-3c7a-4e34-808b-b92eed06f841 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81408) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:27:02.092262 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b2a094bf-db92-48bf-8773-2e8de71be978 None None] Validating token access rules against request {{(pid=81409) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:27:02.098517 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0d9be861-6d4b-4548-999e-9adeacfc18f7 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd8812c3974f84871bb2d761cf0016257', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dcec0431fb6849499d6aec35a234daa6', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81409) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:27:02.100379 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d9be861-6d4b-4548-999e-9adeacfc18f7 service neutron] REQUEST_METHOD: `GET` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:27:02.100819 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d9be861-6d4b-4548-999e-9adeacfc18f7 service neutron] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:27:02.101277 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d9be861-6d4b-4548-999e-9adeacfc18f7 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:27:02.114073 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0d9be861-6d4b-4548-999e-9adeacfc18f7 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:27:02.115871 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0d9be861-6d4b-4548-999e-9adeacfc18f7 service neutron] RBAC: Authorization granted {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:27:02.129778 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 1052/2074] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Oct 13 04:27:02 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 114 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 13 04:27:02.149584 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-312a07ce-3c7a-4e34-808b-b92eed06f841 None None] Validating token access rules against request {{(pid=81408) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:27:02.162245 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-53d4065f-8e98-4d33-840f-2535ee23936e service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90a0f466b664458898cf9dc4112ae651', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dcec0431fb6849499d6aec35a234daa6', 'project_domain_id': 'default', 'roles': ['service', 'member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81408) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:27:02.163406 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53d4065f-8e98-4d33-840f-2535ee23936e service swift] REQUEST_METHOD: `GET` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:27:02.163978 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53d4065f-8e98-4d33-840f-2535ee23936e service swift] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:27:02.164412 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53d4065f-8e98-4d33-840f-2535ee23936e service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:27:02.174176 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-53d4065f-8e98-4d33-840f-2535ee23936e service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:27:02.176455 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-53d4065f-8e98-4d33-840f-2535ee23936e service swift] RBAC: Authorization granted {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:27:02.186124 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 1023/2075] 10.222.0.27 () {60 vars in 1384 bytes} [Thu Oct 13 04:27:02 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 159 msecs (HTTP/1.1 200) 6 headers in 379 bytes (1 switches on core 0) Oct 13 04:27:02.546252 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0d9be861-6d4b-4548-999e-9adeacfc18f7 service neutron] Authenticating user token {{(pid=81409) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:27:02.548196 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.fernet_utils [None req-0d9be861-6d4b-4548-999e-9adeacfc18f7 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81409) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:27:02.600835 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0d9be861-6d4b-4548-999e-9adeacfc18f7 service neutron] Validating token access rules against request {{(pid=81409) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:27:02.615779 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bbd02211-a9f9-4c9f-aa58-48f250cae261 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8ba823f313144da3b683bacef8fd62e0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81409) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:27:02.616475 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bbd02211-a9f9-4c9f-aa58-48f250cae261 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:27:02.616807 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bbd02211-a9f9-4c9f-aa58-48f250cae261 admin admin] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:27:02.617831 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bbd02211-a9f9-4c9f-aa58-48f250cae261 admin admin] PATH_INFO: `/v3/projects/764ba7ae552148ceac4327624ff06db5` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:27:02.619330 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bbd02211-a9f9-4c9f-aa58-48f250cae261 admin admin] RBAC: Authorizing `identity:delete_project(project_id=764ba7ae552148ceac4327624ff06db5)` {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:27:02.621837 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bbd02211-a9f9-4c9f-aa58-48f250cae261 admin admin] RBAC: Authorization granted {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:27:02.696649 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.notifications [None req-bbd02211-a9f9-4c9f-aa58-48f250cae261 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '764ba7ae552148ceac4327624ff06db5', 'request_id': 'req-bbd02211-a9f9-4c9f-aa58-48f250cae261'} {{(pid=81409) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 13 04:27:02.708728 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.notifications [None req-bbd02211-a9f9-4c9f-aa58-48f250cae261 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '764ba7ae552148ceac4327624ff06db5', 'request_id': 'req-bbd02211-a9f9-4c9f-aa58-48f250cae261'} {{(pid=81409) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 13 04:27:02.710714 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG dbcounter [-] [81409] Writing DB stats keystone:SELECT=94,keystone:DELETE=5,keystone:INSERT=1 {{(pid=81409) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:27:02.718048 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.notifications [None req-bbd02211-a9f9-4c9f-aa58-48f250cae261 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '764ba7ae552148ceac4327624ff06db5', 'request_id': 'req-bbd02211-a9f9-4c9f-aa58-48f250cae261'} {{(pid=81409) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 13 04:27:02.735051 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.notifications [None req-bbd02211-a9f9-4c9f-aa58-48f250cae261 admin admin] The token cache is being invalidate because project 764ba7ae552148ceac4327624ff06db5 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81409) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 13 04:27:02.735741 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.notifications [None req-bbd02211-a9f9-4c9f-aa58-48f250cae261 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81409) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 13 04:27:02.737393 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.notifications [None req-bbd02211-a9f9-4c9f-aa58-48f250cae261 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81409) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 13 04:27:02.747766 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 1053/2076] 10.222.0.27 () {58 vars in 1233 bytes} [Thu Oct 13 04:27:02 2022] DELETE /identity/v3/projects/764ba7ae552148ceac4327624ff06db5 => generated 0 bytes in 202 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 13 04:27:02.751550 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-53d4065f-8e98-4d33-840f-2535ee23936e service swift] Authenticating user token {{(pid=81408) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:27:02.758630 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG dbcounter [-] [81408] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81408) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:27:02.774439 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-53d4065f-8e98-4d33-840f-2535ee23936e service swift] Validating token access rules against request {{(pid=81408) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:27:02.883312 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-41d7e9af-3977-40cd-94c4-8e3aca8dffef admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8ba823f313144da3b683bacef8fd62e0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81408) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:27:02.883312 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41d7e9af-3977-40cd-94c4-8e3aca8dffef admin admin] REQUEST_METHOD: `DELETE` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:27:02.883312 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41d7e9af-3977-40cd-94c4-8e3aca8dffef admin admin] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:27:02.883312 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41d7e9af-3977-40cd-94c4-8e3aca8dffef admin admin] PATH_INFO: `/v3/users/b3d06402d9744fc895025d6ebc1130db` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:27:02.883907 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-41d7e9af-3977-40cd-94c4-8e3aca8dffef admin admin] RBAC: Authorizing `identity:delete_user(user_id=b3d06402d9744fc895025d6ebc1130db)` {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:27:02.883907 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-41d7e9af-3977-40cd-94c4-8e3aca8dffef admin admin] RBAC: Authorization granted {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:27:02.954721 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.notifications [None req-41d7e9af-3977-40cd-94c4-8e3aca8dffef admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'b3d06402d9744fc895025d6ebc1130db'} {{(pid=81408) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 13 04:27:02.969069 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.notifications [None req-41d7e9af-3977-40cd-94c4-8e3aca8dffef admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'b3d06402d9744fc895025d6ebc1130db'} {{(pid=81408) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 13 04:27:02.970188 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.notifications [None req-41d7e9af-3977-40cd-94c4-8e3aca8dffef admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'b3d06402d9744fc895025d6ebc1130db'} {{(pid=81408) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 13 04:27:02.971137 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.notifications [None req-41d7e9af-3977-40cd-94c4-8e3aca8dffef admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'b3d06402d9744fc895025d6ebc1130db'} {{(pid=81408) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 13 04:27:02.979393 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.notifications [None req-41d7e9af-3977-40cd-94c4-8e3aca8dffef admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'b3d06402d9744fc895025d6ebc1130db'} {{(pid=81408) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 13 04:27:03.006987 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 1024/2077] 10.222.0.27 () {58 vars in 1224 bytes} [Thu Oct 13 04:27:02 2022] DELETE /identity/v3/users/b3d06402d9744fc895025d6ebc1130db => generated 0 bytes in 256 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 13 04:27:03.377666 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bbd02211-a9f9-4c9f-aa58-48f250cae261 admin admin] Authenticating user token {{(pid=81409) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:27:03.379609 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.fernet_utils [None req-bbd02211-a9f9-4c9f-aa58-48f250cae261 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81409) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:27:03.403725 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bbd02211-a9f9-4c9f-aa58-48f250cae261 admin admin] Validating token access rules against request {{(pid=81409) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:27:03.535314 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-12fb3773-1532-471e-9c70-e8608582fd13 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8ba823f313144da3b683bacef8fd62e0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81409) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:27:03.535314 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12fb3773-1532-471e-9c70-e8608582fd13 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:27:03.535650 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12fb3773-1532-471e-9c70-e8608582fd13 admin admin] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:27:03.535650 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12fb3773-1532-471e-9c70-e8608582fd13 admin admin] PATH_INFO: `/v3/projects/7506c4653ed64bfc9a5321bb780ea714` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:27:03.539465 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-12fb3773-1532-471e-9c70-e8608582fd13 admin admin] RBAC: Authorizing `identity:delete_project(project_id=7506c4653ed64bfc9a5321bb780ea714)` {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:27:03.542727 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-12fb3773-1532-471e-9c70-e8608582fd13 admin admin] RBAC: Authorization granted {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:27:03.627608 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.notifications [None req-12fb3773-1532-471e-9c70-e8608582fd13 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '7506c4653ed64bfc9a5321bb780ea714', 'request_id': 'req-12fb3773-1532-471e-9c70-e8608582fd13'} {{(pid=81409) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 13 04:27:03.637854 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.notifications [None req-12fb3773-1532-471e-9c70-e8608582fd13 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '7506c4653ed64bfc9a5321bb780ea714', 'request_id': 'req-12fb3773-1532-471e-9c70-e8608582fd13'} {{(pid=81409) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 13 04:27:03.646503 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.notifications [None req-12fb3773-1532-471e-9c70-e8608582fd13 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '7506c4653ed64bfc9a5321bb780ea714', 'request_id': 'req-12fb3773-1532-471e-9c70-e8608582fd13'} {{(pid=81409) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 13 04:27:03.659540 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.notifications [None req-12fb3773-1532-471e-9c70-e8608582fd13 admin admin] The token cache is being invalidate because project 7506c4653ed64bfc9a5321bb780ea714 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81409) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 13 04:27:03.659832 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.notifications [None req-12fb3773-1532-471e-9c70-e8608582fd13 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81409) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 13 04:27:03.660515 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.notifications [None req-12fb3773-1532-471e-9c70-e8608582fd13 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81409) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 13 04:27:03.662258 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 1054/2078] 10.222.0.27 () {58 vars in 1233 bytes} [Thu Oct 13 04:27:03 2022] DELETE /identity/v3/projects/7506c4653ed64bfc9a5321bb780ea714 => generated 0 bytes in 286 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 13 04:27:03.667261 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29342e02-af07-4cb8-b3b8-d58f3e8c7967 None None] REQUEST_METHOD: `POST` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:27:03.667543 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29342e02-af07-4cb8-b3b8-d58f3e8c7967 None None] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:27:03.667819 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29342e02-af07-4cb8-b3b8-d58f3e8c7967 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:27:03.691298 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.auth.core [None req-29342e02-af07-4cb8-b3b8-d58f3e8c7967 None None] MFA Rules not processed for user `0e2d979563d6421995451a0e86bf3022`. Rule list: `[]` (Enabled: `True`). {{(pid=81408) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 13 04:27:03.692543 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.fernet_utils [None req-29342e02-af07-4cb8-b3b8-d58f3e8c7967 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81408) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:27:03.782043 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 1025/2079] 10.222.0.27 () {58 vars in 958 bytes} [Thu Oct 13 04:27:03 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 116 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 13 04:27:03.784280 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd418491-f51a-49f9-92a9-4e4490f268ef None None] REQUEST_METHOD: `POST` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:27:03.784902 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd418491-f51a-49f9-92a9-4e4490f268ef None None] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:27:03.785241 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd418491-f51a-49f9-92a9-4e4490f268ef None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:27:03.807337 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.auth.core [None req-dd418491-f51a-49f9-92a9-4e4490f268ef None None] MFA Rules not processed for user `0e2d979563d6421995451a0e86bf3022`. Rule list: `[]` (Enabled: `True`). {{(pid=81409) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 13 04:27:03.808951 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.fernet_utils [None req-dd418491-f51a-49f9-92a9-4e4490f268ef None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81409) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:27:03.821631 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 1055/2080] 10.222.0.27 () {58 vars in 958 bytes} [Thu Oct 13 04:27:03 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 39 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 13 04:27:03.826453 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-29342e02-af07-4cb8-b3b8-d58f3e8c7967 None None] Authenticating user token {{(pid=81408) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:27:03.843282 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-29342e02-af07-4cb8-b3b8-d58f3e8c7967 None None] Validating token access rules against request {{(pid=81408) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:27:03.853936 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-374554ef-13f3-4185-ae0c-b9ac039704b4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8ba823f313144da3b683bacef8fd62e0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81408) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:27:03.854617 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-374554ef-13f3-4185-ae0c-b9ac039704b4 admin admin] REQUEST_METHOD: `GET` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:27:03.854897 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-374554ef-13f3-4185-ae0c-b9ac039704b4 admin admin] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:27:03.855172 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-374554ef-13f3-4185-ae0c-b9ac039704b4 admin admin] PATH_INFO: `/v3/domains` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:27:03.856253 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-374554ef-13f3-4185-ae0c-b9ac039704b4 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81408) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 13 04:27:03.857074 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-374554ef-13f3-4185-ae0c-b9ac039704b4 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:27:03.858750 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-374554ef-13f3-4185-ae0c-b9ac039704b4 admin admin] RBAC: Authorization granted {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:27:03.875098 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 1026/2081] 10.222.0.27 () {58 vars in 1153 bytes} [Thu Oct 13 04:27:03 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 49 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 13 04:27:03.880790 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dd418491-f51a-49f9-92a9-4e4490f268ef None None] Authenticating user token {{(pid=81409) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:27:03.893563 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dd418491-f51a-49f9-92a9-4e4490f268ef None None] Validating token access rules against request {{(pid=81409) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:27:03.912268 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3dde9e89-3a2c-45f6-ac75-08748d4e74ed admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8ba823f313144da3b683bacef8fd62e0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81409) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:27:03.913229 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3dde9e89-3a2c-45f6-ac75-08748d4e74ed admin admin] REQUEST_METHOD: `POST` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:27:03.913229 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3dde9e89-3a2c-45f6-ac75-08748d4e74ed admin admin] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:27:03.913413 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3dde9e89-3a2c-45f6-ac75-08748d4e74ed admin admin] PATH_INFO: `/v3/projects` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:27:03.914042 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3dde9e89-3a2c-45f6-ac75-08748d4e74ed admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:27:03.919460 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3dde9e89-3a2c-45f6-ac75-08748d4e74ed admin admin] RBAC: Authorization granted {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:27:03.961670 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 1056/2082] 10.222.0.27 () {60 vars in 1153 bytes} [Thu Oct 13 04:27:03 2022] POST /identity/v3/projects => generated 385 bytes in 82 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 13 04:27:03.965861 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-374554ef-13f3-4185-ae0c-b9ac039704b4 admin admin] Authenticating user token {{(pid=81408) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:27:03.977210 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-374554ef-13f3-4185-ae0c-b9ac039704b4 admin admin] Validating token access rules against request {{(pid=81408) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:27:03.987847 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG dbcounter [-] [81408] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81408) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:27:04.068590 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-42f26d81-0158-4b3c-b15d-05b938947dba admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8ba823f313144da3b683bacef8fd62e0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81408) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:27:04.069727 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42f26d81-0158-4b3c-b15d-05b938947dba admin admin] REQUEST_METHOD: `POST` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:27:04.070260 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42f26d81-0158-4b3c-b15d-05b938947dba admin admin] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:27:04.070721 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42f26d81-0158-4b3c-b15d-05b938947dba admin admin] PATH_INFO: `/v3/users` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:27:04.071569 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-42f26d81-0158-4b3c-b15d-05b938947dba admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:27:04.073940 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-42f26d81-0158-4b3c-b15d-05b938947dba admin admin] RBAC: Authorization granted {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:27:04.122330 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 1027/2083] 10.222.0.27 () {60 vars in 1144 bytes} [Thu Oct 13 04:27:03 2022] POST /identity/v3/users => generated 293 bytes in 154 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 13 04:27:04.126225 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3dde9e89-3a2c-45f6-ac75-08748d4e74ed admin admin] Authenticating user token {{(pid=81409) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:27:04.152573 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3dde9e89-3a2c-45f6-ac75-08748d4e74ed admin admin] Validating token access rules against request {{(pid=81409) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:27:04.160769 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-11be98fe-8fd8-40e9-99ab-a76ba952002d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8ba823f313144da3b683bacef8fd62e0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81409) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:27:04.161401 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11be98fe-8fd8-40e9-99ab-a76ba952002d admin admin] REQUEST_METHOD: `GET` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:27:04.161693 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11be98fe-8fd8-40e9-99ab-a76ba952002d admin admin] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:27:04.162118 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11be98fe-8fd8-40e9-99ab-a76ba952002d admin admin] PATH_INFO: `/v3/roles` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:27:04.162821 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-11be98fe-8fd8-40e9-99ab-a76ba952002d admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:27:04.164530 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-11be98fe-8fd8-40e9-99ab-a76ba952002d admin admin] RBAC: Authorization granted {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:27:04.181131 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 1057/2084] 10.222.0.27 () {58 vars in 1122 bytes} [Thu Oct 13 04:27:04 2022] GET /identity/v3/roles => generated 1420 bytes in 57 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 13 04:27:04.181860 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-42f26d81-0158-4b3c-b15d-05b938947dba admin admin] Authenticating user token {{(pid=81408) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:27:04.199485 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-42f26d81-0158-4b3c-b15d-05b938947dba admin admin] Validating token access rules against request {{(pid=81408) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:27:04.214705 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8dff3d05-2fdb-487d-8b0f-033f65a63632 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8ba823f313144da3b683bacef8fd62e0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81408) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:27:04.215613 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8dff3d05-2fdb-487d-8b0f-033f65a63632 admin admin] REQUEST_METHOD: `GET` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:27:04.215733 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8dff3d05-2fdb-487d-8b0f-033f65a63632 admin admin] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:27:04.215972 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8dff3d05-2fdb-487d-8b0f-033f65a63632 admin admin] PATH_INFO: `/v3/roles` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:27:04.216697 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8dff3d05-2fdb-487d-8b0f-033f65a63632 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:27:04.218544 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8dff3d05-2fdb-487d-8b0f-033f65a63632 admin admin] RBAC: Authorization granted {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:27:04.250289 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 1028/2085] 10.222.0.27 () {58 vars in 1122 bytes} [Thu Oct 13 04:27:04 2022] GET /identity/v3/roles => generated 1420 bytes in 67 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 13 04:27:04.251537 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-11be98fe-8fd8-40e9-99ab-a76ba952002d admin admin] Authenticating user token {{(pid=81409) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:27:04.271039 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-11be98fe-8fd8-40e9-99ab-a76ba952002d admin admin] Validating token access rules against request {{(pid=81409) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:27:04.280317 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-77f496d6-a42c-4082-a21a-ddc279c1b6fb admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8ba823f313144da3b683bacef8fd62e0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81409) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:27:04.281260 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77f496d6-a42c-4082-a21a-ddc279c1b6fb admin admin] REQUEST_METHOD: `PUT` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:27:04.281629 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77f496d6-a42c-4082-a21a-ddc279c1b6fb admin admin] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:27:04.281995 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77f496d6-a42c-4082-a21a-ddc279c1b6fb admin admin] PATH_INFO: `/v3/projects/fdd0f19eaeff4bf1b33fd1f31729cb06/users/c55592ca6f174115a40bc3230e93b6b7/roles/cc56c4ae13a7424ea771c15da4b153c1` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:27:04.369664 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-77f496d6-a42c-4082-a21a-ddc279c1b6fb admin admin] RBAC: Authorizing `identity:create_grant(project_id=fdd0f19eaeff4bf1b33fd1f31729cb06, user_id=c55592ca6f174115a40bc3230e93b6b7, role_id=cc56c4ae13a7424ea771c15da4b153c1)` {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:27:04.374132 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-77f496d6-a42c-4082-a21a-ddc279c1b6fb admin admin] RBAC: Authorization granted {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:27:04.396300 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 1058/2086] 10.222.0.27 () {60 vars in 1483 bytes} [Thu Oct 13 04:27:04 2022] PUT /identity/v3/projects/fdd0f19eaeff4bf1b33fd1f31729cb06/users/c55592ca6f174115a40bc3230e93b6b7/roles/cc56c4ae13a7424ea771c15da4b153c1 => generated 0 bytes in 145 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 13 04:27:04.416994 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e397b2ae-d857-4bad-9bce-8a64119d61e1 None None] REQUEST_METHOD: `POST` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:27:04.417783 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e397b2ae-d857-4bad-9bce-8a64119d61e1 None None] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:27:04.418151 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e397b2ae-d857-4bad-9bce-8a64119d61e1 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:27:04.468099 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.auth.core [None req-e397b2ae-d857-4bad-9bce-8a64119d61e1 None None] MFA Rules not processed for user `c55592ca6f174115a40bc3230e93b6b7`. Rule list: `[]` (Enabled: `True`). {{(pid=81408) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 13 04:27:04.469353 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.fernet_utils [None req-e397b2ae-d857-4bad-9bce-8a64119d61e1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81408) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:27:04.594709 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 1029/2087] 10.222.0.27 () {58 vars in 958 bytes} [Thu Oct 13 04:27:04 2022] POST /identity/v3/auth/tokens => generated 3262 bytes in 189 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 13 04:27:04.608584 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-77f496d6-a42c-4082-a21a-ddc279c1b6fb admin admin] Authenticating user token {{(pid=81409) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:27:04.610730 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.fernet_utils [None req-77f496d6-a42c-4082-a21a-ddc279c1b6fb admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81409) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:27:04.640021 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG dbcounter [-] [81409] Writing DB stats keystone:SELECT=88,keystone:DELETE=9,keystone:INSERT=3 {{(pid=81409) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:27:04.736191 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-77f496d6-a42c-4082-a21a-ddc279c1b6fb admin admin] Validating token access rules against request {{(pid=81409) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:27:04.740676 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c213cd5c-e590-4a62-b401-255af5e3d048 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6141b8a6c3d54e179d6879bb019f4d12', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dcec0431fb6849499d6aec35a234daa6', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81409) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:27:04.741481 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c213cd5c-e590-4a62-b401-255af5e3d048 service cinder] REQUEST_METHOD: `GET` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:27:04.741912 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c213cd5c-e590-4a62-b401-255af5e3d048 service cinder] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:27:04.742364 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c213cd5c-e590-4a62-b401-255af5e3d048 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:27:04.754008 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c213cd5c-e590-4a62-b401-255af5e3d048 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:27:04.762848 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c213cd5c-e590-4a62-b401-255af5e3d048 service cinder] RBAC: Authorization granted {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:27:04.792750 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 1059/2088] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Oct 13 04:27:04 2022] GET /identity/v3/auth/tokens => generated 3262 bytes in 183 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 13 04:27:12.074886 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e397b2ae-d857-4bad-9bce-8a64119d61e1 None None] Authenticating user token {{(pid=81408) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:27:12.076720 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.fernet_utils [None req-e397b2ae-d857-4bad-9bce-8a64119d61e1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81408) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:27:12.207625 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c213cd5c-e590-4a62-b401-255af5e3d048 service cinder] Authenticating user token {{(pid=81409) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:27:12.209070 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.fernet_utils [None req-c213cd5c-e590-4a62-b401-255af5e3d048 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81409) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:27:12.290906 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e397b2ae-d857-4bad-9bce-8a64119d61e1 None None] Validating token access rules against request {{(pid=81408) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:27:12.297926 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-428f12b3-0c2b-45de-a32b-0a64e9d1d3fb service placement] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ba1b6e5868a549a99760ebe6068fd6b9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dcec0431fb6849499d6aec35a234daa6', 'project_domain_id': 'default', 'roles': ['service', 'member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81408) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:27:12.299287 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-428f12b3-0c2b-45de-a32b-0a64e9d1d3fb service placement] REQUEST_METHOD: `GET` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:27:12.299795 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-428f12b3-0c2b-45de-a32b-0a64e9d1d3fb service placement] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:27:12.300559 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-428f12b3-0c2b-45de-a32b-0a64e9d1d3fb service placement] PATH_INFO: `/v3/auth/tokens` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:27:12.302590 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.fernet_utils [None req-428f12b3-0c2b-45de-a32b-0a64e9d1d3fb service placement] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81408) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:27:12.316772 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-428f12b3-0c2b-45de-a32b-0a64e9d1d3fb service placement] RBAC: Authorizing `identity:validate_token()` {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:27:12.318803 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-428f12b3-0c2b-45de-a32b-0a64e9d1d3fb service placement] RBAC: Authorization granted {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:27:12.331619 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c213cd5c-e590-4a62-b401-255af5e3d048 service cinder] Validating token access rules against request {{(pid=81409) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:27:12.333420 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 1030/2089] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Oct 13 04:27:12 2022] GET /identity/v3/auth/tokens => generated 3308 bytes in 259 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 13 04:27:12.338114 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-64832a8e-6d71-4ca8-98ae-9ffabae53e63 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8ba823f313144da3b683bacef8fd62e0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81409) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:27:12.338748 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64832a8e-6d71-4ca8-98ae-9ffabae53e63 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:27:12.339121 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64832a8e-6d71-4ca8-98ae-9ffabae53e63 admin admin] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:27:12.339484 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64832a8e-6d71-4ca8-98ae-9ffabae53e63 admin admin] PATH_INFO: `/v3/users/7582f5c0de6b4b5dae18b0320921e4a2` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:27:12.340759 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-64832a8e-6d71-4ca8-98ae-9ffabae53e63 admin admin] RBAC: Authorizing `identity:delete_user(user_id=7582f5c0de6b4b5dae18b0320921e4a2)` {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:27:12.343040 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-64832a8e-6d71-4ca8-98ae-9ffabae53e63 admin admin] RBAC: Authorization granted {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:27:12.419426 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.notifications [None req-64832a8e-6d71-4ca8-98ae-9ffabae53e63 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '7582f5c0de6b4b5dae18b0320921e4a2'} {{(pid=81409) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 13 04:27:12.420530 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.notifications [None req-64832a8e-6d71-4ca8-98ae-9ffabae53e63 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '7582f5c0de6b4b5dae18b0320921e4a2'} {{(pid=81409) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 13 04:27:12.437910 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.notifications [None req-64832a8e-6d71-4ca8-98ae-9ffabae53e63 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '7582f5c0de6b4b5dae18b0320921e4a2'} {{(pid=81409) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 13 04:27:12.438682 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.notifications [None req-64832a8e-6d71-4ca8-98ae-9ffabae53e63 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '7582f5c0de6b4b5dae18b0320921e4a2'} {{(pid=81409) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 13 04:27:12.448683 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.notifications [None req-64832a8e-6d71-4ca8-98ae-9ffabae53e63 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '7582f5c0de6b4b5dae18b0320921e4a2'} {{(pid=81409) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 13 04:27:12.469950 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 1060/2090] 10.222.0.27 () {58 vars in 1224 bytes} [Thu Oct 13 04:27:12 2022] DELETE /identity/v3/users/7582f5c0de6b4b5dae18b0320921e4a2 => generated 0 bytes in 263 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 13 04:27:12.641419 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-428f12b3-0c2b-45de-a32b-0a64e9d1d3fb service placement] Authenticating user token {{(pid=81408) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:27:12.656046 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-428f12b3-0c2b-45de-a32b-0a64e9d1d3fb service placement] Validating token access rules against request {{(pid=81408) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:27:12.663090 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG dbcounter [-] [81408] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81408) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:27:12.718109 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d82d1283-2829-474f-b3dd-645c3923d7b9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8ba823f313144da3b683bacef8fd62e0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81408) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:27:12.718771 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d82d1283-2829-474f-b3dd-645c3923d7b9 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:27:12.719047 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d82d1283-2829-474f-b3dd-645c3923d7b9 admin admin] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:27:12.719319 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d82d1283-2829-474f-b3dd-645c3923d7b9 admin admin] PATH_INFO: `/v3/projects/f2b13e92741f40bc8044ed4bc999b07f` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:27:12.721069 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d82d1283-2829-474f-b3dd-645c3923d7b9 admin admin] RBAC: Authorizing `identity:delete_project(project_id=f2b13e92741f40bc8044ed4bc999b07f)` {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:27:12.723179 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d82d1283-2829-474f-b3dd-645c3923d7b9 admin admin] RBAC: Authorization granted {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:27:12.768664 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.notifications [None req-d82d1283-2829-474f-b3dd-645c3923d7b9 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'f2b13e92741f40bc8044ed4bc999b07f', 'request_id': 'req-d82d1283-2829-474f-b3dd-645c3923d7b9'} {{(pid=81408) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 13 04:27:12.775827 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.notifications [None req-d82d1283-2829-474f-b3dd-645c3923d7b9 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'f2b13e92741f40bc8044ed4bc999b07f', 'request_id': 'req-d82d1283-2829-474f-b3dd-645c3923d7b9'} {{(pid=81408) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 13 04:27:12.782230 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.notifications [None req-d82d1283-2829-474f-b3dd-645c3923d7b9 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'f2b13e92741f40bc8044ed4bc999b07f', 'request_id': 'req-d82d1283-2829-474f-b3dd-645c3923d7b9'} {{(pid=81408) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 13 04:27:12.792589 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.notifications [None req-d82d1283-2829-474f-b3dd-645c3923d7b9 admin admin] The token cache is being invalidate because project f2b13e92741f40bc8044ed4bc999b07f was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81408) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 13 04:27:12.792877 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.notifications [None req-d82d1283-2829-474f-b3dd-645c3923d7b9 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81408) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 13 04:27:12.793585 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.notifications [None req-d82d1283-2829-474f-b3dd-645c3923d7b9 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81408) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 13 04:27:12.795158 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 1031/2091] 10.222.0.27 () {58 vars in 1233 bytes} [Thu Oct 13 04:27:12 2022] DELETE /identity/v3/projects/f2b13e92741f40bc8044ed4bc999b07f => generated 0 bytes in 154 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 13 04:27:12.799422 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-64832a8e-6d71-4ca8-98ae-9ffabae53e63 admin admin] Authenticating user token {{(pid=81409) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:27:12.800809 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.fernet_utils [None req-64832a8e-6d71-4ca8-98ae-9ffabae53e63 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81409) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:27:12.813755 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-64832a8e-6d71-4ca8-98ae-9ffabae53e63 admin admin] Validating token access rules against request {{(pid=81409) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:27:12.844470 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG dbcounter [-] [81409] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81409) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:27:12.872839 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-79cec24e-dad3-4a19-9b6d-cdba5f91a26f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8ba823f313144da3b683bacef8fd62e0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81409) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:27:12.873459 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79cec24e-dad3-4a19-9b6d-cdba5f91a26f admin admin] REQUEST_METHOD: `DELETE` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:27:12.873734 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79cec24e-dad3-4a19-9b6d-cdba5f91a26f admin admin] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:27:12.874004 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79cec24e-dad3-4a19-9b6d-cdba5f91a26f admin admin] PATH_INFO: `/v3/users/7b4787618a464777b59ec623551de8dc` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:27:12.875264 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-79cec24e-dad3-4a19-9b6d-cdba5f91a26f admin admin] RBAC: Authorizing `identity:delete_user(user_id=7b4787618a464777b59ec623551de8dc)` {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:27:12.878067 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-79cec24e-dad3-4a19-9b6d-cdba5f91a26f admin admin] RBAC: Authorization granted {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:27:12.925962 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.notifications [None req-79cec24e-dad3-4a19-9b6d-cdba5f91a26f admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '7b4787618a464777b59ec623551de8dc'} {{(pid=81409) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 13 04:27:12.926561 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.notifications [None req-79cec24e-dad3-4a19-9b6d-cdba5f91a26f admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '7b4787618a464777b59ec623551de8dc'} {{(pid=81409) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 13 04:27:12.934434 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.notifications [None req-79cec24e-dad3-4a19-9b6d-cdba5f91a26f admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '7b4787618a464777b59ec623551de8dc'} {{(pid=81409) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 13 04:27:12.935250 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.notifications [None req-79cec24e-dad3-4a19-9b6d-cdba5f91a26f admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '7b4787618a464777b59ec623551de8dc'} {{(pid=81409) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 13 04:27:12.944056 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.notifications [None req-79cec24e-dad3-4a19-9b6d-cdba5f91a26f admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '7b4787618a464777b59ec623551de8dc'} {{(pid=81409) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 13 04:27:12.963106 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 1061/2092] 10.222.0.27 () {58 vars in 1224 bytes} [Thu Oct 13 04:27:12 2022] DELETE /identity/v3/users/7b4787618a464777b59ec623551de8dc => generated 0 bytes in 164 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 13 04:27:13.157884 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d82d1283-2829-474f-b3dd-645c3923d7b9 admin admin] Authenticating user token {{(pid=81408) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:27:13.160030 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.fernet_utils [None req-d82d1283-2829-474f-b3dd-645c3923d7b9 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81408) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:27:13.172418 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d82d1283-2829-474f-b3dd-645c3923d7b9 admin admin] Validating token access rules against request {{(pid=81408) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:27:13.241686 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cf6ccbd2-177a-4061-a210-793e0d75a56c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8ba823f313144da3b683bacef8fd62e0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81408) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:27:13.242344 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf6ccbd2-177a-4061-a210-793e0d75a56c admin admin] REQUEST_METHOD: `DELETE` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:27:13.242627 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf6ccbd2-177a-4061-a210-793e0d75a56c admin admin] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:27:13.242898 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf6ccbd2-177a-4061-a210-793e0d75a56c admin admin] PATH_INFO: `/v3/projects/f3cec5d37e134dc395624f81a9f77fc1` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:27:13.244026 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cf6ccbd2-177a-4061-a210-793e0d75a56c admin admin] RBAC: Authorizing `identity:delete_project(project_id=f3cec5d37e134dc395624f81a9f77fc1)` {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:27:13.245942 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cf6ccbd2-177a-4061-a210-793e0d75a56c admin admin] RBAC: Authorization granted {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:27:13.290365 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG dbcounter [-] [81408] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81408) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:27:13.299064 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.notifications [None req-cf6ccbd2-177a-4061-a210-793e0d75a56c admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'f3cec5d37e134dc395624f81a9f77fc1', 'request_id': 'req-cf6ccbd2-177a-4061-a210-793e0d75a56c'} {{(pid=81408) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 13 04:27:13.306889 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.notifications [None req-cf6ccbd2-177a-4061-a210-793e0d75a56c admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'f3cec5d37e134dc395624f81a9f77fc1', 'request_id': 'req-cf6ccbd2-177a-4061-a210-793e0d75a56c'} {{(pid=81408) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 13 04:27:13.313892 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.notifications [None req-cf6ccbd2-177a-4061-a210-793e0d75a56c admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'f3cec5d37e134dc395624f81a9f77fc1', 'request_id': 'req-cf6ccbd2-177a-4061-a210-793e0d75a56c'} {{(pid=81408) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 13 04:27:13.325478 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.notifications [None req-cf6ccbd2-177a-4061-a210-793e0d75a56c admin admin] The token cache is being invalidate because project f3cec5d37e134dc395624f81a9f77fc1 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81408) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 13 04:27:13.325765 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.notifications [None req-cf6ccbd2-177a-4061-a210-793e0d75a56c admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81408) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 13 04:27:13.326437 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.notifications [None req-cf6ccbd2-177a-4061-a210-793e0d75a56c admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81408) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 13 04:27:13.327947 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 1032/2093] 10.222.0.27 () {58 vars in 1233 bytes} [Thu Oct 13 04:27:13 2022] DELETE /identity/v3/projects/f3cec5d37e134dc395624f81a9f77fc1 => generated 0 bytes in 171 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 13 04:27:14.137213 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-79cec24e-dad3-4a19-9b6d-cdba5f91a26f admin admin] Authenticating user token {{(pid=81409) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:27:14.138659 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.fernet_utils [None req-79cec24e-dad3-4a19-9b6d-cdba5f91a26f admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81409) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:27:14.153538 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-79cec24e-dad3-4a19-9b6d-cdba5f91a26f admin admin] Validating token access rules against request {{(pid=81409) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:27:14.209361 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a374f010-579a-4155-82f2-b8faa85bdcb2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8ba823f313144da3b683bacef8fd62e0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81409) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:27:14.209989 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a374f010-579a-4155-82f2-b8faa85bdcb2 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:27:14.210296 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a374f010-579a-4155-82f2-b8faa85bdcb2 admin admin] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:27:14.210568 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a374f010-579a-4155-82f2-b8faa85bdcb2 admin admin] PATH_INFO: `/v3/users/c55592ca6f174115a40bc3230e93b6b7` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:27:14.211680 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a374f010-579a-4155-82f2-b8faa85bdcb2 admin admin] RBAC: Authorizing `identity:delete_user(user_id=c55592ca6f174115a40bc3230e93b6b7)` {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:27:14.213587 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a374f010-579a-4155-82f2-b8faa85bdcb2 admin admin] RBAC: Authorization granted {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:27:14.260015 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.notifications [None req-a374f010-579a-4155-82f2-b8faa85bdcb2 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'c55592ca6f174115a40bc3230e93b6b7'} {{(pid=81409) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 13 04:27:14.260652 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.notifications [None req-a374f010-579a-4155-82f2-b8faa85bdcb2 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'c55592ca6f174115a40bc3230e93b6b7'} {{(pid=81409) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 13 04:27:14.265802 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG dbcounter [-] [81409] Writing DB stats keystone:SELECT=82,keystone:DELETE=17,keystone:INSERT=1 {{(pid=81409) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:27:14.268222 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.notifications [None req-a374f010-579a-4155-82f2-b8faa85bdcb2 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'c55592ca6f174115a40bc3230e93b6b7'} {{(pid=81409) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 13 04:27:14.269119 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.notifications [None req-a374f010-579a-4155-82f2-b8faa85bdcb2 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'c55592ca6f174115a40bc3230e93b6b7'} {{(pid=81409) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 13 04:27:14.277250 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.notifications [None req-a374f010-579a-4155-82f2-b8faa85bdcb2 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'c55592ca6f174115a40bc3230e93b6b7'} {{(pid=81409) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 13 04:27:14.294616 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 1062/2094] 10.222.0.27 () {58 vars in 1224 bytes} [Thu Oct 13 04:27:14 2022] DELETE /identity/v3/users/c55592ca6f174115a40bc3230e93b6b7 => generated 0 bytes in 158 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 13 04:27:14.299932 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79c29a65-0471-44fc-82c0-3f7962a35210 None None] REQUEST_METHOD: `POST` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:27:14.300444 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79c29a65-0471-44fc-82c0-3f7962a35210 None None] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:27:14.300551 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79c29a65-0471-44fc-82c0-3f7962a35210 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:27:14.319520 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.auth.core [None req-79c29a65-0471-44fc-82c0-3f7962a35210 None None] MFA Rules not processed for user `0e2d979563d6421995451a0e86bf3022`. Rule list: `[]` (Enabled: `True`). {{(pid=81408) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 13 04:27:14.320687 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.fernet_utils [None req-79c29a65-0471-44fc-82c0-3f7962a35210 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81408) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:27:14.378704 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 1033/2095] 10.222.0.27 () {58 vars in 958 bytes} [Thu Oct 13 04:27:14 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 81 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 13 04:27:14.388456 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a374f010-579a-4155-82f2-b8faa85bdcb2 admin admin] Authenticating user token {{(pid=81409) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:27:14.389925 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.fernet_utils [None req-a374f010-579a-4155-82f2-b8faa85bdcb2 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81409) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:27:14.449504 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a374f010-579a-4155-82f2-b8faa85bdcb2 admin admin] Validating token access rules against request {{(pid=81409) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:27:14.453267 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e4cb810d-deb9-493d-8c80-4db87e660b9d service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd8812c3974f84871bb2d761cf0016257', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dcec0431fb6849499d6aec35a234daa6', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81409) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:27:14.453879 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4cb810d-deb9-493d-8c80-4db87e660b9d service neutron] REQUEST_METHOD: `GET` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:27:14.454158 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4cb810d-deb9-493d-8c80-4db87e660b9d service neutron] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:27:14.454471 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4cb810d-deb9-493d-8c80-4db87e660b9d service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:27:14.463607 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e4cb810d-deb9-493d-8c80-4db87e660b9d service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:27:14.465025 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e4cb810d-deb9-493d-8c80-4db87e660b9d service neutron] RBAC: Authorization granted {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:27:14.475090 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 1063/2096] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Oct 13 04:27:14 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 87 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 13 04:27:14.740776 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-79c29a65-0471-44fc-82c0-3f7962a35210 None None] Authenticating user token {{(pid=81408) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:27:14.742395 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.fernet_utils [None req-79c29a65-0471-44fc-82c0-3f7962a35210 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81408) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:27:14.755525 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-79c29a65-0471-44fc-82c0-3f7962a35210 None None] Validating token access rules against request {{(pid=81408) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:27:14.760931 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c42bacbc-0177-4aa3-b6a3-30bb8ac563c8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8ba823f313144da3b683bacef8fd62e0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81408) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:27:14.761569 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c42bacbc-0177-4aa3-b6a3-30bb8ac563c8 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:27:14.761851 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c42bacbc-0177-4aa3-b6a3-30bb8ac563c8 admin admin] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:27:14.762122 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c42bacbc-0177-4aa3-b6a3-30bb8ac563c8 admin admin] PATH_INFO: `/v3/projects/fdd0f19eaeff4bf1b33fd1f31729cb06` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:27:14.763347 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c42bacbc-0177-4aa3-b6a3-30bb8ac563c8 admin admin] RBAC: Authorizing `identity:delete_project(project_id=fdd0f19eaeff4bf1b33fd1f31729cb06)` {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:27:14.765287 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c42bacbc-0177-4aa3-b6a3-30bb8ac563c8 admin admin] RBAC: Authorization granted {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:27:14.815851 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.notifications [None req-c42bacbc-0177-4aa3-b6a3-30bb8ac563c8 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'fdd0f19eaeff4bf1b33fd1f31729cb06', 'request_id': 'req-c42bacbc-0177-4aa3-b6a3-30bb8ac563c8'} {{(pid=81408) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 13 04:27:14.823634 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.notifications [None req-c42bacbc-0177-4aa3-b6a3-30bb8ac563c8 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'fdd0f19eaeff4bf1b33fd1f31729cb06', 'request_id': 'req-c42bacbc-0177-4aa3-b6a3-30bb8ac563c8'} {{(pid=81408) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 13 04:27:14.831111 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.notifications [None req-c42bacbc-0177-4aa3-b6a3-30bb8ac563c8 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'fdd0f19eaeff4bf1b33fd1f31729cb06', 'request_id': 'req-c42bacbc-0177-4aa3-b6a3-30bb8ac563c8'} {{(pid=81408) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 13 04:27:14.842143 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.notifications [None req-c42bacbc-0177-4aa3-b6a3-30bb8ac563c8 admin admin] The token cache is being invalidate because project fdd0f19eaeff4bf1b33fd1f31729cb06 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81408) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 13 04:27:14.842457 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.notifications [None req-c42bacbc-0177-4aa3-b6a3-30bb8ac563c8 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81408) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 13 04:27:14.843123 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.notifications [None req-c42bacbc-0177-4aa3-b6a3-30bb8ac563c8 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81408) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 13 04:27:14.844609 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 1034/2097] 10.222.0.27 () {58 vars in 1233 bytes} [Thu Oct 13 04:27:14 2022] DELETE /identity/v3/projects/fdd0f19eaeff4bf1b33fd1f31729cb06 => generated 0 bytes in 104 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 13 04:27:14.850182 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-175725d9-ce8c-4d10-b636-29e3085f761e None None] REQUEST_METHOD: `POST` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:27:14.850566 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-175725d9-ce8c-4d10-b636-29e3085f761e None None] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:27:14.850938 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-175725d9-ce8c-4d10-b636-29e3085f761e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:27:14.873753 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.auth.core [None req-175725d9-ce8c-4d10-b636-29e3085f761e None None] MFA Rules not processed for user `0e2d979563d6421995451a0e86bf3022`. Rule list: `[]` (Enabled: `True`). {{(pid=81409) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 13 04:27:14.874996 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.fernet_utils [None req-175725d9-ce8c-4d10-b636-29e3085f761e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81409) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:27:14.937535 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 1064/2098] 10.222.0.27 () {58 vars in 958 bytes} [Thu Oct 13 04:27:14 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 88 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 13 04:27:14.946324 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dbe6d545-2c3b-4e7e-bcce-e675a82f70c4 None None] REQUEST_METHOD: `POST` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:27:14.946608 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dbe6d545-2c3b-4e7e-bcce-e675a82f70c4 None None] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:27:14.946881 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dbe6d545-2c3b-4e7e-bcce-e675a82f70c4 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:27:14.966787 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.auth.core [None req-dbe6d545-2c3b-4e7e-bcce-e675a82f70c4 None None] MFA Rules not processed for user `0e2d979563d6421995451a0e86bf3022`. Rule list: `[]` (Enabled: `True`). {{(pid=81408) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 13 04:27:14.967986 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.fernet_utils [None req-dbe6d545-2c3b-4e7e-bcce-e675a82f70c4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81408) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:27:14.976396 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 1035/2099] 10.222.0.27 () {58 vars in 958 bytes} [Thu Oct 13 04:27:14 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 31 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 13 04:27:14.980465 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-175725d9-ce8c-4d10-b636-29e3085f761e None None] Authenticating user token {{(pid=81409) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:27:14.990948 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-175725d9-ce8c-4d10-b636-29e3085f761e None None] Validating token access rules against request {{(pid=81409) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:27:14.995419 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-181bd738-71a8-41fd-8a45-31f093fb5a19 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8ba823f313144da3b683bacef8fd62e0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81409) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:27:14.996003 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-181bd738-71a8-41fd-8a45-31f093fb5a19 admin admin] REQUEST_METHOD: `GET` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:27:14.996275 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-181bd738-71a8-41fd-8a45-31f093fb5a19 admin admin] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:27:14.996543 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-181bd738-71a8-41fd-8a45-31f093fb5a19 admin admin] PATH_INFO: `/v3/domains` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:27:14.997184 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-181bd738-71a8-41fd-8a45-31f093fb5a19 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81409) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 13 04:27:14.997485 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-181bd738-71a8-41fd-8a45-31f093fb5a19 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:27:14.999095 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-181bd738-71a8-41fd-8a45-31f093fb5a19 admin admin] RBAC: Authorization granted {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:27:15.011464 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 1065/2100] 10.222.0.27 () {58 vars in 1153 bytes} [Thu Oct 13 04:27:14 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 13 04:27:15.015633 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dbe6d545-2c3b-4e7e-bcce-e675a82f70c4 None None] Authenticating user token {{(pid=81408) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:27:15.024820 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dbe6d545-2c3b-4e7e-bcce-e675a82f70c4 None None] Validating token access rules against request {{(pid=81408) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:27:15.029436 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-22d7bf12-f7a2-4ce8-b8ad-95c808fc6451 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8ba823f313144da3b683bacef8fd62e0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81408) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:27:15.030018 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22d7bf12-f7a2-4ce8-b8ad-95c808fc6451 admin admin] REQUEST_METHOD: `POST` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:27:15.030324 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22d7bf12-f7a2-4ce8-b8ad-95c808fc6451 admin admin] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:27:15.030590 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22d7bf12-f7a2-4ce8-b8ad-95c808fc6451 admin admin] PATH_INFO: `/v3/projects` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:27:15.031219 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-22d7bf12-f7a2-4ce8-b8ad-95c808fc6451 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:27:15.033660 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-22d7bf12-f7a2-4ce8-b8ad-95c808fc6451 admin admin] RBAC: Authorization granted {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:27:15.060215 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 1036/2101] 10.222.0.27 () {60 vars in 1153 bytes} [Thu Oct 13 04:27:15 2022] POST /identity/v3/projects => generated 383 bytes in 45 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 13 04:27:15.064416 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-181bd738-71a8-41fd-8a45-31f093fb5a19 admin admin] Authenticating user token {{(pid=81409) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:27:15.074420 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-181bd738-71a8-41fd-8a45-31f093fb5a19 admin admin] Validating token access rules against request {{(pid=81409) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:27:15.132322 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-25db3f8a-7f00-445d-b55f-c2338269bdb6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8ba823f313144da3b683bacef8fd62e0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81409) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:27:15.132913 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25db3f8a-7f00-445d-b55f-c2338269bdb6 admin admin] REQUEST_METHOD: `POST` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:27:15.133203 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25db3f8a-7f00-445d-b55f-c2338269bdb6 admin admin] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:27:15.133448 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25db3f8a-7f00-445d-b55f-c2338269bdb6 admin admin] PATH_INFO: `/v3/users` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:27:15.134071 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-25db3f8a-7f00-445d-b55f-c2338269bdb6 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:27:15.136007 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-25db3f8a-7f00-445d-b55f-c2338269bdb6 admin admin] RBAC: Authorization granted {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:27:15.144993 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG dbcounter [-] [81409] Writing DB stats keystone:SELECT=95,keystone:INSERT=2,keystone:DELETE=3 {{(pid=81409) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:27:15.158834 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 1066/2102] 10.222.0.27 () {60 vars in 1144 bytes} [Thu Oct 13 04:27:15 2022] POST /identity/v3/users => generated 292 bytes in 95 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 13 04:27:15.163174 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-22d7bf12-f7a2-4ce8-b8ad-95c808fc6451 admin admin] Authenticating user token {{(pid=81408) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:27:15.173484 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-22d7bf12-f7a2-4ce8-b8ad-95c808fc6451 admin admin] Validating token access rules against request {{(pid=81408) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:27:15.178907 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9182067c-4b8c-458b-af37-aac62156402d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8ba823f313144da3b683bacef8fd62e0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81408) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:27:15.179500 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9182067c-4b8c-458b-af37-aac62156402d admin admin] REQUEST_METHOD: `GET` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:27:15.179775 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9182067c-4b8c-458b-af37-aac62156402d admin admin] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:27:15.180066 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9182067c-4b8c-458b-af37-aac62156402d admin admin] PATH_INFO: `/v3/roles` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:27:15.180703 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9182067c-4b8c-458b-af37-aac62156402d admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:27:15.182351 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9182067c-4b8c-458b-af37-aac62156402d admin admin] RBAC: Authorization granted {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:27:15.192232 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 1037/2103] 10.222.0.27 () {58 vars in 1122 bytes} [Thu Oct 13 04:27:15 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Oct 13 04:27:15.196011 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-25db3f8a-7f00-445d-b55f-c2338269bdb6 admin admin] Authenticating user token {{(pid=81409) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:27:15.206263 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-25db3f8a-7f00-445d-b55f-c2338269bdb6 admin admin] Validating token access rules against request {{(pid=81409) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:27:15.210790 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9148dbe4-0f9e-4511-bc07-71bda2d8dd7d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8ba823f313144da3b683bacef8fd62e0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81409) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:27:15.211397 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9148dbe4-0f9e-4511-bc07-71bda2d8dd7d admin admin] REQUEST_METHOD: `GET` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:27:15.211907 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9148dbe4-0f9e-4511-bc07-71bda2d8dd7d admin admin] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:27:15.212021 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9148dbe4-0f9e-4511-bc07-71bda2d8dd7d admin admin] PATH_INFO: `/v3/roles` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:27:15.212699 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9148dbe4-0f9e-4511-bc07-71bda2d8dd7d admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:27:15.214298 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9148dbe4-0f9e-4511-bc07-71bda2d8dd7d admin admin] RBAC: Authorization granted {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:27:15.225612 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 1067/2104] 10.222.0.27 () {58 vars in 1122 bytes} [Thu Oct 13 04:27:15 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 13 04:27:15.229543 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9182067c-4b8c-458b-af37-aac62156402d admin admin] Authenticating user token {{(pid=81408) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:27:15.240574 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9182067c-4b8c-458b-af37-aac62156402d admin admin] Validating token access rules against request {{(pid=81408) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:27:15.245204 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b9fb29ba-8e44-4348-accd-abe4de5675d7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8ba823f313144da3b683bacef8fd62e0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81408) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:27:15.245846 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9fb29ba-8e44-4348-accd-abe4de5675d7 admin admin] REQUEST_METHOD: `PUT` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:27:15.246125 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9fb29ba-8e44-4348-accd-abe4de5675d7 admin admin] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:27:15.246668 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9fb29ba-8e44-4348-accd-abe4de5675d7 admin admin] PATH_INFO: `/v3/projects/ec37e41bc87b4359b740f2e9f382d4b5/users/6efd2fa33b724a429d2146ad6d5abe6a/roles/cc56c4ae13a7424ea771c15da4b153c1` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:27:15.258260 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG dbcounter [-] [81408] Writing DB stats keystone:SELECT=86,keystone:DELETE=11,keystone:INSERT=3 {{(pid=81408) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:27:15.263267 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b9fb29ba-8e44-4348-accd-abe4de5675d7 admin admin] RBAC: Authorizing `identity:create_grant(project_id=ec37e41bc87b4359b740f2e9f382d4b5, user_id=6efd2fa33b724a429d2146ad6d5abe6a, role_id=cc56c4ae13a7424ea771c15da4b153c1)` {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:27:15.266335 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b9fb29ba-8e44-4348-accd-abe4de5675d7 admin admin] RBAC: Authorization granted {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:27:15.279850 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 1038/2105] 10.222.0.27 () {60 vars in 1483 bytes} [Thu Oct 13 04:27:15 2022] PUT /identity/v3/projects/ec37e41bc87b4359b740f2e9f382d4b5/users/6efd2fa33b724a429d2146ad6d5abe6a/roles/cc56c4ae13a7424ea771c15da4b153c1 => generated 0 bytes in 51 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 13 04:27:15.290793 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e2024e9-42d6-4856-abbb-52428810ca7e None None] REQUEST_METHOD: `POST` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:27:15.291069 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e2024e9-42d6-4856-abbb-52428810ca7e None None] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:27:15.291333 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e2024e9-42d6-4856-abbb-52428810ca7e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:27:15.319479 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.auth.core [None req-8e2024e9-42d6-4856-abbb-52428810ca7e None None] MFA Rules not processed for user `6efd2fa33b724a429d2146ad6d5abe6a`. Rule list: `[]` (Enabled: `True`). {{(pid=81409) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 13 04:27:15.320674 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.fernet_utils [None req-8e2024e9-42d6-4856-abbb-52428810ca7e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81409) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:27:15.385206 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 1068/2106] 10.222.0.27 () {58 vars in 958 bytes} [Thu Oct 13 04:27:15 2022] POST /identity/v3/auth/tokens => generated 3260 bytes in 95 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 13 04:27:15.389726 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b9fb29ba-8e44-4348-accd-abe4de5675d7 admin admin] Authenticating user token {{(pid=81408) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:27:15.451808 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b9fb29ba-8e44-4348-accd-abe4de5675d7 admin admin] Validating token access rules against request {{(pid=81408) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:27:15.456635 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cc1b02b4-f9f1-481d-8efc-1493d50d9adc admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8ba823f313144da3b683bacef8fd62e0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81408) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:27:15.457237 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc1b02b4-f9f1-481d-8efc-1493d50d9adc admin admin] REQUEST_METHOD: `POST` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:27:15.457517 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc1b02b4-f9f1-481d-8efc-1493d50d9adc admin admin] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:27:15.457790 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc1b02b4-f9f1-481d-8efc-1493d50d9adc admin admin] PATH_INFO: `/v3/projects` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:27:15.458477 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cc1b02b4-f9f1-481d-8efc-1493d50d9adc admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:27:15.460392 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cc1b02b4-f9f1-481d-8efc-1493d50d9adc admin admin] RBAC: Authorization granted {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:27:15.487446 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 1039/2107] 10.222.0.27 () {60 vars in 1153 bytes} [Thu Oct 13 04:27:15 2022] POST /identity/v3/projects => generated 383 bytes in 98 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 13 04:27:15.491588 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8e2024e9-42d6-4856-abbb-52428810ca7e None None] Authenticating user token {{(pid=81409) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:27:15.502748 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8e2024e9-42d6-4856-abbb-52428810ca7e None None] Validating token access rules against request {{(pid=81409) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:27:15.557773 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d5a48e84-1587-4ba4-9943-af594a683502 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8ba823f313144da3b683bacef8fd62e0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81409) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:27:15.558405 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5a48e84-1587-4ba4-9943-af594a683502 admin admin] REQUEST_METHOD: `POST` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:27:15.558676 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5a48e84-1587-4ba4-9943-af594a683502 admin admin] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:27:15.558939 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5a48e84-1587-4ba4-9943-af594a683502 admin admin] PATH_INFO: `/v3/users` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:27:15.559558 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d5a48e84-1587-4ba4-9943-af594a683502 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:27:15.561452 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d5a48e84-1587-4ba4-9943-af594a683502 admin admin] RBAC: Authorization granted {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:27:15.583813 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 1069/2108] 10.222.0.27 () {60 vars in 1144 bytes} [Thu Oct 13 04:27:15 2022] POST /identity/v3/users => generated 292 bytes in 93 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 13 04:27:15.587629 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cc1b02b4-f9f1-481d-8efc-1493d50d9adc admin admin] Authenticating user token {{(pid=81408) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:27:15.597910 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cc1b02b4-f9f1-481d-8efc-1493d50d9adc admin admin] Validating token access rules against request {{(pid=81408) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:27:15.602630 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f8cc4e80-7ae6-437f-abcf-b504b01f19a8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8ba823f313144da3b683bacef8fd62e0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81408) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:27:15.603215 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8cc4e80-7ae6-437f-abcf-b504b01f19a8 admin admin] REQUEST_METHOD: `GET` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:27:15.603486 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8cc4e80-7ae6-437f-abcf-b504b01f19a8 admin admin] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:27:15.603751 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8cc4e80-7ae6-437f-abcf-b504b01f19a8 admin admin] PATH_INFO: `/v3/roles` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:27:15.604378 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f8cc4e80-7ae6-437f-abcf-b504b01f19a8 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:27:15.606270 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f8cc4e80-7ae6-437f-abcf-b504b01f19a8 admin admin] RBAC: Authorization granted {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:27:15.617707 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 1040/2109] 10.222.0.27 () {58 vars in 1122 bytes} [Thu Oct 13 04:27:15 2022] GET /identity/v3/roles => generated 1420 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 13 04:27:15.621565 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d5a48e84-1587-4ba4-9943-af594a683502 admin admin] Authenticating user token {{(pid=81409) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:27:15.632382 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d5a48e84-1587-4ba4-9943-af594a683502 admin admin] Validating token access rules against request {{(pid=81409) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:27:15.637033 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-92c18207-23b4-4bdb-b9a9-ee9a277e9e56 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8ba823f313144da3b683bacef8fd62e0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81409) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:27:15.637658 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92c18207-23b4-4bdb-b9a9-ee9a277e9e56 admin admin] REQUEST_METHOD: `GET` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:27:15.637931 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92c18207-23b4-4bdb-b9a9-ee9a277e9e56 admin admin] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:27:15.638364 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92c18207-23b4-4bdb-b9a9-ee9a277e9e56 admin admin] PATH_INFO: `/v3/roles` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:27:15.639004 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-92c18207-23b4-4bdb-b9a9-ee9a277e9e56 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:27:15.640587 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-92c18207-23b4-4bdb-b9a9-ee9a277e9e56 admin admin] RBAC: Authorization granted {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:27:15.651211 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 1070/2110] 10.222.0.27 () {58 vars in 1122 bytes} [Thu Oct 13 04:27:15 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 13 04:27:15.655364 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f8cc4e80-7ae6-437f-abcf-b504b01f19a8 admin admin] Authenticating user token {{(pid=81408) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:27:15.665843 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f8cc4e80-7ae6-437f-abcf-b504b01f19a8 admin admin] Validating token access rules against request {{(pid=81408) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:27:15.670520 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-97da2a66-9418-4627-b8a6-0c0af4cb9eca admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8ba823f313144da3b683bacef8fd62e0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81408) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:27:15.671157 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97da2a66-9418-4627-b8a6-0c0af4cb9eca admin admin] REQUEST_METHOD: `PUT` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:27:15.671430 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97da2a66-9418-4627-b8a6-0c0af4cb9eca admin admin] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:27:15.671703 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97da2a66-9418-4627-b8a6-0c0af4cb9eca admin admin] PATH_INFO: `/v3/projects/593752e699e643cbbeda50363909f091/users/2569a8f7753c4c6e8db4197e250b11aa/roles/cc56c4ae13a7424ea771c15da4b153c1` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:27:15.688081 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-97da2a66-9418-4627-b8a6-0c0af4cb9eca admin admin] RBAC: Authorizing `identity:create_grant(project_id=593752e699e643cbbeda50363909f091, user_id=2569a8f7753c4c6e8db4197e250b11aa, role_id=cc56c4ae13a7424ea771c15da4b153c1)` {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:27:15.691827 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-97da2a66-9418-4627-b8a6-0c0af4cb9eca admin admin] RBAC: Authorization granted {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:27:15.705911 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 1041/2111] 10.222.0.27 () {60 vars in 1483 bytes} [Thu Oct 13 04:27:15 2022] PUT /identity/v3/projects/593752e699e643cbbeda50363909f091/users/2569a8f7753c4c6e8db4197e250b11aa/roles/cc56c4ae13a7424ea771c15da4b153c1 => generated 0 bytes in 51 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 13 04:27:15.716202 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06374d2a-56c5-4021-b4f7-ba96f8f8e890 None None] REQUEST_METHOD: `POST` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:27:15.716484 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06374d2a-56c5-4021-b4f7-ba96f8f8e890 None None] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:27:15.716754 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06374d2a-56c5-4021-b4f7-ba96f8f8e890 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:27:15.745694 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.auth.core [None req-06374d2a-56c5-4021-b4f7-ba96f8f8e890 None None] MFA Rules not processed for user `2569a8f7753c4c6e8db4197e250b11aa`. Rule list: `[]` (Enabled: `True`). {{(pid=81409) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 13 04:27:15.746934 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.fernet_utils [None req-06374d2a-56c5-4021-b4f7-ba96f8f8e890 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81409) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:27:15.788754 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG dbcounter [-] [81409] Writing DB stats keystone:INSERT=5,keystone:SELECT=95 {{(pid=81409) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:27:15.818462 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 1071/2112] 10.222.0.27 () {58 vars in 958 bytes} [Thu Oct 13 04:27:15 2022] POST /identity/v3/auth/tokens => generated 3260 bytes in 103 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 13 04:27:15.823067 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-97da2a66-9418-4627-b8a6-0c0af4cb9eca admin admin] Authenticating user token {{(pid=81408) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:27:15.886620 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-97da2a66-9418-4627-b8a6-0c0af4cb9eca admin admin] Validating token access rules against request {{(pid=81408) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:27:15.891425 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7ec66284-cf87-4220-9932-034c0ca957de admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8ba823f313144da3b683bacef8fd62e0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81408) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:27:15.892026 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ec66284-cf87-4220-9932-034c0ca957de admin admin] REQUEST_METHOD: `POST` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:27:15.892462 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ec66284-cf87-4220-9932-034c0ca957de admin admin] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:27:15.892754 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ec66284-cf87-4220-9932-034c0ca957de admin admin] PATH_INFO: `/v3/projects` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:27:15.893399 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7ec66284-cf87-4220-9932-034c0ca957de admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:27:15.895344 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7ec66284-cf87-4220-9932-034c0ca957de admin admin] RBAC: Authorization granted {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:27:15.930392 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 1042/2113] 10.222.0.27 () {60 vars in 1153 bytes} [Thu Oct 13 04:27:15 2022] POST /identity/v3/projects => generated 385 bytes in 104 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 13 04:27:15.932032 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-06374d2a-56c5-4021-b4f7-ba96f8f8e890 None None] Authenticating user token {{(pid=81409) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:27:15.942833 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-06374d2a-56c5-4021-b4f7-ba96f8f8e890 None None] Validating token access rules against request {{(pid=81409) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:27:16.000173 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-de8c71bd-d202-4755-a9fc-ab95f48a16de admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8ba823f313144da3b683bacef8fd62e0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81409) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:27:16.000813 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de8c71bd-d202-4755-a9fc-ab95f48a16de admin admin] REQUEST_METHOD: `POST` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:27:16.001088 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de8c71bd-d202-4755-a9fc-ab95f48a16de admin admin] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:27:16.001357 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de8c71bd-d202-4755-a9fc-ab95f48a16de admin admin] PATH_INFO: `/v3/users` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:27:16.002895 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-de8c71bd-d202-4755-a9fc-ab95f48a16de admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:27:16.004540 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-de8c71bd-d202-4755-a9fc-ab95f48a16de admin admin] RBAC: Authorization granted {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:27:16.028958 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 1072/2114] 10.222.0.27 () {60 vars in 1144 bytes} [Thu Oct 13 04:27:15 2022] POST /identity/v3/users => generated 299 bytes in 95 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 13 04:27:16.030127 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7ec66284-cf87-4220-9932-034c0ca957de admin admin] Authenticating user token {{(pid=81408) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:27:16.040364 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7ec66284-cf87-4220-9932-034c0ca957de admin admin] Validating token access rules against request {{(pid=81408) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:27:16.045121 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c55a10de-b4bc-4f47-95b5-77626e64a74e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8ba823f313144da3b683bacef8fd62e0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81408) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:27:16.045720 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c55a10de-b4bc-4f47-95b5-77626e64a74e admin admin] REQUEST_METHOD: `GET` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:27:16.045998 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c55a10de-b4bc-4f47-95b5-77626e64a74e admin admin] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:27:16.046299 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c55a10de-b4bc-4f47-95b5-77626e64a74e admin admin] PATH_INFO: `/v3/roles` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:27:16.046942 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c55a10de-b4bc-4f47-95b5-77626e64a74e admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:27:16.049751 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c55a10de-b4bc-4f47-95b5-77626e64a74e admin admin] RBAC: Authorization granted {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:27:16.063685 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 1043/2115] 10.222.0.27 () {58 vars in 1122 bytes} [Thu Oct 13 04:27:16 2022] GET /identity/v3/roles => generated 1420 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 13 04:27:16.067684 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-de8c71bd-d202-4755-a9fc-ab95f48a16de admin admin] Authenticating user token {{(pid=81409) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:27:16.077664 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-de8c71bd-d202-4755-a9fc-ab95f48a16de admin admin] Validating token access rules against request {{(pid=81409) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:27:16.083413 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f1eba745-3882-4f96-acb9-acc98010ff74 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8ba823f313144da3b683bacef8fd62e0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81409) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:27:16.084058 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1eba745-3882-4f96-acb9-acc98010ff74 admin admin] REQUEST_METHOD: `PUT` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:27:16.084330 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1eba745-3882-4f96-acb9-acc98010ff74 admin admin] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:27:16.084598 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1eba745-3882-4f96-acb9-acc98010ff74 admin admin] PATH_INFO: `/v3/projects/26c8523b6281498896eee7d7de498aca/users/f966e72aa2b64aec96e80edb9134a26d/roles/ef55fbd94e1040be83695754075a7ed5` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:27:16.101854 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f1eba745-3882-4f96-acb9-acc98010ff74 admin admin] RBAC: Authorizing `identity:create_grant(project_id=26c8523b6281498896eee7d7de498aca, user_id=f966e72aa2b64aec96e80edb9134a26d, role_id=ef55fbd94e1040be83695754075a7ed5)` {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:27:16.104555 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f1eba745-3882-4f96-acb9-acc98010ff74 admin admin] RBAC: Authorization granted {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:27:16.118684 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 1073/2116] 10.222.0.27 () {60 vars in 1483 bytes} [Thu Oct 13 04:27:16 2022] PUT /identity/v3/projects/26c8523b6281498896eee7d7de498aca/users/f966e72aa2b64aec96e80edb9134a26d/roles/ef55fbd94e1040be83695754075a7ed5 => generated 0 bytes in 51 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 13 04:27:16.123076 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c55a10de-b4bc-4f47-95b5-77626e64a74e admin admin] Authenticating user token {{(pid=81408) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:27:16.133342 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c55a10de-b4bc-4f47-95b5-77626e64a74e admin admin] Validating token access rules against request {{(pid=81408) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:27:16.144488 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG dbcounter [-] [81408] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=81408) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:27:16.196946 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-331cca16-b335-4609-b526-9e2d72086fa6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8ba823f313144da3b683bacef8fd62e0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81408) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:27:16.197557 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-331cca16-b335-4609-b526-9e2d72086fa6 admin admin] REQUEST_METHOD: `GET` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:27:16.197835 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-331cca16-b335-4609-b526-9e2d72086fa6 admin admin] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:27:16.198107 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-331cca16-b335-4609-b526-9e2d72086fa6 admin admin] PATH_INFO: `/v3/roles` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:27:16.198801 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-331cca16-b335-4609-b526-9e2d72086fa6 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:27:16.200720 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-331cca16-b335-4609-b526-9e2d72086fa6 admin admin] RBAC: Authorization granted {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:27:16.216335 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 1044/2117] 10.222.0.27 () {58 vars in 1122 bytes} [Thu Oct 13 04:27:16 2022] GET /identity/v3/roles => generated 1420 bytes in 91 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 13 04:27:16.217562 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f1eba745-3882-4f96-acb9-acc98010ff74 admin admin] Authenticating user token {{(pid=81409) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:27:16.232234 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f1eba745-3882-4f96-acb9-acc98010ff74 admin admin] Validating token access rules against request {{(pid=81409) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:27:16.238065 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8496f4c5-f93a-4e19-ab1d-2bfcfe811f07 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8ba823f313144da3b683bacef8fd62e0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81409) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:27:16.238829 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8496f4c5-f93a-4e19-ab1d-2bfcfe811f07 admin admin] REQUEST_METHOD: `PUT` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:27:16.239099 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8496f4c5-f93a-4e19-ab1d-2bfcfe811f07 admin admin] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:27:16.239384 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8496f4c5-f93a-4e19-ab1d-2bfcfe811f07 admin admin] PATH_INFO: `/v3/projects/26c8523b6281498896eee7d7de498aca/users/f966e72aa2b64aec96e80edb9134a26d/roles/ef55fbd94e1040be83695754075a7ed5` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:27:16.241435 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8496f4c5-f93a-4e19-ab1d-2bfcfe811f07 admin admin] RBAC: Authorizing `identity:create_grant(project_id=26c8523b6281498896eee7d7de498aca, user_id=f966e72aa2b64aec96e80edb9134a26d, role_id=ef55fbd94e1040be83695754075a7ed5)` {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:27:16.244218 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8496f4c5-f93a-4e19-ab1d-2bfcfe811f07 admin admin] RBAC: Authorization granted {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:27:16.257431 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 1074/2118] 10.222.0.27 () {60 vars in 1483 bytes} [Thu Oct 13 04:27:16 2022] PUT /identity/v3/projects/26c8523b6281498896eee7d7de498aca/users/f966e72aa2b64aec96e80edb9134a26d/roles/ef55fbd94e1040be83695754075a7ed5 => generated 0 bytes in 40 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 13 04:27:16.327234 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b2edc7b-9cf1-4c73-bb10-69cf25937401 None None] REQUEST_METHOD: `POST` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:27:16.327515 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b2edc7b-9cf1-4c73-bb10-69cf25937401 None None] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:27:16.327802 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b2edc7b-9cf1-4c73-bb10-69cf25937401 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:27:16.357711 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.auth.core [None req-0b2edc7b-9cf1-4c73-bb10-69cf25937401 None None] MFA Rules not processed for user `f966e72aa2b64aec96e80edb9134a26d`. Rule list: `[]` (Enabled: `True`). {{(pid=81408) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 13 04:27:16.358949 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.fernet_utils [None req-0b2edc7b-9cf1-4c73-bb10-69cf25937401 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81408) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:27:16.439285 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 1045/2119] 10.222.0.27 () {58 vars in 958 bytes} [Thu Oct 13 04:27:16 2022] POST /identity/v3/auth/tokens => generated 3329 bytes in 113 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 13 04:27:16.454581 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8496f4c5-f93a-4e19-ab1d-2bfcfe811f07 admin admin] Authenticating user token {{(pid=81409) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:27:16.454581 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.fernet_utils [None req-8496f4c5-f93a-4e19-ab1d-2bfcfe811f07 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81409) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:27:16.512352 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8496f4c5-f93a-4e19-ab1d-2bfcfe811f07 admin admin] Validating token access rules against request {{(pid=81409) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:27:16.515934 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fbf6c269-25be-469f-a00d-c4a542d2202a service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6141b8a6c3d54e179d6879bb019f4d12', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dcec0431fb6849499d6aec35a234daa6', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81409) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:27:16.516532 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fbf6c269-25be-469f-a00d-c4a542d2202a service cinder] REQUEST_METHOD: `GET` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:27:16.516801 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fbf6c269-25be-469f-a00d-c4a542d2202a service cinder] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:27:16.517072 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fbf6c269-25be-469f-a00d-c4a542d2202a service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:27:16.561294 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG dbcounter [-] [81409] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=81409) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:27:16.575871 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fbf6c269-25be-469f-a00d-c4a542d2202a service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:27:16.577324 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fbf6c269-25be-469f-a00d-c4a542d2202a service cinder] RBAC: Authorization granted {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:27:16.587237 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 1075/2120] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Oct 13 04:27:16 2022] GET /identity/v3/auth/tokens => generated 3260 bytes in 137 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 13 04:27:17.018811 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-699c5392-ae35-4a7a-a23d-8fd566726303 None None] REQUEST_METHOD: `GET` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:27:17.019102 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-699c5392-ae35-4a7a-a23d-8fd566726303 None None] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:27:17.019376 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-699c5392-ae35-4a7a-a23d-8fd566726303 None None] PATH_INFO: `/` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:27:17.020897 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 1046/2121] 10.222.0.27 () {54 vars in 921 bytes} [Thu Oct 13 04:27:17 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 13 04:27:17.025961 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2500f465-c3db-41e4-bb38-8c1bae7ab40f None None] REQUEST_METHOD: `POST` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:27:17.026309 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2500f465-c3db-41e4-bb38-8c1bae7ab40f None None] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:27:17.026595 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2500f465-c3db-41e4-bb38-8c1bae7ab40f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:27:17.047916 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.auth.core [None req-2500f465-c3db-41e4-bb38-8c1bae7ab40f None None] MFA Rules not processed for user `cb740ed7f7684b079e45f887be83f795`. Rule list: `[]` (Enabled: `True`). {{(pid=81409) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 13 04:27:17.049625 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.fernet_utils [None req-2500f465-c3db-41e4-bb38-8c1bae7ab40f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81409) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:27:17.120883 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 1076/2122] 10.222.0.27 () {58 vars in 1018 bytes} [Thu Oct 13 04:27:17 2022] POST /identity/v3/auth/tokens => generated 3372 bytes in 96 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 13 04:27:17.133354 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-699c5392-ae35-4a7a-a23d-8fd566726303 None None] Authenticating user token {{(pid=81408) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:27:17.135065 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.fernet_utils [None req-699c5392-ae35-4a7a-a23d-8fd566726303 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81408) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:27:17.194026 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-699c5392-ae35-4a7a-a23d-8fd566726303 None None] Validating token access rules against request {{(pid=81408) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:27:17.197503 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f167c9a-61cc-49ca-b265-966a22cb74d1 service glance] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a6700f49b86543f6a5b92cad8faaba42', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dcec0431fb6849499d6aec35a234daa6', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81408) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:27:17.198127 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f167c9a-61cc-49ca-b265-966a22cb74d1 service glance] REQUEST_METHOD: `GET` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:27:17.198431 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f167c9a-61cc-49ca-b265-966a22cb74d1 service glance] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:27:17.198701 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f167c9a-61cc-49ca-b265-966a22cb74d1 service glance] PATH_INFO: `/v3/auth/tokens` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:27:17.209249 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5f167c9a-61cc-49ca-b265-966a22cb74d1 service glance] RBAC: Authorizing `identity:validate_token()` {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:27:17.210709 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5f167c9a-61cc-49ca-b265-966a22cb74d1 service glance] RBAC: Authorization granted {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:27:17.221699 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 1047/2123] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Oct 13 04:27:17 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 89 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Oct 13 04:27:18.129102 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2500f465-c3db-41e4-bb38-8c1bae7ab40f None None] Authenticating user token {{(pid=81409) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:27:18.138521 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2500f465-c3db-41e4-bb38-8c1bae7ab40f None None] Validating token access rules against request {{(pid=81409) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:27:18.142328 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2527e749-2417-4c5e-bb8b-ed4fa251c0cc service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6141b8a6c3d54e179d6879bb019f4d12', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dcec0431fb6849499d6aec35a234daa6', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81409) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:27:18.143153 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2527e749-2417-4c5e-bb8b-ed4fa251c0cc service cinder] REQUEST_METHOD: `GET` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:27:18.143525 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2527e749-2417-4c5e-bb8b-ed4fa251c0cc service cinder] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:27:18.143900 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2527e749-2417-4c5e-bb8b-ed4fa251c0cc service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:27:18.209045 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2527e749-2417-4c5e-bb8b-ed4fa251c0cc service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:27:18.210491 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2527e749-2417-4c5e-bb8b-ed4fa251c0cc service cinder] RBAC: Authorization granted {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:27:18.219535 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 1077/2124] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Oct 13 04:27:18 2022] GET /identity/v3/auth/tokens => generated 3260 bytes in 91 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 13 04:27:18.781491 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f167c9a-61cc-49ca-b265-966a22cb74d1 service glance] Authenticating user token {{(pid=81408) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:27:18.791431 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f167c9a-61cc-49ca-b265-966a22cb74d1 service glance] Validating token access rules against request {{(pid=81408) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:27:18.795019 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-768b0c63-ce74-47c1-bb8c-7603e904a929 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6141b8a6c3d54e179d6879bb019f4d12', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dcec0431fb6849499d6aec35a234daa6', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81408) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:27:18.795682 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-768b0c63-ce74-47c1-bb8c-7603e904a929 service cinder] REQUEST_METHOD: `GET` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:27:18.795961 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-768b0c63-ce74-47c1-bb8c-7603e904a929 service cinder] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:27:18.796229 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-768b0c63-ce74-47c1-bb8c-7603e904a929 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:27:18.805711 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-768b0c63-ce74-47c1-bb8c-7603e904a929 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:27:18.807158 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-768b0c63-ce74-47c1-bb8c-7603e904a929 service cinder] RBAC: Authorization granted {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:27:18.817549 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 1048/2125] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Oct 13 04:27:18 2022] GET /identity/v3/auth/tokens => generated 3329 bytes in 37 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 13 04:27:19.342894 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1005e2a-533d-48e5-96bc-bc01d5cdf5ef None None] REQUEST_METHOD: `POST` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:27:19.343375 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1005e2a-533d-48e5-96bc-bc01d5cdf5ef None None] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:27:19.343468 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1005e2a-533d-48e5-96bc-bc01d5cdf5ef None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:27:19.363662 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.auth.core [None req-c1005e2a-533d-48e5-96bc-bc01d5cdf5ef None None] MFA Rules not processed for user `487b1605fd4143cfaa422a3a7a0344c7`. Rule list: `[]` (Enabled: `True`). {{(pid=81409) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 13 04:27:19.364895 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.fernet_utils [None req-c1005e2a-533d-48e5-96bc-bc01d5cdf5ef None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81409) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:27:19.422177 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 1078/2126] 10.222.0.27 () {58 vars in 1009 bytes} [Thu Oct 13 04:27:19 2022] POST /identity/v3/auth/tokens => generated 3195 bytes in 80 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 13 04:27:19.434431 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-768b0c63-ce74-47c1-bb8c-7603e904a929 service cinder] Authenticating user token {{(pid=81408) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:27:19.435825 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.fernet_utils [None req-768b0c63-ce74-47c1-bb8c-7603e904a929 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81408) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:27:19.465248 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG dbcounter [-] [81408] Writing DB stats keystone:SELECT=100 {{(pid=81408) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:27:19.508833 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-768b0c63-ce74-47c1-bb8c-7603e904a929 service cinder] Validating token access rules against request {{(pid=81408) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:27:19.513835 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5e6de42b-3517-4a40-bd0f-2a8da089fe25 service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90a0f466b664458898cf9dc4112ae651', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dcec0431fb6849499d6aec35a234daa6', 'project_domain_id': 'default', 'roles': ['service', 'member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81408) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:27:19.514474 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e6de42b-3517-4a40-bd0f-2a8da089fe25 service swift] REQUEST_METHOD: `GET` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:27:19.514748 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e6de42b-3517-4a40-bd0f-2a8da089fe25 service swift] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:27:19.515017 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e6de42b-3517-4a40-bd0f-2a8da089fe25 service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:27:19.523624 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5e6de42b-3517-4a40-bd0f-2a8da089fe25 service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:27:19.525202 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5e6de42b-3517-4a40-bd0f-2a8da089fe25 service swift] RBAC: Authorization granted {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:27:19.534091 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 1049/2127] 10.222.0.27 () {60 vars in 1384 bytes} [Thu Oct 13 04:27:19 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 100 msecs (HTTP/1.1 200) 6 headers in 379 bytes (1 switches on core 0) Oct 13 04:27:22.790439 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c1005e2a-533d-48e5-96bc-bc01d5cdf5ef None None] Authenticating user token {{(pid=81409) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:27:22.814855 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG dbcounter [-] [81409] Writing DB stats keystone:SELECT=100 {{(pid=81409) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:27:22.865329 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c1005e2a-533d-48e5-96bc-bc01d5cdf5ef None None] Validating token access rules against request {{(pid=81409) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:27:22.869870 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f15436c8-b13f-4235-b6e1-9d380984fb68 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8ba823f313144da3b683bacef8fd62e0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81409) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:27:22.870538 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f15436c8-b13f-4235-b6e1-9d380984fb68 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:27:22.870832 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f15436c8-b13f-4235-b6e1-9d380984fb68 admin admin] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:27:22.871082 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f15436c8-b13f-4235-b6e1-9d380984fb68 admin admin] PATH_INFO: `/v3/users/6efd2fa33b724a429d2146ad6d5abe6a` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:27:22.872294 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f15436c8-b13f-4235-b6e1-9d380984fb68 admin admin] RBAC: Authorizing `identity:delete_user(user_id=6efd2fa33b724a429d2146ad6d5abe6a)` {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:27:22.874242 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f15436c8-b13f-4235-b6e1-9d380984fb68 admin admin] RBAC: Authorization granted {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:27:22.930419 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.notifications [None req-f15436c8-b13f-4235-b6e1-9d380984fb68 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '6efd2fa33b724a429d2146ad6d5abe6a'} {{(pid=81409) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 13 04:27:22.931462 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.notifications [None req-f15436c8-b13f-4235-b6e1-9d380984fb68 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '6efd2fa33b724a429d2146ad6d5abe6a'} {{(pid=81409) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 13 04:27:22.939107 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.notifications [None req-f15436c8-b13f-4235-b6e1-9d380984fb68 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '6efd2fa33b724a429d2146ad6d5abe6a'} {{(pid=81409) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 13 04:27:22.939766 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.notifications [None req-f15436c8-b13f-4235-b6e1-9d380984fb68 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '6efd2fa33b724a429d2146ad6d5abe6a'} {{(pid=81409) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 13 04:27:22.949007 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.notifications [None req-f15436c8-b13f-4235-b6e1-9d380984fb68 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '6efd2fa33b724a429d2146ad6d5abe6a'} {{(pid=81409) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 13 04:27:22.967953 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 1079/2128] 10.222.0.27 () {58 vars in 1224 bytes} [Thu Oct 13 04:27:22 2022] DELETE /identity/v3/users/6efd2fa33b724a429d2146ad6d5abe6a => generated 0 bytes in 178 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 13 04:27:22.974979 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1951584-8071-4b49-bd8d-ced48285c1fc None None] REQUEST_METHOD: `POST` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:27:22.975504 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1951584-8071-4b49-bd8d-ced48285c1fc None None] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:27:22.975504 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1951584-8071-4b49-bd8d-ced48285c1fc None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:27:23.005396 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.auth.core [None req-c1951584-8071-4b49-bd8d-ced48285c1fc None None] MFA Rules not processed for user `0e2d979563d6421995451a0e86bf3022`. Rule list: `[]` (Enabled: `True`). {{(pid=81408) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 13 04:27:23.006859 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.fernet_utils [None req-c1951584-8071-4b49-bd8d-ced48285c1fc None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81408) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:27:23.071213 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 1050/2129] 10.222.0.27 () {58 vars in 958 bytes} [Thu Oct 13 04:27:22 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 99 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 13 04:27:23.081181 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f15436c8-b13f-4235-b6e1-9d380984fb68 admin admin] Authenticating user token {{(pid=81409) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:27:23.083090 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.fernet_utils [None req-f15436c8-b13f-4235-b6e1-9d380984fb68 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81409) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:27:23.145980 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f15436c8-b13f-4235-b6e1-9d380984fb68 admin admin] Validating token access rules against request {{(pid=81409) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:27:23.150532 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-89cccfc5-d23c-4624-97c5-8a2e2336444b service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd8812c3974f84871bb2d761cf0016257', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dcec0431fb6849499d6aec35a234daa6', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81409) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:27:23.151146 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89cccfc5-d23c-4624-97c5-8a2e2336444b service neutron] REQUEST_METHOD: `GET` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:27:23.151420 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89cccfc5-d23c-4624-97c5-8a2e2336444b service neutron] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:27:23.151687 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89cccfc5-d23c-4624-97c5-8a2e2336444b service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:27:23.161940 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-89cccfc5-d23c-4624-97c5-8a2e2336444b service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:27:23.163405 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-89cccfc5-d23c-4624-97c5-8a2e2336444b service neutron] RBAC: Authorization granted {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:27:23.173259 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 1080/2130] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Oct 13 04:27:23 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 92 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 13 04:27:23.522490 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c1951584-8071-4b49-bd8d-ced48285c1fc None None] Authenticating user token {{(pid=81408) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:27:23.524737 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.fernet_utils [None req-c1951584-8071-4b49-bd8d-ced48285c1fc None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81408) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:27:23.543534 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c1951584-8071-4b49-bd8d-ced48285c1fc None None] Validating token access rules against request {{(pid=81408) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:27:23.554183 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ab708e8f-63ad-4a07-b94d-1d77c83b9d14 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8ba823f313144da3b683bacef8fd62e0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81408) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:27:23.554891 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab708e8f-63ad-4a07-b94d-1d77c83b9d14 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:27:23.555175 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab708e8f-63ad-4a07-b94d-1d77c83b9d14 admin admin] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:27:23.555446 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab708e8f-63ad-4a07-b94d-1d77c83b9d14 admin admin] PATH_INFO: `/v3/projects/ec37e41bc87b4359b740f2e9f382d4b5` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:27:23.556619 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ab708e8f-63ad-4a07-b94d-1d77c83b9d14 admin admin] RBAC: Authorizing `identity:delete_project(project_id=ec37e41bc87b4359b740f2e9f382d4b5)` {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:27:23.558615 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ab708e8f-63ad-4a07-b94d-1d77c83b9d14 admin admin] RBAC: Authorization granted {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:27:23.623371 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.notifications [None req-ab708e8f-63ad-4a07-b94d-1d77c83b9d14 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'ec37e41bc87b4359b740f2e9f382d4b5', 'request_id': 'req-ab708e8f-63ad-4a07-b94d-1d77c83b9d14'} {{(pid=81408) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 13 04:27:23.631388 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.notifications [None req-ab708e8f-63ad-4a07-b94d-1d77c83b9d14 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'ec37e41bc87b4359b740f2e9f382d4b5', 'request_id': 'req-ab708e8f-63ad-4a07-b94d-1d77c83b9d14'} {{(pid=81408) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 13 04:27:23.638560 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.notifications [None req-ab708e8f-63ad-4a07-b94d-1d77c83b9d14 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'ec37e41bc87b4359b740f2e9f382d4b5', 'request_id': 'req-ab708e8f-63ad-4a07-b94d-1d77c83b9d14'} {{(pid=81408) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 13 04:27:23.650555 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.notifications [None req-ab708e8f-63ad-4a07-b94d-1d77c83b9d14 admin admin] The token cache is being invalidate because project ec37e41bc87b4359b740f2e9f382d4b5 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81408) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 13 04:27:23.650917 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.notifications [None req-ab708e8f-63ad-4a07-b94d-1d77c83b9d14 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81408) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 13 04:27:23.651664 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.notifications [None req-ab708e8f-63ad-4a07-b94d-1d77c83b9d14 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81408) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 13 04:27:23.656900 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 1051/2131] 10.222.0.27 () {58 vars in 1233 bytes} [Thu Oct 13 04:27:23 2022] DELETE /identity/v3/projects/ec37e41bc87b4359b740f2e9f382d4b5 => generated 0 bytes in 133 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 13 04:27:23.658504 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-89cccfc5-d23c-4624-97c5-8a2e2336444b service neutron] Authenticating user token {{(pid=81409) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:27:23.669516 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-89cccfc5-d23c-4624-97c5-8a2e2336444b service neutron] Validating token access rules against request {{(pid=81409) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:27:23.712755 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG dbcounter [-] [81409] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81409) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:27:23.732784 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9874e00d-076a-472d-a041-1cb0439cc26d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8ba823f313144da3b683bacef8fd62e0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81409) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:27:23.733413 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9874e00d-076a-472d-a041-1cb0439cc26d admin admin] REQUEST_METHOD: `DELETE` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:27:23.733681 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9874e00d-076a-472d-a041-1cb0439cc26d admin admin] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:27:23.733945 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9874e00d-076a-472d-a041-1cb0439cc26d admin admin] PATH_INFO: `/v3/users/2569a8f7753c4c6e8db4197e250b11aa` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:27:23.735117 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9874e00d-076a-472d-a041-1cb0439cc26d admin admin] RBAC: Authorizing `identity:delete_user(user_id=2569a8f7753c4c6e8db4197e250b11aa)` {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:27:23.737045 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9874e00d-076a-472d-a041-1cb0439cc26d admin admin] RBAC: Authorization granted {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:27:23.798841 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.notifications [None req-9874e00d-076a-472d-a041-1cb0439cc26d admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '2569a8f7753c4c6e8db4197e250b11aa'} {{(pid=81409) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 13 04:27:23.799907 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.notifications [None req-9874e00d-076a-472d-a041-1cb0439cc26d admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '2569a8f7753c4c6e8db4197e250b11aa'} {{(pid=81409) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 13 04:27:23.808082 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.notifications [None req-9874e00d-076a-472d-a041-1cb0439cc26d admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '2569a8f7753c4c6e8db4197e250b11aa'} {{(pid=81409) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 13 04:27:23.809030 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.notifications [None req-9874e00d-076a-472d-a041-1cb0439cc26d admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '2569a8f7753c4c6e8db4197e250b11aa'} {{(pid=81409) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 13 04:27:23.824337 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.notifications [None req-9874e00d-076a-472d-a041-1cb0439cc26d admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '2569a8f7753c4c6e8db4197e250b11aa'} {{(pid=81409) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 13 04:27:23.847232 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 1081/2132] 10.222.0.27 () {58 vars in 1224 bytes} [Thu Oct 13 04:27:23 2022] DELETE /identity/v3/users/2569a8f7753c4c6e8db4197e250b11aa => generated 0 bytes in 189 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 13 04:27:24.148221 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ab708e8f-63ad-4a07-b94d-1d77c83b9d14 admin admin] Authenticating user token {{(pid=81408) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:27:24.150057 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.fernet_utils [None req-ab708e8f-63ad-4a07-b94d-1d77c83b9d14 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81408) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:27:24.165859 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ab708e8f-63ad-4a07-b94d-1d77c83b9d14 admin admin] Validating token access rules against request {{(pid=81408) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:27:24.209968 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG dbcounter [-] [81408] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=81408) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:27:24.239077 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1fab825d-9296-40f1-bd37-db726874bda7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8ba823f313144da3b683bacef8fd62e0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81408) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:27:24.239734 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1fab825d-9296-40f1-bd37-db726874bda7 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:27:24.240044 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1fab825d-9296-40f1-bd37-db726874bda7 admin admin] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:27:24.240317 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1fab825d-9296-40f1-bd37-db726874bda7 admin admin] PATH_INFO: `/v3/projects/593752e699e643cbbeda50363909f091` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:27:24.241478 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1fab825d-9296-40f1-bd37-db726874bda7 admin admin] RBAC: Authorizing `identity:delete_project(project_id=593752e699e643cbbeda50363909f091)` {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:27:24.243491 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1fab825d-9296-40f1-bd37-db726874bda7 admin admin] RBAC: Authorization granted {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:27:24.299119 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.notifications [None req-1fab825d-9296-40f1-bd37-db726874bda7 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '593752e699e643cbbeda50363909f091', 'request_id': 'req-1fab825d-9296-40f1-bd37-db726874bda7'} {{(pid=81408) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 13 04:27:24.308521 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.notifications [None req-1fab825d-9296-40f1-bd37-db726874bda7 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '593752e699e643cbbeda50363909f091', 'request_id': 'req-1fab825d-9296-40f1-bd37-db726874bda7'} {{(pid=81408) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 13 04:27:24.314611 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.notifications [None req-1fab825d-9296-40f1-bd37-db726874bda7 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '593752e699e643cbbeda50363909f091', 'request_id': 'req-1fab825d-9296-40f1-bd37-db726874bda7'} {{(pid=81408) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 13 04:27:24.333595 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.notifications [None req-1fab825d-9296-40f1-bd37-db726874bda7 admin admin] The token cache is being invalidate because project 593752e699e643cbbeda50363909f091 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81408) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 13 04:27:24.334048 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.notifications [None req-1fab825d-9296-40f1-bd37-db726874bda7 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81408) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 13 04:27:24.334857 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.notifications [None req-1fab825d-9296-40f1-bd37-db726874bda7 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81408) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 13 04:27:24.337265 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 1052/2133] 10.222.0.27 () {58 vars in 1233 bytes} [Thu Oct 13 04:27:24 2022] DELETE /identity/v3/projects/593752e699e643cbbeda50363909f091 => generated 0 bytes in 189 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 13 04:27:24.341016 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9874e00d-076a-472d-a041-1cb0439cc26d admin admin] Authenticating user token {{(pid=81409) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:27:24.342609 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.fernet_utils [None req-9874e00d-076a-472d-a041-1cb0439cc26d admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81409) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:27:24.355647 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9874e00d-076a-472d-a041-1cb0439cc26d admin admin] Validating token access rules against request {{(pid=81409) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:27:24.427656 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-965f7fb4-d60f-49eb-b3bd-890385b8451a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8ba823f313144da3b683bacef8fd62e0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81409) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:27:24.428398 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-965f7fb4-d60f-49eb-b3bd-890385b8451a admin admin] REQUEST_METHOD: `DELETE` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:27:24.428769 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-965f7fb4-d60f-49eb-b3bd-890385b8451a admin admin] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:27:24.429084 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-965f7fb4-d60f-49eb-b3bd-890385b8451a admin admin] PATH_INFO: `/v3/users/f966e72aa2b64aec96e80edb9134a26d` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:27:24.430474 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-965f7fb4-d60f-49eb-b3bd-890385b8451a admin admin] RBAC: Authorizing `identity:delete_user(user_id=f966e72aa2b64aec96e80edb9134a26d)` {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:27:24.432873 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-965f7fb4-d60f-49eb-b3bd-890385b8451a admin admin] RBAC: Authorization granted {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:27:24.496393 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.notifications [None req-965f7fb4-d60f-49eb-b3bd-890385b8451a admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'f966e72aa2b64aec96e80edb9134a26d'} {{(pid=81409) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 13 04:27:24.497138 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.notifications [None req-965f7fb4-d60f-49eb-b3bd-890385b8451a admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'f966e72aa2b64aec96e80edb9134a26d'} {{(pid=81409) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 13 04:27:24.508672 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.notifications [None req-965f7fb4-d60f-49eb-b3bd-890385b8451a admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'f966e72aa2b64aec96e80edb9134a26d'} {{(pid=81409) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 13 04:27:24.509490 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.notifications [None req-965f7fb4-d60f-49eb-b3bd-890385b8451a admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'f966e72aa2b64aec96e80edb9134a26d'} {{(pid=81409) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 13 04:27:24.519182 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.notifications [None req-965f7fb4-d60f-49eb-b3bd-890385b8451a admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'f966e72aa2b64aec96e80edb9134a26d'} {{(pid=81409) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 13 04:27:24.520539 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG dbcounter [-] [81409] Writing DB stats keystone:SELECT=80,keystone:DELETE=18,keystone:INSERT=2 {{(pid=81409) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:27:24.554749 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 1082/2134] 10.222.0.27 () {58 vars in 1224 bytes} [Thu Oct 13 04:27:24 2022] DELETE /identity/v3/users/f966e72aa2b64aec96e80edb9134a26d => generated 0 bytes in 211 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 13 04:27:24.860858 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1fab825d-9296-40f1-bd37-db726874bda7 admin admin] Authenticating user token {{(pid=81408) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:27:24.862344 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.fernet_utils [None req-1fab825d-9296-40f1-bd37-db726874bda7 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81408) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:27:24.887139 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1fab825d-9296-40f1-bd37-db726874bda7 admin admin] Validating token access rules against request {{(pid=81408) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:27:24.956843 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b80ca6cd-03c5-48ec-ab71-c6ebfbcb6ac6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8ba823f313144da3b683bacef8fd62e0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81408) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:27:24.957479 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b80ca6cd-03c5-48ec-ab71-c6ebfbcb6ac6 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:27:24.957759 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b80ca6cd-03c5-48ec-ab71-c6ebfbcb6ac6 admin admin] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:27:24.958028 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b80ca6cd-03c5-48ec-ab71-c6ebfbcb6ac6 admin admin] PATH_INFO: `/v3/projects/26c8523b6281498896eee7d7de498aca` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:27:24.959181 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b80ca6cd-03c5-48ec-ab71-c6ebfbcb6ac6 admin admin] RBAC: Authorizing `identity:delete_project(project_id=26c8523b6281498896eee7d7de498aca)` {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:27:24.961167 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b80ca6cd-03c5-48ec-ab71-c6ebfbcb6ac6 admin admin] RBAC: Authorization granted {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:27:25.011377 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.notifications [None req-b80ca6cd-03c5-48ec-ab71-c6ebfbcb6ac6 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '26c8523b6281498896eee7d7de498aca', 'request_id': 'req-b80ca6cd-03c5-48ec-ab71-c6ebfbcb6ac6'} {{(pid=81408) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 13 04:27:25.019732 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.notifications [None req-b80ca6cd-03c5-48ec-ab71-c6ebfbcb6ac6 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '26c8523b6281498896eee7d7de498aca', 'request_id': 'req-b80ca6cd-03c5-48ec-ab71-c6ebfbcb6ac6'} {{(pid=81408) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 13 04:27:25.026757 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.notifications [None req-b80ca6cd-03c5-48ec-ab71-c6ebfbcb6ac6 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '26c8523b6281498896eee7d7de498aca', 'request_id': 'req-b80ca6cd-03c5-48ec-ab71-c6ebfbcb6ac6'} {{(pid=81408) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 13 04:27:25.034048 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG dbcounter [-] [81408] Writing DB stats keystone:SELECT=84,keystone:DELETE=14,keystone:INSERT=2 {{(pid=81408) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:27:25.039311 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.notifications [None req-b80ca6cd-03c5-48ec-ab71-c6ebfbcb6ac6 admin admin] The token cache is being invalidate because project 26c8523b6281498896eee7d7de498aca was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81408) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 13 04:27:25.039867 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.notifications [None req-b80ca6cd-03c5-48ec-ab71-c6ebfbcb6ac6 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81408) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 13 04:27:25.041074 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.notifications [None req-b80ca6cd-03c5-48ec-ab71-c6ebfbcb6ac6 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81408) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 13 04:27:25.046246 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 1053/2135] 10.222.0.27 () {58 vars in 1233 bytes} [Thu Oct 13 04:27:24 2022] DELETE /identity/v3/projects/26c8523b6281498896eee7d7de498aca => generated 0 bytes in 183 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 13 04:27:25.049337 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95835b69-8e95-48d4-96cc-98d1343abd08 None None] REQUEST_METHOD: `POST` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:27:25.049838 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95835b69-8e95-48d4-96cc-98d1343abd08 None None] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:27:25.049982 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95835b69-8e95-48d4-96cc-98d1343abd08 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:27:25.071893 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.auth.core [None req-95835b69-8e95-48d4-96cc-98d1343abd08 None None] MFA Rules not processed for user `0e2d979563d6421995451a0e86bf3022`. Rule list: `[]` (Enabled: `True`). {{(pid=81409) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 13 04:27:25.073126 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.fernet_utils [None req-95835b69-8e95-48d4-96cc-98d1343abd08 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81409) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:27:25.139256 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 1083/2136] 10.222.0.27 () {58 vars in 958 bytes} [Thu Oct 13 04:27:25 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 91 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 13 04:27:25.147265 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a99c84d-c6d0-4b4a-b0e7-79a3a190b829 None None] REQUEST_METHOD: `POST` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:27:25.147764 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a99c84d-c6d0-4b4a-b0e7-79a3a190b829 None None] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:27:25.147891 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a99c84d-c6d0-4b4a-b0e7-79a3a190b829 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:27:25.170249 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.auth.core [None req-8a99c84d-c6d0-4b4a-b0e7-79a3a190b829 None None] MFA Rules not processed for user `0e2d979563d6421995451a0e86bf3022`. Rule list: `[]` (Enabled: `True`). {{(pid=81408) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 13 04:27:25.171593 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.fernet_utils [None req-8a99c84d-c6d0-4b4a-b0e7-79a3a190b829 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81408) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:27:25.181955 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 1054/2137] 10.222.0.27 () {58 vars in 958 bytes} [Thu Oct 13 04:27:25 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 36 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 13 04:27:25.186188 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-95835b69-8e95-48d4-96cc-98d1343abd08 None None] Authenticating user token {{(pid=81409) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:27:25.197306 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-95835b69-8e95-48d4-96cc-98d1343abd08 None None] Validating token access rules against request {{(pid=81409) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:27:25.201971 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f25a384c-7a39-452b-a8c2-9a2d1b96212a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8ba823f313144da3b683bacef8fd62e0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81409) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:27:25.202611 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f25a384c-7a39-452b-a8c2-9a2d1b96212a admin admin] REQUEST_METHOD: `GET` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:27:25.202880 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f25a384c-7a39-452b-a8c2-9a2d1b96212a admin admin] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:27:25.203147 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f25a384c-7a39-452b-a8c2-9a2d1b96212a admin admin] PATH_INFO: `/v3/domains` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:27:25.203792 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f25a384c-7a39-452b-a8c2-9a2d1b96212a admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81409) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 13 04:27:25.204077 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f25a384c-7a39-452b-a8c2-9a2d1b96212a admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:27:25.205686 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f25a384c-7a39-452b-a8c2-9a2d1b96212a admin admin] RBAC: Authorization granted {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:27:25.221725 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 1084/2138] 10.222.0.27 () {58 vars in 1153 bytes} [Thu Oct 13 04:27:25 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 33 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 13 04:27:25.223327 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8a99c84d-c6d0-4b4a-b0e7-79a3a190b829 None None] Authenticating user token {{(pid=81408) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:27:25.234144 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8a99c84d-c6d0-4b4a-b0e7-79a3a190b829 None None] Validating token access rules against request {{(pid=81408) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:27:25.239437 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-85f48778-1545-41c1-a793-43a2e5697ef4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8ba823f313144da3b683bacef8fd62e0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81408) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:27:25.240034 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85f48778-1545-41c1-a793-43a2e5697ef4 admin admin] REQUEST_METHOD: `POST` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:27:25.240313 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85f48778-1545-41c1-a793-43a2e5697ef4 admin admin] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:27:25.240583 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85f48778-1545-41c1-a793-43a2e5697ef4 admin admin] PATH_INFO: `/v3/projects` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:27:25.241220 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-85f48778-1545-41c1-a793-43a2e5697ef4 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:27:25.243215 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-85f48778-1545-41c1-a793-43a2e5697ef4 admin admin] RBAC: Authorization granted {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:27:25.274240 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 1055/2139] 10.222.0.27 () {60 vars in 1153 bytes} [Thu Oct 13 04:27:25 2022] POST /identity/v3/projects => generated 383 bytes in 51 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 13 04:27:25.278226 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f25a384c-7a39-452b-a8c2-9a2d1b96212a admin admin] Authenticating user token {{(pid=81409) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:27:25.288360 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f25a384c-7a39-452b-a8c2-9a2d1b96212a admin admin] Validating token access rules against request {{(pid=81409) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:27:25.359089 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bd4a99f8-c3dd-4610-b157-92e8a0b88c06 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8ba823f313144da3b683bacef8fd62e0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81409) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:27:25.359822 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd4a99f8-c3dd-4610-b157-92e8a0b88c06 admin admin] REQUEST_METHOD: `POST` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:27:25.360196 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd4a99f8-c3dd-4610-b157-92e8a0b88c06 admin admin] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:27:25.360559 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd4a99f8-c3dd-4610-b157-92e8a0b88c06 admin admin] PATH_INFO: `/v3/users` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:27:25.361267 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bd4a99f8-c3dd-4610-b157-92e8a0b88c06 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:27:25.363995 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bd4a99f8-c3dd-4610-b157-92e8a0b88c06 admin admin] RBAC: Authorization granted {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:27:25.394259 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 1085/2140] 10.222.0.27 () {60 vars in 1144 bytes} [Thu Oct 13 04:27:25 2022] POST /identity/v3/users => generated 292 bytes in 113 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 13 04:27:25.395552 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-85f48778-1545-41c1-a793-43a2e5697ef4 admin admin] Authenticating user token {{(pid=81408) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:27:25.406119 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-85f48778-1545-41c1-a793-43a2e5697ef4 admin admin] Validating token access rules against request {{(pid=81408) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:27:25.414837 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-64a15ae1-1820-43c7-9e11-a103e83283a4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8ba823f313144da3b683bacef8fd62e0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81408) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:27:25.415653 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64a15ae1-1820-43c7-9e11-a103e83283a4 admin admin] REQUEST_METHOD: `GET` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:27:25.415990 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64a15ae1-1820-43c7-9e11-a103e83283a4 admin admin] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:27:25.416329 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64a15ae1-1820-43c7-9e11-a103e83283a4 admin admin] PATH_INFO: `/v3/roles` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:27:25.417096 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-64a15ae1-1820-43c7-9e11-a103e83283a4 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:27:25.418769 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-64a15ae1-1820-43c7-9e11-a103e83283a4 admin admin] RBAC: Authorization granted {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:27:25.431481 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 1056/2141] 10.222.0.27 () {58 vars in 1122 bytes} [Thu Oct 13 04:27:25 2022] GET /identity/v3/roles => generated 1420 bytes in 36 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 13 04:27:25.435436 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bd4a99f8-c3dd-4610-b157-92e8a0b88c06 admin admin] Authenticating user token {{(pid=81409) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:27:25.447117 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bd4a99f8-c3dd-4610-b157-92e8a0b88c06 admin admin] Validating token access rules against request {{(pid=81409) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:27:25.453850 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0fb4691b-085c-4ce0-8995-7ff771a8e779 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8ba823f313144da3b683bacef8fd62e0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81409) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:27:25.454520 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0fb4691b-085c-4ce0-8995-7ff771a8e779 admin admin] REQUEST_METHOD: `GET` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:27:25.454792 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0fb4691b-085c-4ce0-8995-7ff771a8e779 admin admin] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:27:25.455061 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0fb4691b-085c-4ce0-8995-7ff771a8e779 admin admin] PATH_INFO: `/v3/roles` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:27:25.455699 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0fb4691b-085c-4ce0-8995-7ff771a8e779 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:27:25.457344 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0fb4691b-085c-4ce0-8995-7ff771a8e779 admin admin] RBAC: Authorization granted {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:27:25.469549 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 1086/2142] 10.222.0.27 () {58 vars in 1122 bytes} [Thu Oct 13 04:27:25 2022] GET /identity/v3/roles => generated 1420 bytes in 34 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 13 04:27:25.474444 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-64a15ae1-1820-43c7-9e11-a103e83283a4 admin admin] Authenticating user token {{(pid=81408) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:27:25.488815 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-64a15ae1-1820-43c7-9e11-a103e83283a4 admin admin] Validating token access rules against request {{(pid=81408) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:27:25.493701 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-91ade47b-5780-4897-b2c3-2ea1654cde36 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8ba823f313144da3b683bacef8fd62e0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81408) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:27:25.494379 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91ade47b-5780-4897-b2c3-2ea1654cde36 admin admin] REQUEST_METHOD: `PUT` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:27:25.494577 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91ade47b-5780-4897-b2c3-2ea1654cde36 admin admin] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:27:25.494848 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91ade47b-5780-4897-b2c3-2ea1654cde36 admin admin] PATH_INFO: `/v3/projects/7acebb735f7d4aca8f7728e25af182de/users/24bfc234aebf45f1a868b1ae593ba32b/roles/cc56c4ae13a7424ea771c15da4b153c1` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:27:25.511347 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-91ade47b-5780-4897-b2c3-2ea1654cde36 admin admin] RBAC: Authorizing `identity:create_grant(project_id=7acebb735f7d4aca8f7728e25af182de, user_id=24bfc234aebf45f1a868b1ae593ba32b, role_id=cc56c4ae13a7424ea771c15da4b153c1)` {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:27:25.518373 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-91ade47b-5780-4897-b2c3-2ea1654cde36 admin admin] RBAC: Authorization granted {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:27:25.535333 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 1057/2143] 10.222.0.27 () {60 vars in 1483 bytes} [Thu Oct 13 04:27:25 2022] PUT /identity/v3/projects/7acebb735f7d4aca8f7728e25af182de/users/24bfc234aebf45f1a868b1ae593ba32b/roles/cc56c4ae13a7424ea771c15da4b153c1 => generated 0 bytes in 62 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 13 04:27:25.547253 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e5550f3b-3a7a-43f6-b71d-e46d77877248 None None] REQUEST_METHOD: `POST` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:27:25.547563 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e5550f3b-3a7a-43f6-b71d-e46d77877248 None None] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:27:25.547812 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e5550f3b-3a7a-43f6-b71d-e46d77877248 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:27:25.578685 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.auth.core [None req-e5550f3b-3a7a-43f6-b71d-e46d77877248 None None] MFA Rules not processed for user `24bfc234aebf45f1a868b1ae593ba32b`. Rule list: `[]` (Enabled: `True`). {{(pid=81409) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 13 04:27:25.579919 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.fernet_utils [None req-e5550f3b-3a7a-43f6-b71d-e46d77877248 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81409) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:27:25.616597 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG dbcounter [-] [81409] Writing DB stats keystone:SELECT=95,keystone:DELETE=2,keystone:INSERT=3 {{(pid=81409) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:27:25.655300 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 1087/2144] 10.222.0.27 () {58 vars in 958 bytes} [Thu Oct 13 04:27:25 2022] POST /identity/v3/auth/tokens => generated 3260 bytes in 109 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 13 04:27:25.670132 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-91ade47b-5780-4897-b2c3-2ea1654cde36 admin admin] Authenticating user token {{(pid=81408) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:27:25.672721 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.fernet_utils [None req-91ade47b-5780-4897-b2c3-2ea1654cde36 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81408) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:27:25.765451 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-91ade47b-5780-4897-b2c3-2ea1654cde36 admin admin] Validating token access rules against request {{(pid=81408) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:27:25.769301 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e268532a-38a7-471e-8a72-3289ceed3bdf service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6141b8a6c3d54e179d6879bb019f4d12', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dcec0431fb6849499d6aec35a234daa6', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81408) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:27:25.770188 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e268532a-38a7-471e-8a72-3289ceed3bdf service cinder] REQUEST_METHOD: `GET` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:27:25.770533 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e268532a-38a7-471e-8a72-3289ceed3bdf service cinder] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:27:25.771778 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e268532a-38a7-471e-8a72-3289ceed3bdf service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:27:25.782363 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e268532a-38a7-471e-8a72-3289ceed3bdf service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:27:25.784847 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e268532a-38a7-471e-8a72-3289ceed3bdf service cinder] RBAC: Authorization granted {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:27:25.804641 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 1058/2145] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Oct 13 04:27:25 2022] GET /identity/v3/auth/tokens => generated 3260 bytes in 135 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 13 04:27:28.472583 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e5550f3b-3a7a-43f6-b71d-e46d77877248 None None] Authenticating user token {{(pid=81409) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:27:28.539514 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e5550f3b-3a7a-43f6-b71d-e46d77877248 None None] Validating token access rules against request {{(pid=81409) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:27:28.543932 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-32e6b0af-8771-4b1b-8fb3-5678f7cc8c1c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8ba823f313144da3b683bacef8fd62e0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81409) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:27:28.544548 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32e6b0af-8771-4b1b-8fb3-5678f7cc8c1c admin admin] REQUEST_METHOD: `DELETE` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:27:28.544819 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32e6b0af-8771-4b1b-8fb3-5678f7cc8c1c admin admin] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:27:28.545085 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32e6b0af-8771-4b1b-8fb3-5678f7cc8c1c admin admin] PATH_INFO: `/v3/users/24bfc234aebf45f1a868b1ae593ba32b` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:27:28.546178 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-32e6b0af-8771-4b1b-8fb3-5678f7cc8c1c admin admin] RBAC: Authorizing `identity:delete_user(user_id=24bfc234aebf45f1a868b1ae593ba32b)` {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:27:28.548112 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-32e6b0af-8771-4b1b-8fb3-5678f7cc8c1c admin admin] RBAC: Authorization granted {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:27:28.594832 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.notifications [None req-32e6b0af-8771-4b1b-8fb3-5678f7cc8c1c admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '24bfc234aebf45f1a868b1ae593ba32b'} {{(pid=81409) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 13 04:27:28.595473 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.notifications [None req-32e6b0af-8771-4b1b-8fb3-5678f7cc8c1c admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '24bfc234aebf45f1a868b1ae593ba32b'} {{(pid=81409) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 13 04:27:28.603069 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.notifications [None req-32e6b0af-8771-4b1b-8fb3-5678f7cc8c1c admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '24bfc234aebf45f1a868b1ae593ba32b'} {{(pid=81409) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 13 04:27:28.603683 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.notifications [None req-32e6b0af-8771-4b1b-8fb3-5678f7cc8c1c admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '24bfc234aebf45f1a868b1ae593ba32b'} {{(pid=81409) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 13 04:27:28.610963 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.notifications [None req-32e6b0af-8771-4b1b-8fb3-5678f7cc8c1c admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '24bfc234aebf45f1a868b1ae593ba32b'} {{(pid=81409) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 13 04:27:28.628536 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 1088/2146] 10.222.0.27 () {58 vars in 1224 bytes} [Thu Oct 13 04:27:28 2022] DELETE /identity/v3/users/24bfc234aebf45f1a868b1ae593ba32b => generated 0 bytes in 156 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 13 04:27:28.632979 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8789e13-c6c4-416f-98ac-d2518e920536 None None] REQUEST_METHOD: `POST` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:27:28.633469 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8789e13-c6c4-416f-98ac-d2518e920536 None None] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:27:28.633682 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8789e13-c6c4-416f-98ac-d2518e920536 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:27:28.652183 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.auth.core [None req-d8789e13-c6c4-416f-98ac-d2518e920536 None None] MFA Rules not processed for user `0e2d979563d6421995451a0e86bf3022`. Rule list: `[]` (Enabled: `True`). {{(pid=81408) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 13 04:27:28.653289 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.fernet_utils [None req-d8789e13-c6c4-416f-98ac-d2518e920536 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81408) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:27:28.710436 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 1059/2147] 10.222.0.27 () {58 vars in 958 bytes} [Thu Oct 13 04:27:28 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 78 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 13 04:27:28.720065 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-32e6b0af-8771-4b1b-8fb3-5678f7cc8c1c admin admin] Authenticating user token {{(pid=81409) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:27:28.721446 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.fernet_utils [None req-32e6b0af-8771-4b1b-8fb3-5678f7cc8c1c admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81409) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:27:28.778448 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-32e6b0af-8771-4b1b-8fb3-5678f7cc8c1c admin admin] Validating token access rules against request {{(pid=81409) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:27:28.782030 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4ab4ecf0-bc96-40d7-b88f-795db9c1046c service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd8812c3974f84871bb2d761cf0016257', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dcec0431fb6849499d6aec35a234daa6', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81409) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:27:28.782661 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ab4ecf0-bc96-40d7-b88f-795db9c1046c service neutron] REQUEST_METHOD: `GET` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:27:28.782938 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ab4ecf0-bc96-40d7-b88f-795db9c1046c service neutron] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:27:28.783205 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ab4ecf0-bc96-40d7-b88f-795db9c1046c service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:27:28.792715 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4ab4ecf0-bc96-40d7-b88f-795db9c1046c service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:27:28.794123 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4ab4ecf0-bc96-40d7-b88f-795db9c1046c service neutron] RBAC: Authorization granted {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:27:28.797967 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG dbcounter [-] [81409] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81409) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:27:28.805496 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 1089/2148] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Oct 13 04:27:28 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 86 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 13 04:27:29.071228 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d8789e13-c6c4-416f-98ac-d2518e920536 None None] Authenticating user token {{(pid=81408) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:27:29.072721 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.fernet_utils [None req-d8789e13-c6c4-416f-98ac-d2518e920536 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81408) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:27:29.085810 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d8789e13-c6c4-416f-98ac-d2518e920536 None None] Validating token access rules against request {{(pid=81408) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:27:29.090437 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4561405b-51a1-4579-88a3-9165bec65288 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8ba823f313144da3b683bacef8fd62e0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81408) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:27:29.091066 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4561405b-51a1-4579-88a3-9165bec65288 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:27:29.091336 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4561405b-51a1-4579-88a3-9165bec65288 admin admin] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:27:29.091603 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4561405b-51a1-4579-88a3-9165bec65288 admin admin] PATH_INFO: `/v3/projects/7acebb735f7d4aca8f7728e25af182de` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:27:29.092716 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4561405b-51a1-4579-88a3-9165bec65288 admin admin] RBAC: Authorizing `identity:delete_project(project_id=7acebb735f7d4aca8f7728e25af182de)` {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:27:29.094671 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4561405b-51a1-4579-88a3-9165bec65288 admin admin] RBAC: Authorization granted {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:27:29.130406 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG dbcounter [-] [81408] Writing DB stats keystone:SELECT=96,keystone:INSERT=2,keystone:DELETE=2 {{(pid=81408) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:27:29.143608 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.notifications [None req-4561405b-51a1-4579-88a3-9165bec65288 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '7acebb735f7d4aca8f7728e25af182de', 'request_id': 'req-4561405b-51a1-4579-88a3-9165bec65288'} {{(pid=81408) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 13 04:27:29.151702 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.notifications [None req-4561405b-51a1-4579-88a3-9165bec65288 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '7acebb735f7d4aca8f7728e25af182de', 'request_id': 'req-4561405b-51a1-4579-88a3-9165bec65288'} {{(pid=81408) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 13 04:27:29.158023 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.notifications [None req-4561405b-51a1-4579-88a3-9165bec65288 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '7acebb735f7d4aca8f7728e25af182de', 'request_id': 'req-4561405b-51a1-4579-88a3-9165bec65288'} {{(pid=81408) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 13 04:27:29.169320 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.notifications [None req-4561405b-51a1-4579-88a3-9165bec65288 admin admin] The token cache is being invalidate because project 7acebb735f7d4aca8f7728e25af182de was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81408) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 13 04:27:29.169599 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.notifications [None req-4561405b-51a1-4579-88a3-9165bec65288 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81408) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 13 04:27:29.170242 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.notifications [None req-4561405b-51a1-4579-88a3-9165bec65288 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81408) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 13 04:27:29.171729 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 1060/2149] 10.222.0.27 () {58 vars in 1233 bytes} [Thu Oct 13 04:27:29 2022] DELETE /identity/v3/projects/7acebb735f7d4aca8f7728e25af182de => generated 0 bytes in 101 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 13 04:27:29.176558 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31123181-7f27-4895-adb5-3f928a2a8d4e None None] REQUEST_METHOD: `POST` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:27:29.176925 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31123181-7f27-4895-adb5-3f928a2a8d4e None None] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:27:29.177236 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31123181-7f27-4895-adb5-3f928a2a8d4e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:27:29.197315 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.auth.core [None req-31123181-7f27-4895-adb5-3f928a2a8d4e None None] MFA Rules not processed for user `0e2d979563d6421995451a0e86bf3022`. Rule list: `[]` (Enabled: `True`). {{(pid=81409) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 13 04:27:29.198488 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.fernet_utils [None req-31123181-7f27-4895-adb5-3f928a2a8d4e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81409) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:27:29.256731 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 1090/2150] 10.222.0.27 () {58 vars in 958 bytes} [Thu Oct 13 04:27:29 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 81 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 13 04:27:29.264446 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11a60b50-037e-43ad-ac63-9c1540d6c729 None None] REQUEST_METHOD: `POST` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:27:29.264820 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11a60b50-037e-43ad-ac63-9c1540d6c729 None None] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:27:29.265193 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11a60b50-037e-43ad-ac63-9c1540d6c729 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:27:29.285159 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.auth.core [None req-11a60b50-037e-43ad-ac63-9c1540d6c729 None None] MFA Rules not processed for user `0e2d979563d6421995451a0e86bf3022`. Rule list: `[]` (Enabled: `True`). {{(pid=81408) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 13 04:27:29.286927 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.fernet_utils [None req-11a60b50-037e-43ad-ac63-9c1540d6c729 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81408) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:27:29.294889 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 1061/2151] 10.222.0.27 () {58 vars in 958 bytes} [Thu Oct 13 04:27:29 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 31 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 13 04:27:29.299143 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-31123181-7f27-4895-adb5-3f928a2a8d4e None None] Authenticating user token {{(pid=81409) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:27:29.310357 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-31123181-7f27-4895-adb5-3f928a2a8d4e None None] Validating token access rules against request {{(pid=81409) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:27:29.315155 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c2c4260e-aee1-4049-a7c9-574ffbcb44b7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8ba823f313144da3b683bacef8fd62e0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81409) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:27:29.315762 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2c4260e-aee1-4049-a7c9-574ffbcb44b7 admin admin] REQUEST_METHOD: `GET` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:27:29.316037 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2c4260e-aee1-4049-a7c9-574ffbcb44b7 admin admin] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:27:29.316305 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2c4260e-aee1-4049-a7c9-574ffbcb44b7 admin admin] PATH_INFO: `/v3/domains` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:27:29.316946 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c2c4260e-aee1-4049-a7c9-574ffbcb44b7 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81409) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 13 04:27:29.317364 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c2c4260e-aee1-4049-a7c9-574ffbcb44b7 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:27:29.319028 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c2c4260e-aee1-4049-a7c9-574ffbcb44b7 admin admin] RBAC: Authorization granted {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:27:29.331758 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 1091/2152] 10.222.0.27 () {58 vars in 1153 bytes} [Thu Oct 13 04:27:29 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 33 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 13 04:27:29.336191 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-11a60b50-037e-43ad-ac63-9c1540d6c729 None None] Authenticating user token {{(pid=81408) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:27:29.345613 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-11a60b50-037e-43ad-ac63-9c1540d6c729 None None] Validating token access rules against request {{(pid=81408) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:27:29.351531 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fbbc4858-1f82-439f-9086-a41c4851603b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8ba823f313144da3b683bacef8fd62e0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81408) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:27:29.352136 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fbbc4858-1f82-439f-9086-a41c4851603b admin admin] REQUEST_METHOD: `POST` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:27:29.352415 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fbbc4858-1f82-439f-9086-a41c4851603b admin admin] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:27:29.352720 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fbbc4858-1f82-439f-9086-a41c4851603b admin admin] PATH_INFO: `/v3/projects` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:27:29.353401 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fbbc4858-1f82-439f-9086-a41c4851603b admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:27:29.355360 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fbbc4858-1f82-439f-9086-a41c4851603b admin admin] RBAC: Authorization granted {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:27:29.385069 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 1062/2153] 10.222.0.27 () {60 vars in 1153 bytes} [Thu Oct 13 04:27:29 2022] POST /identity/v3/projects => generated 375 bytes in 49 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 13 04:27:29.389195 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c2c4260e-aee1-4049-a7c9-574ffbcb44b7 admin admin] Authenticating user token {{(pid=81409) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:27:29.399642 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c2c4260e-aee1-4049-a7c9-574ffbcb44b7 admin admin] Validating token access rules against request {{(pid=81409) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:27:29.460706 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c7a8913e-46b1-48e7-8886-bbbe89f29622 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8ba823f313144da3b683bacef8fd62e0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81409) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:27:29.461261 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7a8913e-46b1-48e7-8886-bbbe89f29622 admin admin] REQUEST_METHOD: `POST` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:27:29.461533 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7a8913e-46b1-48e7-8886-bbbe89f29622 admin admin] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:27:29.461801 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7a8913e-46b1-48e7-8886-bbbe89f29622 admin admin] PATH_INFO: `/v3/users` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:27:29.462446 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c7a8913e-46b1-48e7-8886-bbbe89f29622 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:27:29.464342 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c7a8913e-46b1-48e7-8886-bbbe89f29622 admin admin] RBAC: Authorization granted {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:27:29.485196 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 1092/2154] 10.222.0.27 () {60 vars in 1144 bytes} [Thu Oct 13 04:27:29 2022] POST /identity/v3/users => generated 288 bytes in 96 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 13 04:27:29.488619 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fbbc4858-1f82-439f-9086-a41c4851603b admin admin] Authenticating user token {{(pid=81408) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:27:29.498776 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fbbc4858-1f82-439f-9086-a41c4851603b admin admin] Validating token access rules against request {{(pid=81408) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:27:29.503351 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c44a635e-fa9c-411e-9bc3-426c401d1e3c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8ba823f313144da3b683bacef8fd62e0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81408) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:27:29.503939 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c44a635e-fa9c-411e-9bc3-426c401d1e3c admin admin] REQUEST_METHOD: `GET` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:27:29.504213 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c44a635e-fa9c-411e-9bc3-426c401d1e3c admin admin] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:27:29.504484 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c44a635e-fa9c-411e-9bc3-426c401d1e3c admin admin] PATH_INFO: `/v3/roles` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:27:29.505658 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c44a635e-fa9c-411e-9bc3-426c401d1e3c admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:27:29.507290 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c44a635e-fa9c-411e-9bc3-426c401d1e3c admin admin] RBAC: Authorization granted {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:27:29.517045 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 1063/2155] 10.222.0.27 () {58 vars in 1122 bytes} [Thu Oct 13 04:27:29 2022] GET /identity/v3/roles => generated 1420 bytes in 29 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 13 04:27:29.521590 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c7a8913e-46b1-48e7-8886-bbbe89f29622 admin admin] Authenticating user token {{(pid=81409) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:27:29.531507 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c7a8913e-46b1-48e7-8886-bbbe89f29622 admin admin] Validating token access rules against request {{(pid=81409) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:27:29.536163 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-04c1da73-3145-4c4d-8c96-c1d70df904ac admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8ba823f313144da3b683bacef8fd62e0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81409) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:27:29.536741 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04c1da73-3145-4c4d-8c96-c1d70df904ac admin admin] REQUEST_METHOD: `GET` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:27:29.537014 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04c1da73-3145-4c4d-8c96-c1d70df904ac admin admin] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:27:29.537282 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04c1da73-3145-4c4d-8c96-c1d70df904ac admin admin] PATH_INFO: `/v3/roles` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:27:29.537909 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-04c1da73-3145-4c4d-8c96-c1d70df904ac admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:27:29.539525 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-04c1da73-3145-4c4d-8c96-c1d70df904ac admin admin] RBAC: Authorization granted {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:27:29.551059 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 1093/2156] 10.222.0.27 () {58 vars in 1122 bytes} [Thu Oct 13 04:27:29 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Oct 13 04:27:29.556095 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c44a635e-fa9c-411e-9bc3-426c401d1e3c admin admin] Authenticating user token {{(pid=81408) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:27:29.565815 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c44a635e-fa9c-411e-9bc3-426c401d1e3c admin admin] Validating token access rules against request {{(pid=81408) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:27:29.571349 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f4c749ee-ecd3-413c-a4cf-ad9a59012418 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8ba823f313144da3b683bacef8fd62e0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81408) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:27:29.571981 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f4c749ee-ecd3-413c-a4cf-ad9a59012418 admin admin] REQUEST_METHOD: `PUT` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:27:29.572253 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f4c749ee-ecd3-413c-a4cf-ad9a59012418 admin admin] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:27:29.572632 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f4c749ee-ecd3-413c-a4cf-ad9a59012418 admin admin] PATH_INFO: `/v3/projects/975fa16a6d294da4b6cd3bc36088467a/users/a52c46a777bd4807a889a1356c3105cc/roles/cc56c4ae13a7424ea771c15da4b153c1` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:27:29.588159 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f4c749ee-ecd3-413c-a4cf-ad9a59012418 admin admin] RBAC: Authorizing `identity:create_grant(project_id=975fa16a6d294da4b6cd3bc36088467a, user_id=a52c46a777bd4807a889a1356c3105cc, role_id=cc56c4ae13a7424ea771c15da4b153c1)` {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:27:29.590998 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f4c749ee-ecd3-413c-a4cf-ad9a59012418 admin admin] RBAC: Authorization granted {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:27:29.616953 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 1064/2157] 10.222.0.27 () {60 vars in 1483 bytes} [Thu Oct 13 04:27:29 2022] PUT /identity/v3/projects/975fa16a6d294da4b6cd3bc36088467a/users/a52c46a777bd4807a889a1356c3105cc/roles/cc56c4ae13a7424ea771c15da4b153c1 => generated 0 bytes in 61 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 13 04:27:29.627546 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e018112d-9fd5-4c38-a6a7-f92a4b6a7149 None None] REQUEST_METHOD: `POST` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:27:29.627823 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e018112d-9fd5-4c38-a6a7-f92a4b6a7149 None None] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:27:29.628091 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e018112d-9fd5-4c38-a6a7-f92a4b6a7149 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:27:29.655549 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.auth.core [None req-e018112d-9fd5-4c38-a6a7-f92a4b6a7149 None None] MFA Rules not processed for user `a52c46a777bd4807a889a1356c3105cc`. Rule list: `[]` (Enabled: `True`). {{(pid=81409) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 13 04:27:29.656746 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.fernet_utils [None req-e018112d-9fd5-4c38-a6a7-f92a4b6a7149 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81409) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:27:29.709679 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG dbcounter [-] [81409] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81409) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:27:29.720573 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 1094/2158] 10.222.0.27 () {58 vars in 958 bytes} [Thu Oct 13 04:27:29 2022] POST /identity/v3/auth/tokens => generated 3252 bytes in 94 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 13 04:27:29.731438 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f4c749ee-ecd3-413c-a4cf-ad9a59012418 admin admin] Authenticating user token {{(pid=81408) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:27:29.733086 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.fernet_utils [None req-f4c749ee-ecd3-413c-a4cf-ad9a59012418 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81408) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:27:29.790190 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f4c749ee-ecd3-413c-a4cf-ad9a59012418 admin admin] Validating token access rules against request {{(pid=81408) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:27:29.793863 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2e6ea9c1-7a90-4531-becd-4523cee6cf5e service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6141b8a6c3d54e179d6879bb019f4d12', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dcec0431fb6849499d6aec35a234daa6', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81408) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:27:29.794487 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e6ea9c1-7a90-4531-becd-4523cee6cf5e service cinder] REQUEST_METHOD: `GET` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:27:29.794784 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e6ea9c1-7a90-4531-becd-4523cee6cf5e service cinder] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:27:29.795068 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e6ea9c1-7a90-4531-becd-4523cee6cf5e service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:27:29.803324 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2e6ea9c1-7a90-4531-becd-4523cee6cf5e service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:27:29.804733 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2e6ea9c1-7a90-4531-becd-4523cee6cf5e service cinder] RBAC: Authorization granted {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:27:29.813287 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 1065/2159] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Oct 13 04:27:29 2022] GET /identity/v3/auth/tokens => generated 3252 bytes in 83 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 13 04:27:35.671622 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18d71bc8-09b0-446c-bea5-e1be5e72121e None None] REQUEST_METHOD: `POST` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:27:35.672077 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18d71bc8-09b0-446c-bea5-e1be5e72121e None None] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:27:35.672184 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18d71bc8-09b0-446c-bea5-e1be5e72121e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:27:35.691881 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.auth.core [None req-18d71bc8-09b0-446c-bea5-e1be5e72121e None None] MFA Rules not processed for user `487b1605fd4143cfaa422a3a7a0344c7`. Rule list: `[]` (Enabled: `True`). {{(pid=81409) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 13 04:27:35.693086 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.fernet_utils [None req-18d71bc8-09b0-446c-bea5-e1be5e72121e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81409) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:27:35.752758 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 1095/2160] 10.222.0.27 () {58 vars in 967 bytes} [Thu Oct 13 04:27:35 2022] POST /identity/v3/auth/tokens => generated 3195 bytes in 82 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 13 04:27:35.766185 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2e6ea9c1-7a90-4531-becd-4523cee6cf5e service cinder] Authenticating user token {{(pid=81408) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:27:35.768021 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.fernet_utils [None req-2e6ea9c1-7a90-4531-becd-4523cee6cf5e service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81408) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:27:35.832756 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG dbcounter [-] [81408] Writing DB stats keystone:SELECT=92,keystone:DELETE=5,keystone:INSERT=3 {{(pid=81408) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:27:35.840894 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2e6ea9c1-7a90-4531-becd-4523cee6cf5e service cinder] Validating token access rules against request {{(pid=81408) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:27:35.848169 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f14beebe-76c6-459a-99eb-d0bfc02255ae service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90a0f466b664458898cf9dc4112ae651', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dcec0431fb6849499d6aec35a234daa6', 'project_domain_id': 'default', 'roles': ['service', 'member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81408) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:27:35.848781 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f14beebe-76c6-459a-99eb-d0bfc02255ae service swift] REQUEST_METHOD: `GET` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:27:35.849180 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f14beebe-76c6-459a-99eb-d0bfc02255ae service swift] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:27:35.849630 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f14beebe-76c6-459a-99eb-d0bfc02255ae service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:27:35.858900 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f14beebe-76c6-459a-99eb-d0bfc02255ae service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:27:35.860618 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f14beebe-76c6-459a-99eb-d0bfc02255ae service swift] RBAC: Authorization granted {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:27:35.868995 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 1066/2161] 10.222.0.27 () {60 vars in 1384 bytes} [Thu Oct 13 04:27:35 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 103 msecs (HTTP/1.1 200) 6 headers in 379 bytes (1 switches on core 0) Oct 13 04:27:40.118108 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-18d71bc8-09b0-446c-bea5-e1be5e72121e None None] Authenticating user token {{(pid=81409) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:27:40.180535 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-18d71bc8-09b0-446c-bea5-e1be5e72121e None None] Validating token access rules against request {{(pid=81409) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:27:40.184878 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-48a49930-b2ce-42db-adaa-76f59e1bfc8f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8ba823f313144da3b683bacef8fd62e0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81409) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:27:40.185488 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48a49930-b2ce-42db-adaa-76f59e1bfc8f admin admin] REQUEST_METHOD: `DELETE` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:27:40.185758 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48a49930-b2ce-42db-adaa-76f59e1bfc8f admin admin] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:27:40.186022 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48a49930-b2ce-42db-adaa-76f59e1bfc8f admin admin] PATH_INFO: `/v3/users/a52c46a777bd4807a889a1356c3105cc` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:27:40.187176 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-48a49930-b2ce-42db-adaa-76f59e1bfc8f admin admin] RBAC: Authorizing `identity:delete_user(user_id=a52c46a777bd4807a889a1356c3105cc)` {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:27:40.189065 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-48a49930-b2ce-42db-adaa-76f59e1bfc8f admin admin] RBAC: Authorization granted {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:27:40.237794 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.notifications [None req-48a49930-b2ce-42db-adaa-76f59e1bfc8f admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'a52c46a777bd4807a889a1356c3105cc'} {{(pid=81409) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 13 04:27:40.238541 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.notifications [None req-48a49930-b2ce-42db-adaa-76f59e1bfc8f admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'a52c46a777bd4807a889a1356c3105cc'} {{(pid=81409) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 13 04:27:40.246765 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.notifications [None req-48a49930-b2ce-42db-adaa-76f59e1bfc8f admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'a52c46a777bd4807a889a1356c3105cc'} {{(pid=81409) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 13 04:27:40.247571 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.notifications [None req-48a49930-b2ce-42db-adaa-76f59e1bfc8f admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'a52c46a777bd4807a889a1356c3105cc'} {{(pid=81409) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 13 04:27:40.254497 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.notifications [None req-48a49930-b2ce-42db-adaa-76f59e1bfc8f admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'a52c46a777bd4807a889a1356c3105cc'} {{(pid=81409) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 13 04:27:40.272357 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 1096/2162] 10.222.0.27 () {58 vars in 1224 bytes} [Thu Oct 13 04:27:40 2022] DELETE /identity/v3/users/a52c46a777bd4807a889a1356c3105cc => generated 0 bytes in 155 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 13 04:27:40.277120 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fce5376c-9ae2-4883-93f2-b10bfb3a62d0 None None] REQUEST_METHOD: `POST` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:27:40.277644 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fce5376c-9ae2-4883-93f2-b10bfb3a62d0 None None] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:27:40.277747 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fce5376c-9ae2-4883-93f2-b10bfb3a62d0 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:27:40.296550 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.auth.core [None req-fce5376c-9ae2-4883-93f2-b10bfb3a62d0 None None] MFA Rules not processed for user `0e2d979563d6421995451a0e86bf3022`. Rule list: `[]` (Enabled: `True`). {{(pid=81408) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 13 04:27:40.297730 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.fernet_utils [None req-fce5376c-9ae2-4883-93f2-b10bfb3a62d0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81408) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:27:40.357081 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 1067/2163] 10.222.0.27 () {58 vars in 958 bytes} [Thu Oct 13 04:27:40 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 81 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 13 04:27:40.367699 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-48a49930-b2ce-42db-adaa-76f59e1bfc8f admin admin] Authenticating user token {{(pid=81409) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:27:40.369208 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.fernet_utils [None req-48a49930-b2ce-42db-adaa-76f59e1bfc8f admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81409) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:27:40.390709 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG dbcounter [-] [81409] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81409) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:27:40.431918 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-48a49930-b2ce-42db-adaa-76f59e1bfc8f admin admin] Validating token access rules against request {{(pid=81409) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:27:40.435442 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0d3daab6-85cb-4c77-9d64-566f4d743809 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd8812c3974f84871bb2d761cf0016257', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dcec0431fb6849499d6aec35a234daa6', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81409) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:27:40.436046 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d3daab6-85cb-4c77-9d64-566f4d743809 service neutron] REQUEST_METHOD: `GET` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:27:40.436346 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d3daab6-85cb-4c77-9d64-566f4d743809 service neutron] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:27:40.436613 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d3daab6-85cb-4c77-9d64-566f4d743809 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:27:40.445882 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0d3daab6-85cb-4c77-9d64-566f4d743809 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:27:40.447353 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0d3daab6-85cb-4c77-9d64-566f4d743809 service neutron] RBAC: Authorization granted {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:27:40.457334 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 1097/2164] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Oct 13 04:27:40 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 91 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 13 04:27:40.735454 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fce5376c-9ae2-4883-93f2-b10bfb3a62d0 None None] Authenticating user token {{(pid=81408) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:27:40.736896 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.fernet_utils [None req-fce5376c-9ae2-4883-93f2-b10bfb3a62d0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81408) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:27:40.749364 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fce5376c-9ae2-4883-93f2-b10bfb3a62d0 None None] Validating token access rules against request {{(pid=81408) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:27:40.754040 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-873eb96f-b972-414d-9c0f-4eb7c4043cac admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8ba823f313144da3b683bacef8fd62e0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81408) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:27:40.754767 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-873eb96f-b972-414d-9c0f-4eb7c4043cac admin admin] REQUEST_METHOD: `DELETE` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:27:40.755047 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-873eb96f-b972-414d-9c0f-4eb7c4043cac admin admin] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:27:40.755315 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-873eb96f-b972-414d-9c0f-4eb7c4043cac admin admin] PATH_INFO: `/v3/projects/975fa16a6d294da4b6cd3bc36088467a` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:27:40.756462 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-873eb96f-b972-414d-9c0f-4eb7c4043cac admin admin] RBAC: Authorizing `identity:delete_project(project_id=975fa16a6d294da4b6cd3bc36088467a)` {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:27:40.758820 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-873eb96f-b972-414d-9c0f-4eb7c4043cac admin admin] RBAC: Authorization granted {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:27:40.805512 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.notifications [None req-873eb96f-b972-414d-9c0f-4eb7c4043cac admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '975fa16a6d294da4b6cd3bc36088467a', 'request_id': 'req-873eb96f-b972-414d-9c0f-4eb7c4043cac'} {{(pid=81408) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 13 04:27:40.812423 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.notifications [None req-873eb96f-b972-414d-9c0f-4eb7c4043cac admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '975fa16a6d294da4b6cd3bc36088467a', 'request_id': 'req-873eb96f-b972-414d-9c0f-4eb7c4043cac'} {{(pid=81408) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 13 04:27:40.818691 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.notifications [None req-873eb96f-b972-414d-9c0f-4eb7c4043cac admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '975fa16a6d294da4b6cd3bc36088467a', 'request_id': 'req-873eb96f-b972-414d-9c0f-4eb7c4043cac'} {{(pid=81408) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 13 04:27:40.829019 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.notifications [None req-873eb96f-b972-414d-9c0f-4eb7c4043cac admin admin] The token cache is being invalidate because project 975fa16a6d294da4b6cd3bc36088467a was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81408) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 13 04:27:40.829303 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.notifications [None req-873eb96f-b972-414d-9c0f-4eb7c4043cac admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81408) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 13 04:27:40.829927 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.notifications [None req-873eb96f-b972-414d-9c0f-4eb7c4043cac admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81408) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 13 04:27:40.831400 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 1068/2165] 10.222.0.27 () {58 vars in 1233 bytes} [Thu Oct 13 04:27:40 2022] DELETE /identity/v3/projects/975fa16a6d294da4b6cd3bc36088467a => generated 0 bytes in 97 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 13 04:27:40.837214 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68de13c0-1dc1-4dd9-bbd9-3fb64d422d32 None None] REQUEST_METHOD: `POST` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:27:40.837493 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68de13c0-1dc1-4dd9-bbd9-3fb64d422d32 None None] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:27:40.837760 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68de13c0-1dc1-4dd9-bbd9-3fb64d422d32 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:27:40.857927 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.auth.core [None req-68de13c0-1dc1-4dd9-bbd9-3fb64d422d32 None None] MFA Rules not processed for user `0e2d979563d6421995451a0e86bf3022`. Rule list: `[]` (Enabled: `True`). {{(pid=81409) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 13 04:27:40.859158 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.fernet_utils [None req-68de13c0-1dc1-4dd9-bbd9-3fb64d422d32 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81409) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:27:40.919040 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 1098/2166] 10.222.0.27 () {58 vars in 958 bytes} [Thu Oct 13 04:27:40 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 83 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 13 04:27:40.926689 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9774b871-a5ba-4fad-9ba1-c62afe6a5140 None None] REQUEST_METHOD: `POST` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:27:40.926973 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9774b871-a5ba-4fad-9ba1-c62afe6a5140 None None] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:27:40.927246 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9774b871-a5ba-4fad-9ba1-c62afe6a5140 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:27:40.946983 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.auth.core [None req-9774b871-a5ba-4fad-9ba1-c62afe6a5140 None None] MFA Rules not processed for user `0e2d979563d6421995451a0e86bf3022`. Rule list: `[]` (Enabled: `True`). {{(pid=81408) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 13 04:27:40.948226 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.fernet_utils [None req-9774b871-a5ba-4fad-9ba1-c62afe6a5140 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81408) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:27:40.955918 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 1069/2167] 10.222.0.27 () {58 vars in 958 bytes} [Thu Oct 13 04:27:40 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 30 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 13 04:27:40.960153 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-68de13c0-1dc1-4dd9-bbd9-3fb64d422d32 None None] Authenticating user token {{(pid=81409) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:27:40.970445 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-68de13c0-1dc1-4dd9-bbd9-3fb64d422d32 None None] Validating token access rules against request {{(pid=81409) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:27:40.975034 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e5f17c58-4e09-4b7b-b658-cb4bdea7f33b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8ba823f313144da3b683bacef8fd62e0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81409) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:27:40.975625 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e5f17c58-4e09-4b7b-b658-cb4bdea7f33b admin admin] REQUEST_METHOD: `GET` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:27:40.975897 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e5f17c58-4e09-4b7b-b658-cb4bdea7f33b admin admin] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:27:40.976732 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e5f17c58-4e09-4b7b-b658-cb4bdea7f33b admin admin] PATH_INFO: `/v3/domains` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:27:40.977419 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e5f17c58-4e09-4b7b-b658-cb4bdea7f33b admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81409) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 13 04:27:40.977719 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e5f17c58-4e09-4b7b-b658-cb4bdea7f33b admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:27:40.979345 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e5f17c58-4e09-4b7b-b658-cb4bdea7f33b admin admin] RBAC: Authorization granted {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:27:40.991318 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 1099/2168] 10.222.0.27 () {58 vars in 1153 bytes} [Thu Oct 13 04:27:40 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 13 04:27:40.995970 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9774b871-a5ba-4fad-9ba1-c62afe6a5140 None None] Authenticating user token {{(pid=81408) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:27:41.005557 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9774b871-a5ba-4fad-9ba1-c62afe6a5140 None None] Validating token access rules against request {{(pid=81408) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:27:41.010591 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-23bf6805-a7ec-4db6-9715-97d98ef8c1b1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8ba823f313144da3b683bacef8fd62e0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81408) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:27:41.011338 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23bf6805-a7ec-4db6-9715-97d98ef8c1b1 admin admin] REQUEST_METHOD: `POST` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:27:41.011795 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23bf6805-a7ec-4db6-9715-97d98ef8c1b1 admin admin] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:27:41.012026 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23bf6805-a7ec-4db6-9715-97d98ef8c1b1 admin admin] PATH_INFO: `/v3/projects` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:27:41.012662 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-23bf6805-a7ec-4db6-9715-97d98ef8c1b1 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:27:41.016221 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-23bf6805-a7ec-4db6-9715-97d98ef8c1b1 admin admin] RBAC: Authorization granted {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:27:41.047575 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 1070/2169] 10.222.0.27 () {60 vars in 1153 bytes} [Thu Oct 13 04:27:40 2022] POST /identity/v3/projects => generated 373 bytes in 52 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 13 04:27:41.051861 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e5f17c58-4e09-4b7b-b658-cb4bdea7f33b admin admin] Authenticating user token {{(pid=81409) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:27:41.062607 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e5f17c58-4e09-4b7b-b658-cb4bdea7f33b admin admin] Validating token access rules against request {{(pid=81409) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:27:41.122574 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3b3e97c3-c3e6-4be1-b322-8ac5fed64e55 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8ba823f313144da3b683bacef8fd62e0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81409) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:27:41.123281 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b3e97c3-c3e6-4be1-b322-8ac5fed64e55 admin admin] REQUEST_METHOD: `POST` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:27:41.123450 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b3e97c3-c3e6-4be1-b322-8ac5fed64e55 admin admin] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:27:41.123717 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b3e97c3-c3e6-4be1-b322-8ac5fed64e55 admin admin] PATH_INFO: `/v3/users` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:27:41.124350 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3b3e97c3-c3e6-4be1-b322-8ac5fed64e55 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:27:41.126280 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3b3e97c3-c3e6-4be1-b322-8ac5fed64e55 admin admin] RBAC: Authorization granted {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:27:41.150403 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 1100/2170] 10.222.0.27 () {60 vars in 1144 bytes} [Thu Oct 13 04:27:41 2022] POST /identity/v3/users => generated 287 bytes in 96 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 13 04:27:41.151631 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-23bf6805-a7ec-4db6-9715-97d98ef8c1b1 admin admin] Authenticating user token {{(pid=81408) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:27:41.162602 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-23bf6805-a7ec-4db6-9715-97d98ef8c1b1 admin admin] Validating token access rules against request {{(pid=81408) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:27:41.168339 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3f52cb45-335b-4de7-b083-ec465ff5ca38 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8ba823f313144da3b683bacef8fd62e0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81408) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:27:41.168949 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f52cb45-335b-4de7-b083-ec465ff5ca38 admin admin] REQUEST_METHOD: `GET` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:27:41.169226 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f52cb45-335b-4de7-b083-ec465ff5ca38 admin admin] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:27:41.169499 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f52cb45-335b-4de7-b083-ec465ff5ca38 admin admin] PATH_INFO: `/v3/roles` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:27:41.170146 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3f52cb45-335b-4de7-b083-ec465ff5ca38 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:27:41.171813 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3f52cb45-335b-4de7-b083-ec465ff5ca38 admin admin] RBAC: Authorization granted {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:27:41.183153 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 1071/2171] 10.222.0.27 () {58 vars in 1122 bytes} [Thu Oct 13 04:27:41 2022] GET /identity/v3/roles => generated 1420 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 13 04:27:41.187193 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3b3e97c3-c3e6-4be1-b322-8ac5fed64e55 admin admin] Authenticating user token {{(pid=81409) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:27:41.197831 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3b3e97c3-c3e6-4be1-b322-8ac5fed64e55 admin admin] Validating token access rules against request {{(pid=81409) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:27:41.202845 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3a1c6dd4-1672-404f-a7bb-6d05dcfd0234 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8ba823f313144da3b683bacef8fd62e0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81409) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:27:41.203466 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a1c6dd4-1672-404f-a7bb-6d05dcfd0234 admin admin] REQUEST_METHOD: `GET` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:27:41.203738 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a1c6dd4-1672-404f-a7bb-6d05dcfd0234 admin admin] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:27:41.204030 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a1c6dd4-1672-404f-a7bb-6d05dcfd0234 admin admin] PATH_INFO: `/v3/roles` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:27:41.204636 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3a1c6dd4-1672-404f-a7bb-6d05dcfd0234 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:27:41.206236 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3a1c6dd4-1672-404f-a7bb-6d05dcfd0234 admin admin] RBAC: Authorization granted {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:27:41.217955 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 1101/2172] 10.222.0.27 () {58 vars in 1122 bytes} [Thu Oct 13 04:27:41 2022] GET /identity/v3/roles => generated 1420 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 13 04:27:41.222037 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3f52cb45-335b-4de7-b083-ec465ff5ca38 admin admin] Authenticating user token {{(pid=81408) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:27:41.232993 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3f52cb45-335b-4de7-b083-ec465ff5ca38 admin admin] Validating token access rules against request {{(pid=81408) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:27:41.237662 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c45c5415-f33c-4900-9055-62d3098e0caf admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8ba823f313144da3b683bacef8fd62e0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81408) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:27:41.238349 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c45c5415-f33c-4900-9055-62d3098e0caf admin admin] REQUEST_METHOD: `PUT` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:27:41.238627 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c45c5415-f33c-4900-9055-62d3098e0caf admin admin] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:27:41.238899 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c45c5415-f33c-4900-9055-62d3098e0caf admin admin] PATH_INFO: `/v3/projects/cecec3eab7d3499a984c510355f61ebc/users/f38a674195a4438f87d467179d2c8860/roles/cc56c4ae13a7424ea771c15da4b153c1` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:27:41.255259 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c45c5415-f33c-4900-9055-62d3098e0caf admin admin] RBAC: Authorizing `identity:create_grant(project_id=cecec3eab7d3499a984c510355f61ebc, user_id=f38a674195a4438f87d467179d2c8860, role_id=cc56c4ae13a7424ea771c15da4b153c1)` {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:27:41.258380 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c45c5415-f33c-4900-9055-62d3098e0caf admin admin] RBAC: Authorization granted {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:27:41.272696 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 1072/2173] 10.222.0.27 () {60 vars in 1483 bytes} [Thu Oct 13 04:27:41 2022] PUT /identity/v3/projects/cecec3eab7d3499a984c510355f61ebc/users/f38a674195a4438f87d467179d2c8860/roles/cc56c4ae13a7424ea771c15da4b153c1 => generated 0 bytes in 51 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 13 04:27:41.283130 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd0104cf-0029-47ba-a55c-25a5286590f2 None None] REQUEST_METHOD: `POST` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:27:41.283437 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd0104cf-0029-47ba-a55c-25a5286590f2 None None] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:27:41.283708 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd0104cf-0029-47ba-a55c-25a5286590f2 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:27:41.315622 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.auth.core [None req-fd0104cf-0029-47ba-a55c-25a5286590f2 None None] MFA Rules not processed for user `f38a674195a4438f87d467179d2c8860`. Rule list: `[]` (Enabled: `True`). {{(pid=81409) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 13 04:27:41.316876 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.fernet_utils [None req-fd0104cf-0029-47ba-a55c-25a5286590f2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81409) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:27:41.339445 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG dbcounter [-] [81409] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81409) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:27:41.396813 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 1102/2174] 10.222.0.27 () {58 vars in 958 bytes} [Thu Oct 13 04:27:41 2022] POST /identity/v3/auth/tokens => generated 3250 bytes in 115 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 13 04:27:41.408437 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c45c5415-f33c-4900-9055-62d3098e0caf admin admin] Authenticating user token {{(pid=81408) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:27:41.410344 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.fernet_utils [None req-c45c5415-f33c-4900-9055-62d3098e0caf admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81408) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:27:41.421909 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG dbcounter [-] [81408] Writing DB stats keystone:SELECT=90,keystone:DELETE=7,keystone:INSERT=3 {{(pid=81408) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:27:41.470767 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c45c5415-f33c-4900-9055-62d3098e0caf admin admin] Validating token access rules against request {{(pid=81408) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:27:41.474323 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d05ecf91-1e04-4ab8-b3ad-de1a417e7fc2 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6141b8a6c3d54e179d6879bb019f4d12', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dcec0431fb6849499d6aec35a234daa6', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81408) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:27:41.474924 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d05ecf91-1e04-4ab8-b3ad-de1a417e7fc2 service cinder] REQUEST_METHOD: `GET` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:27:41.475198 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d05ecf91-1e04-4ab8-b3ad-de1a417e7fc2 service cinder] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:27:41.475469 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d05ecf91-1e04-4ab8-b3ad-de1a417e7fc2 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:27:41.484378 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d05ecf91-1e04-4ab8-b3ad-de1a417e7fc2 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:27:41.485810 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d05ecf91-1e04-4ab8-b3ad-de1a417e7fc2 service cinder] RBAC: Authorization granted {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:27:41.494582 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 1073/2175] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Oct 13 04:27:41 2022] GET /identity/v3/auth/tokens => generated 3250 bytes in 87 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Oct 13 04:27:42.855563 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d904665-91f4-4622-8db0-f9d26e013a28 None None] REQUEST_METHOD: `POST` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:27:42.856349 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d904665-91f4-4622-8db0-f9d26e013a28 None None] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:27:42.856349 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d904665-91f4-4622-8db0-f9d26e013a28 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:27:42.875001 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.auth.core [None req-8d904665-91f4-4622-8db0-f9d26e013a28 None None] MFA Rules not processed for user `f38a674195a4438f87d467179d2c8860`. Rule list: `[]` (Enabled: `True`). {{(pid=81409) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 13 04:27:42.876200 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.fernet_utils [None req-8d904665-91f4-4622-8db0-f9d26e013a28 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81409) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:27:42.883564 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 1103/2176] 10.222.0.27 () {58 vars in 958 bytes} [Thu Oct 13 04:27:42 2022] POST /identity/v3/auth/tokens => generated 3250 bytes in 29 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 13 04:27:51.490521 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG dbcounter [-] [81408] Writing DB stats keystone:SELECT=22 {{(pid=81408) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:27:52.869569 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG dbcounter [-] [81409] Writing DB stats keystone:SELECT=23 {{(pid=81409) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:27:59.323348 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22742087-eb0e-4226-8916-fd4f641cee40 None None] REQUEST_METHOD: `POST` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:27:59.323898 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22742087-eb0e-4226-8916-fd4f641cee40 None None] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:27:59.324121 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22742087-eb0e-4226-8916-fd4f641cee40 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:27:59.344363 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.auth.core [None req-22742087-eb0e-4226-8916-fd4f641cee40 None None] MFA Rules not processed for user `487b1605fd4143cfaa422a3a7a0344c7`. Rule list: `[]` (Enabled: `True`). {{(pid=81408) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 13 04:27:59.345602 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.fernet_utils [None req-22742087-eb0e-4226-8916-fd4f641cee40 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81408) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:27:59.409948 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 1074/2177] 10.222.0.27 () {58 vars in 1009 bytes} [Thu Oct 13 04:27:59 2022] POST /identity/v3/auth/tokens => generated 3195 bytes in 88 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 13 04:27:59.440788 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8d904665-91f4-4622-8db0-f9d26e013a28 None None] Authenticating user token {{(pid=81409) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:27:59.442923 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.fernet_utils [None req-8d904665-91f4-4622-8db0-f9d26e013a28 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81409) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:27:59.531725 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8d904665-91f4-4622-8db0-f9d26e013a28 None None] Validating token access rules against request {{(pid=81409) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:27:59.536872 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2f232c93-7b38-4ea3-bf45-329772c9b8fa service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90a0f466b664458898cf9dc4112ae651', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dcec0431fb6849499d6aec35a234daa6', 'project_domain_id': 'default', 'roles': ['service', 'member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81409) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:27:59.537964 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f232c93-7b38-4ea3-bf45-329772c9b8fa service swift] REQUEST_METHOD: `GET` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:27:59.538165 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f232c93-7b38-4ea3-bf45-329772c9b8fa service swift] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:27:59.538432 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f232c93-7b38-4ea3-bf45-329772c9b8fa service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:27:59.548135 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2f232c93-7b38-4ea3-bf45-329772c9b8fa service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:27:59.549858 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2f232c93-7b38-4ea3-bf45-329772c9b8fa service swift] RBAC: Authorization granted {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:27:59.559577 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 1104/2178] 10.222.0.27 () {60 vars in 1384 bytes} [Thu Oct 13 04:27:59 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 119 msecs (HTTP/1.1 200) 6 headers in 379 bytes (1 switches on core 0) Oct 13 04:28:00.089803 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0048ee20-238d-491b-9ac9-2b5f8794e739 None None] REQUEST_METHOD: `POST` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:28:00.090090 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0048ee20-238d-491b-9ac9-2b5f8794e739 None None] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:28:00.090410 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0048ee20-238d-491b-9ac9-2b5f8794e739 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:28:00.111420 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.auth.core [None req-0048ee20-238d-491b-9ac9-2b5f8794e739 None None] MFA Rules not processed for user `487b1605fd4143cfaa422a3a7a0344c7`. Rule list: `[]` (Enabled: `True`). {{(pid=81408) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 13 04:28:00.112602 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.fernet_utils [None req-0048ee20-238d-491b-9ac9-2b5f8794e739 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81408) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:28:00.120353 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 1075/2179] 10.222.0.27 () {58 vars in 967 bytes} [Thu Oct 13 04:28:00 2022] POST /identity/v3/auth/tokens => generated 3195 bytes in 32 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 13 04:28:00.133453 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2f232c93-7b38-4ea3-bf45-329772c9b8fa service swift] Authenticating user token {{(pid=81409) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:28:00.143988 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2f232c93-7b38-4ea3-bf45-329772c9b8fa service swift] Validating token access rules against request {{(pid=81409) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:28:00.149102 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8047c480-b70d-434f-90af-cd18d3535100 service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90a0f466b664458898cf9dc4112ae651', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dcec0431fb6849499d6aec35a234daa6', 'project_domain_id': 'default', 'roles': ['service', 'member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81409) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:28:00.149693 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8047c480-b70d-434f-90af-cd18d3535100 service swift] REQUEST_METHOD: `GET` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:28:00.149971 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8047c480-b70d-434f-90af-cd18d3535100 service swift] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:28:00.150272 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8047c480-b70d-434f-90af-cd18d3535100 service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:28:00.158198 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8047c480-b70d-434f-90af-cd18d3535100 service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:28:00.159787 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8047c480-b70d-434f-90af-cd18d3535100 service swift] RBAC: Authorization granted {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:28:00.168269 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 1105/2180] 10.222.0.27 () {60 vars in 1384 bytes} [Thu Oct 13 04:28:00 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 35 msecs (HTTP/1.1 200) 6 headers in 379 bytes (1 switches on core 0) Oct 13 04:28:00.415772 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b6782c9-6095-4671-9c42-ac2d6317f864 None None] REQUEST_METHOD: `POST` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:28:00.416266 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b6782c9-6095-4671-9c42-ac2d6317f864 None None] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:28:00.416385 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b6782c9-6095-4671-9c42-ac2d6317f864 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:28:00.439064 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.auth.core [None req-7b6782c9-6095-4671-9c42-ac2d6317f864 None None] MFA Rules not processed for user `487b1605fd4143cfaa422a3a7a0344c7`. Rule list: `[]` (Enabled: `True`). {{(pid=81408) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 13 04:28:00.440279 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.fernet_utils [None req-7b6782c9-6095-4671-9c42-ac2d6317f864 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81408) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:28:00.449897 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 1076/2181] 10.222.0.27 () {58 vars in 1009 bytes} [Thu Oct 13 04:28:00 2022] POST /identity/v3/auth/tokens => generated 3195 bytes in 35 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 13 04:28:00.468027 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8047c480-b70d-434f-90af-cd18d3535100 service swift] Authenticating user token {{(pid=81409) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:28:00.481180 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8047c480-b70d-434f-90af-cd18d3535100 service swift] Validating token access rules against request {{(pid=81409) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:28:00.486609 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-803cb9ef-8f89-46a2-886f-ab656227c4ef service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90a0f466b664458898cf9dc4112ae651', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dcec0431fb6849499d6aec35a234daa6', 'project_domain_id': 'default', 'roles': ['service', 'member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81409) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:28:00.487219 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-803cb9ef-8f89-46a2-886f-ab656227c4ef service swift] REQUEST_METHOD: `GET` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:28:00.487494 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-803cb9ef-8f89-46a2-886f-ab656227c4ef service swift] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:28:00.487760 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-803cb9ef-8f89-46a2-886f-ab656227c4ef service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:28:00.497711 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-803cb9ef-8f89-46a2-886f-ab656227c4ef service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:28:00.499505 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-803cb9ef-8f89-46a2-886f-ab656227c4ef service swift] RBAC: Authorization granted {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:28:00.508000 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 1106/2182] 10.222.0.27 () {60 vars in 1384 bytes} [Thu Oct 13 04:28:00 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 40 msecs (HTTP/1.1 200) 6 headers in 379 bytes (1 switches on core 0) Oct 13 04:28:00.867923 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a0903c3-4048-45bf-8831-ce7a8a0591a9 None None] REQUEST_METHOD: `POST` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:28:00.868212 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a0903c3-4048-45bf-8831-ce7a8a0591a9 None None] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:28:00.868490 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a0903c3-4048-45bf-8831-ce7a8a0591a9 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:28:00.890848 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.auth.core [None req-5a0903c3-4048-45bf-8831-ce7a8a0591a9 None None] MFA Rules not processed for user `487b1605fd4143cfaa422a3a7a0344c7`. Rule list: `[]` (Enabled: `True`). {{(pid=81408) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 13 04:28:00.892068 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.fernet_utils [None req-5a0903c3-4048-45bf-8831-ce7a8a0591a9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81408) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:28:00.899647 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 1077/2183] 10.222.0.27 () {58 vars in 967 bytes} [Thu Oct 13 04:28:00 2022] POST /identity/v3/auth/tokens => generated 3195 bytes in 33 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 13 04:28:00.912379 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-803cb9ef-8f89-46a2-886f-ab656227c4ef service swift] Authenticating user token {{(pid=81409) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:28:00.924101 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-803cb9ef-8f89-46a2-886f-ab656227c4ef service swift] Validating token access rules against request {{(pid=81409) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:28:00.929154 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-76997d3d-1130-4950-8a7c-51499219cc13 service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90a0f466b664458898cf9dc4112ae651', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dcec0431fb6849499d6aec35a234daa6', 'project_domain_id': 'default', 'roles': ['service', 'member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81409) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:28:00.929851 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76997d3d-1130-4950-8a7c-51499219cc13 service swift] REQUEST_METHOD: `GET` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:28:00.930267 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76997d3d-1130-4950-8a7c-51499219cc13 service swift] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:28:00.930580 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76997d3d-1130-4950-8a7c-51499219cc13 service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:28:00.939155 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-76997d3d-1130-4950-8a7c-51499219cc13 service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:28:00.940737 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-76997d3d-1130-4950-8a7c-51499219cc13 service swift] RBAC: Authorization granted {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:28:00.949832 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 1107/2184] 10.222.0.27 () {60 vars in 1384 bytes} [Thu Oct 13 04:28:00 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 38 msecs (HTTP/1.1 200) 6 headers in 379 bytes (1 switches on core 0) Oct 13 04:28:03.148259 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5a0903c3-4048-45bf-8831-ce7a8a0591a9 None None] Authenticating user token {{(pid=81408) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:28:03.212256 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5a0903c3-4048-45bf-8831-ce7a8a0591a9 None None] Validating token access rules against request {{(pid=81408) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:28:03.216855 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7eaafa8c-8dbe-4340-9df3-dcf067d1bf42 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8ba823f313144da3b683bacef8fd62e0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81408) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:28:03.217480 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7eaafa8c-8dbe-4340-9df3-dcf067d1bf42 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:28:03.217758 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7eaafa8c-8dbe-4340-9df3-dcf067d1bf42 admin admin] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:28:03.218058 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7eaafa8c-8dbe-4340-9df3-dcf067d1bf42 admin admin] PATH_INFO: `/v3/users/f38a674195a4438f87d467179d2c8860` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:28:03.219231 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7eaafa8c-8dbe-4340-9df3-dcf067d1bf42 admin admin] RBAC: Authorizing `identity:delete_user(user_id=f38a674195a4438f87d467179d2c8860)` {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:28:03.221164 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7eaafa8c-8dbe-4340-9df3-dcf067d1bf42 admin admin] RBAC: Authorization granted {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:28:03.268237 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.notifications [None req-7eaafa8c-8dbe-4340-9df3-dcf067d1bf42 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'f38a674195a4438f87d467179d2c8860'} {{(pid=81408) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 13 04:28:03.275208 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.notifications [None req-7eaafa8c-8dbe-4340-9df3-dcf067d1bf42 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'f38a674195a4438f87d467179d2c8860'} {{(pid=81408) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 13 04:28:03.275868 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.notifications [None req-7eaafa8c-8dbe-4340-9df3-dcf067d1bf42 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'f38a674195a4438f87d467179d2c8860'} {{(pid=81408) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 13 04:28:03.276532 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.notifications [None req-7eaafa8c-8dbe-4340-9df3-dcf067d1bf42 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'f38a674195a4438f87d467179d2c8860'} {{(pid=81408) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 13 04:28:03.282867 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.notifications [None req-7eaafa8c-8dbe-4340-9df3-dcf067d1bf42 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'f38a674195a4438f87d467179d2c8860'} {{(pid=81408) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 13 04:28:03.300136 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 1078/2185] 10.222.0.27 () {58 vars in 1224 bytes} [Thu Oct 13 04:28:03 2022] DELETE /identity/v3/users/f38a674195a4438f87d467179d2c8860 => generated 0 bytes in 152 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 13 04:28:03.305117 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5240ac0f-0875-4917-9824-ff83d84e3f23 None None] REQUEST_METHOD: `POST` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:28:03.305672 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5240ac0f-0875-4917-9824-ff83d84e3f23 None None] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:28:03.305889 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5240ac0f-0875-4917-9824-ff83d84e3f23 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:28:03.324947 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.auth.core [None req-5240ac0f-0875-4917-9824-ff83d84e3f23 None None] MFA Rules not processed for user `0e2d979563d6421995451a0e86bf3022`. Rule list: `[]` (Enabled: `True`). {{(pid=81409) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 13 04:28:03.327357 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.fernet_utils [None req-5240ac0f-0875-4917-9824-ff83d84e3f23 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81409) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:28:03.390510 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 1108/2186] 10.222.0.27 () {58 vars in 958 bytes} [Thu Oct 13 04:28:03 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 87 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 13 04:28:03.400911 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7eaafa8c-8dbe-4340-9df3-dcf067d1bf42 admin admin] Authenticating user token {{(pid=81408) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:28:03.402523 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.fernet_utils [None req-7eaafa8c-8dbe-4340-9df3-dcf067d1bf42 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81408) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:28:03.409767 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG dbcounter [-] [81408] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81408) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:28:03.474682 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7eaafa8c-8dbe-4340-9df3-dcf067d1bf42 admin admin] Validating token access rules against request {{(pid=81408) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:28:03.478551 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-115e2d60-3c9e-4414-a898-af57ac95d68a service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd8812c3974f84871bb2d761cf0016257', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dcec0431fb6849499d6aec35a234daa6', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81408) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:28:03.479160 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-115e2d60-3c9e-4414-a898-af57ac95d68a service neutron] REQUEST_METHOD: `GET` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:28:03.479440 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-115e2d60-3c9e-4414-a898-af57ac95d68a service neutron] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:28:03.479713 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-115e2d60-3c9e-4414-a898-af57ac95d68a service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:28:03.489170 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-115e2d60-3c9e-4414-a898-af57ac95d68a service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:28:03.490633 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-115e2d60-3c9e-4414-a898-af57ac95d68a service neutron] RBAC: Authorization granted {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:28:03.501801 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 1079/2187] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Oct 13 04:28:03 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 101 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 13 04:28:03.689175 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5240ac0f-0875-4917-9824-ff83d84e3f23 None None] Authenticating user token {{(pid=81409) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:28:03.690655 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.fernet_utils [None req-5240ac0f-0875-4917-9824-ff83d84e3f23 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81409) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:28:03.709578 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5240ac0f-0875-4917-9824-ff83d84e3f23 None None] Validating token access rules against request {{(pid=81409) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:28:03.715572 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-33a93f2b-e04f-4a53-a88e-3ddff71f5d27 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8ba823f313144da3b683bacef8fd62e0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81409) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:28:03.716248 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33a93f2b-e04f-4a53-a88e-3ddff71f5d27 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:28:03.716615 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33a93f2b-e04f-4a53-a88e-3ddff71f5d27 admin admin] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:28:03.716755 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33a93f2b-e04f-4a53-a88e-3ddff71f5d27 admin admin] PATH_INFO: `/v3/projects/cecec3eab7d3499a984c510355f61ebc` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:28:03.717878 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-33a93f2b-e04f-4a53-a88e-3ddff71f5d27 admin admin] RBAC: Authorizing `identity:delete_project(project_id=cecec3eab7d3499a984c510355f61ebc)` {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:28:03.719821 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-33a93f2b-e04f-4a53-a88e-3ddff71f5d27 admin admin] RBAC: Authorization granted {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:28:03.764760 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG dbcounter [-] [81409] Writing DB stats keystone:SELECT=97,keystone:DELETE=3 {{(pid=81409) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:28:03.767928 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.notifications [None req-33a93f2b-e04f-4a53-a88e-3ddff71f5d27 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'cecec3eab7d3499a984c510355f61ebc', 'request_id': 'req-33a93f2b-e04f-4a53-a88e-3ddff71f5d27'} {{(pid=81409) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 13 04:28:03.775424 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.notifications [None req-33a93f2b-e04f-4a53-a88e-3ddff71f5d27 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'cecec3eab7d3499a984c510355f61ebc', 'request_id': 'req-33a93f2b-e04f-4a53-a88e-3ddff71f5d27'} {{(pid=81409) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 13 04:28:03.781710 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.notifications [None req-33a93f2b-e04f-4a53-a88e-3ddff71f5d27 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'cecec3eab7d3499a984c510355f61ebc', 'request_id': 'req-33a93f2b-e04f-4a53-a88e-3ddff71f5d27'} {{(pid=81409) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 13 04:28:03.792985 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.notifications [None req-33a93f2b-e04f-4a53-a88e-3ddff71f5d27 admin admin] The token cache is being invalidate because project cecec3eab7d3499a984c510355f61ebc was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81409) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 13 04:28:03.793469 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.notifications [None req-33a93f2b-e04f-4a53-a88e-3ddff71f5d27 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81409) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 13 04:28:03.794339 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.notifications [None req-33a93f2b-e04f-4a53-a88e-3ddff71f5d27 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81409) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 13 04:28:03.796188 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 1109/2188] 10.222.0.27 () {58 vars in 1233 bytes} [Thu Oct 13 04:28:03 2022] DELETE /identity/v3/projects/cecec3eab7d3499a984c510355f61ebc => generated 0 bytes in 108 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 13 04:28:04.729993 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-115e2d60-3c9e-4414-a898-af57ac95d68a service neutron] Authenticating user token {{(pid=81408) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:28:04.738976 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-115e2d60-3c9e-4414-a898-af57ac95d68a service neutron] Validating token access rules against request {{(pid=81408) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:28:04.787778 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a94f7dd9-aea8-4bc3-9e14-88579dbbe1d3 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd8812c3974f84871bb2d761cf0016257', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dcec0431fb6849499d6aec35a234daa6', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81408) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:28:04.788378 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a94f7dd9-aea8-4bc3-9e14-88579dbbe1d3 service neutron] REQUEST_METHOD: `GET` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:28:04.788655 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a94f7dd9-aea8-4bc3-9e14-88579dbbe1d3 service neutron] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:28:04.788925 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a94f7dd9-aea8-4bc3-9e14-88579dbbe1d3 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:28:04.791384 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.fernet_utils [None req-a94f7dd9-aea8-4bc3-9e14-88579dbbe1d3 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81408) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:28:04.852649 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a94f7dd9-aea8-4bc3-9e14-88579dbbe1d3 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:28:04.854081 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a94f7dd9-aea8-4bc3-9e14-88579dbbe1d3 service neutron] RBAC: Authorization granted {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:28:04.863996 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 1080/2189] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Oct 13 04:28:04 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 135 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 13 04:28:08.195873 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-33a93f2b-e04f-4a53-a88e-3ddff71f5d27 admin admin] Authenticating user token {{(pid=81409) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:28:08.198392 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.fernet_utils [None req-33a93f2b-e04f-4a53-a88e-3ddff71f5d27 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81409) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:28:08.212831 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-33a93f2b-e04f-4a53-a88e-3ddff71f5d27 admin admin] Validating token access rules against request {{(pid=81409) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:28:08.217807 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2548ca0a-c195-41fc-a90d-0a62ccc12aea admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8ba823f313144da3b683bacef8fd62e0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81409) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:28:08.218471 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2548ca0a-c195-41fc-a90d-0a62ccc12aea admin admin] REQUEST_METHOD: `DELETE` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:28:08.218748 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2548ca0a-c195-41fc-a90d-0a62ccc12aea admin admin] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:28:08.219014 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2548ca0a-c195-41fc-a90d-0a62ccc12aea admin admin] PATH_INFO: `/v3/users/999c89e1bc7a426886fc802e28d38596` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:28:08.220134 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2548ca0a-c195-41fc-a90d-0a62ccc12aea admin admin] RBAC: Authorizing `identity:delete_user(user_id=999c89e1bc7a426886fc802e28d38596)` {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:28:08.222057 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2548ca0a-c195-41fc-a90d-0a62ccc12aea admin admin] RBAC: Authorization granted {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:28:08.272395 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.notifications [None req-2548ca0a-c195-41fc-a90d-0a62ccc12aea admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '999c89e1bc7a426886fc802e28d38596'} {{(pid=81409) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 13 04:28:08.273037 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.notifications [None req-2548ca0a-c195-41fc-a90d-0a62ccc12aea admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '999c89e1bc7a426886fc802e28d38596'} {{(pid=81409) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 13 04:28:08.280941 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.notifications [None req-2548ca0a-c195-41fc-a90d-0a62ccc12aea admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '999c89e1bc7a426886fc802e28d38596'} {{(pid=81409) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 13 04:28:08.281712 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.notifications [None req-2548ca0a-c195-41fc-a90d-0a62ccc12aea admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '999c89e1bc7a426886fc802e28d38596'} {{(pid=81409) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 13 04:28:08.291546 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.notifications [None req-2548ca0a-c195-41fc-a90d-0a62ccc12aea admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '999c89e1bc7a426886fc802e28d38596'} {{(pid=81409) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 13 04:28:08.311274 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 1110/2190] 10.222.0.27 () {58 vars in 1224 bytes} [Thu Oct 13 04:28:08 2022] DELETE /identity/v3/users/999c89e1bc7a426886fc802e28d38596 => generated 0 bytes in 117 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 13 04:28:08.518392 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a94f7dd9-aea8-4bc3-9e14-88579dbbe1d3 service neutron] Authenticating user token {{(pid=81408) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:28:08.532563 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a94f7dd9-aea8-4bc3-9e14-88579dbbe1d3 service neutron] Validating token access rules against request {{(pid=81408) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:28:08.592420 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG dbcounter [-] [81408] Writing DB stats keystone:SELECT=100 {{(pid=81408) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:28:08.599834 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8aca6841-965a-4459-9e52-398eb2b2c1a7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8ba823f313144da3b683bacef8fd62e0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81408) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:28:08.600860 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8aca6841-965a-4459-9e52-398eb2b2c1a7 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:28:08.601169 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8aca6841-965a-4459-9e52-398eb2b2c1a7 admin admin] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:28:08.601493 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8aca6841-965a-4459-9e52-398eb2b2c1a7 admin admin] PATH_INFO: `/v3/projects/7397177414dd4ed5baed133cc89e7299` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:28:08.602796 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8aca6841-965a-4459-9e52-398eb2b2c1a7 admin admin] RBAC: Authorizing `identity:delete_project(project_id=7397177414dd4ed5baed133cc89e7299)` {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:28:08.604726 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8aca6841-965a-4459-9e52-398eb2b2c1a7 admin admin] RBAC: Authorization granted {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:28:08.661535 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.notifications [None req-8aca6841-965a-4459-9e52-398eb2b2c1a7 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '7397177414dd4ed5baed133cc89e7299', 'request_id': 'req-8aca6841-965a-4459-9e52-398eb2b2c1a7'} {{(pid=81408) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 13 04:28:08.671507 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.notifications [None req-8aca6841-965a-4459-9e52-398eb2b2c1a7 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '7397177414dd4ed5baed133cc89e7299', 'request_id': 'req-8aca6841-965a-4459-9e52-398eb2b2c1a7'} {{(pid=81408) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 13 04:28:08.680022 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.notifications [None req-8aca6841-965a-4459-9e52-398eb2b2c1a7 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '7397177414dd4ed5baed133cc89e7299', 'request_id': 'req-8aca6841-965a-4459-9e52-398eb2b2c1a7'} {{(pid=81408) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 13 04:28:08.691745 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.notifications [None req-8aca6841-965a-4459-9e52-398eb2b2c1a7 admin admin] The token cache is being invalidate because project 7397177414dd4ed5baed133cc89e7299 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81408) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 13 04:28:08.692067 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.notifications [None req-8aca6841-965a-4459-9e52-398eb2b2c1a7 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81408) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 13 04:28:08.692718 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.notifications [None req-8aca6841-965a-4459-9e52-398eb2b2c1a7 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81408) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 13 04:28:08.694228 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 1081/2191] 10.222.0.27 () {58 vars in 1233 bytes} [Thu Oct 13 04:28:08 2022] DELETE /identity/v3/projects/7397177414dd4ed5baed133cc89e7299 => generated 0 bytes in 176 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 13 04:28:08.699115 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-424af2cd-ef39-40c4-8185-2696c9e00f15 None None] REQUEST_METHOD: `POST` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:28:08.699397 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-424af2cd-ef39-40c4-8185-2696c9e00f15 None None] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:28:08.699793 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-424af2cd-ef39-40c4-8185-2696c9e00f15 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:28:08.720857 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.auth.core [None req-424af2cd-ef39-40c4-8185-2696c9e00f15 None None] MFA Rules not processed for user `0e2d979563d6421995451a0e86bf3022`. Rule list: `[]` (Enabled: `True`). {{(pid=81409) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 13 04:28:08.722060 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.fernet_utils [None req-424af2cd-ef39-40c4-8185-2696c9e00f15 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81409) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:28:08.782518 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 1111/2192] 10.222.0.27 () {58 vars in 958 bytes} [Thu Oct 13 04:28:08 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 84 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 13 04:28:08.790853 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8adfb3b5-99b4-412d-88bc-cd6048c98594 None None] REQUEST_METHOD: `POST` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:28:08.791347 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8adfb3b5-99b4-412d-88bc-cd6048c98594 None None] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:28:08.791627 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8adfb3b5-99b4-412d-88bc-cd6048c98594 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:28:08.811646 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.auth.core [None req-8adfb3b5-99b4-412d-88bc-cd6048c98594 None None] MFA Rules not processed for user `0e2d979563d6421995451a0e86bf3022`. Rule list: `[]` (Enabled: `True`). {{(pid=81408) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 13 04:28:08.812927 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.fernet_utils [None req-8adfb3b5-99b4-412d-88bc-cd6048c98594 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81408) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:28:08.821640 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 1082/2193] 10.222.0.27 () {58 vars in 958 bytes} [Thu Oct 13 04:28:08 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 32 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 13 04:28:08.826655 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-424af2cd-ef39-40c4-8185-2696c9e00f15 None None] Authenticating user token {{(pid=81409) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:28:08.837474 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-424af2cd-ef39-40c4-8185-2696c9e00f15 None None] Validating token access rules against request {{(pid=81409) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:28:08.842587 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4810a732-8a82-4c5b-8bb6-4499a3b81445 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8ba823f313144da3b683bacef8fd62e0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81409) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:28:08.843236 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4810a732-8a82-4c5b-8bb6-4499a3b81445 admin admin] REQUEST_METHOD: `GET` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:28:08.843509 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4810a732-8a82-4c5b-8bb6-4499a3b81445 admin admin] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:28:08.843798 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4810a732-8a82-4c5b-8bb6-4499a3b81445 admin admin] PATH_INFO: `/v3/domains` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:28:08.844457 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4810a732-8a82-4c5b-8bb6-4499a3b81445 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81409) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 13 04:28:08.844763 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4810a732-8a82-4c5b-8bb6-4499a3b81445 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:28:08.846459 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4810a732-8a82-4c5b-8bb6-4499a3b81445 admin admin] RBAC: Authorization granted {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:28:08.860845 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 1112/2194] 10.222.0.27 () {58 vars in 1153 bytes} [Thu Oct 13 04:28:08 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 35 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 13 04:28:08.865508 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8adfb3b5-99b4-412d-88bc-cd6048c98594 None None] Authenticating user token {{(pid=81408) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:28:08.877832 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8adfb3b5-99b4-412d-88bc-cd6048c98594 None None] Validating token access rules against request {{(pid=81408) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:28:08.883118 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9a0358c7-41c2-48ac-b364-ee3e1d9820d4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8ba823f313144da3b683bacef8fd62e0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81408) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:28:08.883748 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a0358c7-41c2-48ac-b364-ee3e1d9820d4 admin admin] REQUEST_METHOD: `POST` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:28:08.884139 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a0358c7-41c2-48ac-b364-ee3e1d9820d4 admin admin] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:28:08.884417 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a0358c7-41c2-48ac-b364-ee3e1d9820d4 admin admin] PATH_INFO: `/v3/projects` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:28:08.885072 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9a0358c7-41c2-48ac-b364-ee3e1d9820d4 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:28:08.887025 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9a0358c7-41c2-48ac-b364-ee3e1d9820d4 admin admin] RBAC: Authorization granted {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:28:08.916295 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 1083/2195] 10.222.0.27 () {60 vars in 1153 bytes} [Thu Oct 13 04:28:08 2022] POST /identity/v3/projects => generated 365 bytes in 51 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 13 04:28:08.920776 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4810a732-8a82-4c5b-8bb6-4499a3b81445 admin admin] Authenticating user token {{(pid=81409) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:28:08.931967 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4810a732-8a82-4c5b-8bb6-4499a3b81445 admin admin] Validating token access rules against request {{(pid=81409) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:28:08.966621 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG dbcounter [-] [81409] Writing DB stats keystone:DELETE=14,keystone:SELECT=84,keystone:INSERT=2 {{(pid=81409) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:28:08.996369 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c67db4fb-a06b-42b7-a077-0b460e2b4a29 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8ba823f313144da3b683bacef8fd62e0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81409) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:28:08.996973 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c67db4fb-a06b-42b7-a077-0b460e2b4a29 admin admin] REQUEST_METHOD: `POST` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:28:08.997250 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c67db4fb-a06b-42b7-a077-0b460e2b4a29 admin admin] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:28:08.997519 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c67db4fb-a06b-42b7-a077-0b460e2b4a29 admin admin] PATH_INFO: `/v3/users` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:28:08.998153 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c67db4fb-a06b-42b7-a077-0b460e2b4a29 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:28:09.000117 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c67db4fb-a06b-42b7-a077-0b460e2b4a29 admin admin] RBAC: Authorization granted {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:28:09.020445 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 1113/2196] 10.222.0.27 () {60 vars in 1144 bytes} [Thu Oct 13 04:28:08 2022] POST /identity/v3/users => generated 283 bytes in 100 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 13 04:28:09.024536 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9a0358c7-41c2-48ac-b364-ee3e1d9820d4 admin admin] Authenticating user token {{(pid=81408) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:28:09.034654 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9a0358c7-41c2-48ac-b364-ee3e1d9820d4 admin admin] Validating token access rules against request {{(pid=81408) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:28:09.039850 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-18fcf089-57da-4e3f-a677-71fdc5401021 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8ba823f313144da3b683bacef8fd62e0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81408) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:28:09.040436 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18fcf089-57da-4e3f-a677-71fdc5401021 admin admin] REQUEST_METHOD: `GET` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:28:09.040713 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18fcf089-57da-4e3f-a677-71fdc5401021 admin admin] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:28:09.040982 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18fcf089-57da-4e3f-a677-71fdc5401021 admin admin] PATH_INFO: `/v3/roles` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:28:09.041614 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-18fcf089-57da-4e3f-a677-71fdc5401021 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:28:09.043258 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-18fcf089-57da-4e3f-a677-71fdc5401021 admin admin] RBAC: Authorization granted {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:28:09.054720 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 1084/2197] 10.222.0.27 () {58 vars in 1122 bytes} [Thu Oct 13 04:28:09 2022] GET /identity/v3/roles => generated 1420 bytes in 29 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 13 04:28:09.057909 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c67db4fb-a06b-42b7-a077-0b460e2b4a29 admin admin] Authenticating user token {{(pid=81409) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:28:09.067863 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c67db4fb-a06b-42b7-a077-0b460e2b4a29 admin admin] Validating token access rules against request {{(pid=81409) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:28:09.072527 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3e3bbc71-2ce2-4c15-b4b8-ba7497f49a99 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8ba823f313144da3b683bacef8fd62e0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81409) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:28:09.073119 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e3bbc71-2ce2-4c15-b4b8-ba7497f49a99 admin admin] REQUEST_METHOD: `GET` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:28:09.073392 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e3bbc71-2ce2-4c15-b4b8-ba7497f49a99 admin admin] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:28:09.073659 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e3bbc71-2ce2-4c15-b4b8-ba7497f49a99 admin admin] PATH_INFO: `/v3/roles` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:28:09.074489 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3e3bbc71-2ce2-4c15-b4b8-ba7497f49a99 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:28:09.076192 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3e3bbc71-2ce2-4c15-b4b8-ba7497f49a99 admin admin] RBAC: Authorization granted {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:28:09.087355 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 1114/2198] 10.222.0.27 () {58 vars in 1122 bytes} [Thu Oct 13 04:28:09 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 13 04:28:09.091636 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-18fcf089-57da-4e3f-a677-71fdc5401021 admin admin] Authenticating user token {{(pid=81408) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:28:09.102372 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-18fcf089-57da-4e3f-a677-71fdc5401021 admin admin] Validating token access rules against request {{(pid=81408) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:28:09.107318 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-00adc6c1-2698-4db8-9419-904c0191bfb4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8ba823f313144da3b683bacef8fd62e0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81408) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:28:09.107959 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00adc6c1-2698-4db8-9419-904c0191bfb4 admin admin] REQUEST_METHOD: `PUT` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:28:09.108237 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00adc6c1-2698-4db8-9419-904c0191bfb4 admin admin] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:28:09.108512 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00adc6c1-2698-4db8-9419-904c0191bfb4 admin admin] PATH_INFO: `/v3/projects/2180a790f75442e2b2c230c7f247103b/users/0d7c8742c38444edb9c3d8932d617a92/roles/cc56c4ae13a7424ea771c15da4b153c1` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:28:09.125607 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-00adc6c1-2698-4db8-9419-904c0191bfb4 admin admin] RBAC: Authorizing `identity:create_grant(project_id=2180a790f75442e2b2c230c7f247103b, user_id=0d7c8742c38444edb9c3d8932d617a92, role_id=cc56c4ae13a7424ea771c15da4b153c1)` {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:28:09.128384 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-00adc6c1-2698-4db8-9419-904c0191bfb4 admin admin] RBAC: Authorization granted {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:28:09.142184 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 1085/2199] 10.222.0.27 () {60 vars in 1483 bytes} [Thu Oct 13 04:28:09 2022] PUT /identity/v3/projects/2180a790f75442e2b2c230c7f247103b/users/0d7c8742c38444edb9c3d8932d617a92/roles/cc56c4ae13a7424ea771c15da4b153c1 => generated 0 bytes in 51 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 13 04:28:09.155994 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bbdc74f6-cafc-4e6c-bfd0-26f6ffd09699 None None] REQUEST_METHOD: `POST` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:28:09.156273 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bbdc74f6-cafc-4e6c-bfd0-26f6ffd09699 None None] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:28:09.156543 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bbdc74f6-cafc-4e6c-bfd0-26f6ffd09699 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:28:09.186835 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.auth.core [None req-bbdc74f6-cafc-4e6c-bfd0-26f6ffd09699 None None] MFA Rules not processed for user `0d7c8742c38444edb9c3d8932d617a92`. Rule list: `[]` (Enabled: `True`). {{(pid=81409) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 13 04:28:09.188162 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.fernet_utils [None req-bbdc74f6-cafc-4e6c-bfd0-26f6ffd09699 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81409) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:28:09.256035 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 1115/2200] 10.222.0.27 () {58 vars in 958 bytes} [Thu Oct 13 04:28:09 2022] POST /identity/v3/auth/tokens => generated 3242 bytes in 101 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 13 04:28:09.260999 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-00adc6c1-2698-4db8-9419-904c0191bfb4 admin admin] Authenticating user token {{(pid=81408) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:28:09.323277 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-00adc6c1-2698-4db8-9419-904c0191bfb4 admin admin] Validating token access rules against request {{(pid=81408) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:28:09.327927 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7b8e815d-d6e0-4253-a694-10ea06c0f8f9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8ba823f313144da3b683bacef8fd62e0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81408) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:28:09.328529 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b8e815d-d6e0-4253-a694-10ea06c0f8f9 admin admin] REQUEST_METHOD: `POST` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:28:09.328807 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b8e815d-d6e0-4253-a694-10ea06c0f8f9 admin admin] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:28:09.329079 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b8e815d-d6e0-4253-a694-10ea06c0f8f9 admin admin] PATH_INFO: `/v3/projects` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:28:09.329713 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7b8e815d-d6e0-4253-a694-10ea06c0f8f9 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:28:09.331653 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7b8e815d-d6e0-4253-a694-10ea06c0f8f9 admin admin] RBAC: Authorization granted {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:28:09.362543 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 1086/2201] 10.222.0.27 () {60 vars in 1153 bytes} [Thu Oct 13 04:28:09 2022] POST /identity/v3/projects => generated 365 bytes in 99 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 13 04:28:09.363665 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bbdc74f6-cafc-4e6c-bfd0-26f6ffd09699 None None] Authenticating user token {{(pid=81409) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:28:09.374851 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bbdc74f6-cafc-4e6c-bfd0-26f6ffd09699 None None] Validating token access rules against request {{(pid=81409) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:28:09.437567 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3f0af6a3-3ebf-4231-9ad3-ed057bb8b223 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8ba823f313144da3b683bacef8fd62e0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81409) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:28:09.438182 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f0af6a3-3ebf-4231-9ad3-ed057bb8b223 admin admin] REQUEST_METHOD: `POST` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:28:09.438492 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f0af6a3-3ebf-4231-9ad3-ed057bb8b223 admin admin] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:28:09.438763 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f0af6a3-3ebf-4231-9ad3-ed057bb8b223 admin admin] PATH_INFO: `/v3/users` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:28:09.439449 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3f0af6a3-3ebf-4231-9ad3-ed057bb8b223 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:28:09.441342 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3f0af6a3-3ebf-4231-9ad3-ed057bb8b223 admin admin] RBAC: Authorization granted {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:28:09.466332 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 1116/2202] 10.222.0.27 () {60 vars in 1144 bytes} [Thu Oct 13 04:28:09 2022] POST /identity/v3/users => generated 289 bytes in 100 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 13 04:28:09.467454 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7b8e815d-d6e0-4253-a694-10ea06c0f8f9 admin admin] Authenticating user token {{(pid=81408) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:28:09.479127 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7b8e815d-d6e0-4253-a694-10ea06c0f8f9 admin admin] Validating token access rules against request {{(pid=81408) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:28:09.484219 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3b827c22-de93-483a-82f4-5dfe6380021e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8ba823f313144da3b683bacef8fd62e0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81408) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:28:09.484929 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b827c22-de93-483a-82f4-5dfe6380021e admin admin] REQUEST_METHOD: `GET` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:28:09.485273 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b827c22-de93-483a-82f4-5dfe6380021e admin admin] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:28:09.485607 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b827c22-de93-483a-82f4-5dfe6380021e admin admin] PATH_INFO: `/v3/roles` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:28:09.486459 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3b827c22-de93-483a-82f4-5dfe6380021e admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:28:09.488131 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3b827c22-de93-483a-82f4-5dfe6380021e admin admin] RBAC: Authorization granted {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:28:09.490580 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG dbcounter [-] [81408] Writing DB stats keystone:SELECT=89,keystone:DELETE=7,keystone:INSERT=4 {{(pid=81408) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:28:09.501257 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 1087/2203] 10.222.0.27 () {58 vars in 1122 bytes} [Thu Oct 13 04:28:09 2022] GET /identity/v3/roles => generated 1420 bytes in 34 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 13 04:28:09.505043 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3f0af6a3-3ebf-4231-9ad3-ed057bb8b223 admin admin] Authenticating user token {{(pid=81409) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:28:09.515380 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3f0af6a3-3ebf-4231-9ad3-ed057bb8b223 admin admin] Validating token access rules against request {{(pid=81409) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:28:09.520052 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-28dfd789-e7b9-4af0-80b7-c99f6aca5c9f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8ba823f313144da3b683bacef8fd62e0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81409) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:28:09.520710 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28dfd789-e7b9-4af0-80b7-c99f6aca5c9f admin admin] REQUEST_METHOD: `PUT` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:28:09.520983 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28dfd789-e7b9-4af0-80b7-c99f6aca5c9f admin admin] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:28:09.521253 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28dfd789-e7b9-4af0-80b7-c99f6aca5c9f admin admin] PATH_INFO: `/v3/projects/f7fa6871e2d54542bc285456ead4ac27/users/3cc02f2a741b41d7a6deeeb502f79624/roles/ef55fbd94e1040be83695754075a7ed5` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:28:09.538693 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-28dfd789-e7b9-4af0-80b7-c99f6aca5c9f admin admin] RBAC: Authorizing `identity:create_grant(project_id=f7fa6871e2d54542bc285456ead4ac27, user_id=3cc02f2a741b41d7a6deeeb502f79624, role_id=ef55fbd94e1040be83695754075a7ed5)` {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:28:09.541674 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-28dfd789-e7b9-4af0-80b7-c99f6aca5c9f admin admin] RBAC: Authorization granted {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:28:09.557015 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 1117/2204] 10.222.0.27 () {60 vars in 1483 bytes} [Thu Oct 13 04:28:09 2022] PUT /identity/v3/projects/f7fa6871e2d54542bc285456ead4ac27/users/3cc02f2a741b41d7a6deeeb502f79624/roles/ef55fbd94e1040be83695754075a7ed5 => generated 0 bytes in 52 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 13 04:28:09.561052 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3b827c22-de93-483a-82f4-5dfe6380021e admin admin] Authenticating user token {{(pid=81408) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:28:09.572789 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3b827c22-de93-483a-82f4-5dfe6380021e admin admin] Validating token access rules against request {{(pid=81408) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:28:09.631557 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8179628c-d4d7-4b14-a4a4-e5e0e6f08cdf admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8ba823f313144da3b683bacef8fd62e0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81408) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:28:09.632263 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8179628c-d4d7-4b14-a4a4-e5e0e6f08cdf admin admin] REQUEST_METHOD: `GET` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:28:09.632551 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8179628c-d4d7-4b14-a4a4-e5e0e6f08cdf admin admin] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:28:09.633424 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8179628c-d4d7-4b14-a4a4-e5e0e6f08cdf admin admin] PATH_INFO: `/v3/roles` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:28:09.634070 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8179628c-d4d7-4b14-a4a4-e5e0e6f08cdf admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:28:09.635708 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8179628c-d4d7-4b14-a4a4-e5e0e6f08cdf admin admin] RBAC: Authorization granted {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:28:09.646450 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 1088/2205] 10.222.0.27 () {58 vars in 1122 bytes} [Thu Oct 13 04:28:09 2022] GET /identity/v3/roles => generated 1420 bytes in 86 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 13 04:28:09.650543 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-28dfd789-e7b9-4af0-80b7-c99f6aca5c9f admin admin] Authenticating user token {{(pid=81409) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:28:09.661500 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-28dfd789-e7b9-4af0-80b7-c99f6aca5c9f admin admin] Validating token access rules against request {{(pid=81409) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:28:09.667169 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e7ac4ee3-ffc9-4490-bfff-31bf0d5fe3ad admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8ba823f313144da3b683bacef8fd62e0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81409) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:28:09.667822 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7ac4ee3-ffc9-4490-bfff-31bf0d5fe3ad admin admin] REQUEST_METHOD: `PUT` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:28:09.668098 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7ac4ee3-ffc9-4490-bfff-31bf0d5fe3ad admin admin] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:28:09.668421 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7ac4ee3-ffc9-4490-bfff-31bf0d5fe3ad admin admin] PATH_INFO: `/v3/projects/f7fa6871e2d54542bc285456ead4ac27/users/3cc02f2a741b41d7a6deeeb502f79624/roles/ef55fbd94e1040be83695754075a7ed5` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:28:09.670332 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e7ac4ee3-ffc9-4490-bfff-31bf0d5fe3ad admin admin] RBAC: Authorizing `identity:create_grant(project_id=f7fa6871e2d54542bc285456ead4ac27, user_id=3cc02f2a741b41d7a6deeeb502f79624, role_id=ef55fbd94e1040be83695754075a7ed5)` {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:28:09.672993 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e7ac4ee3-ffc9-4490-bfff-31bf0d5fe3ad admin admin] RBAC: Authorization granted {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:28:09.675355 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG dbcounter [-] [81409] Writing DB stats keystone:SELECT=93,keystone:INSERT=7 {{(pid=81409) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:28:09.686266 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 1118/2206] 10.222.0.27 () {60 vars in 1483 bytes} [Thu Oct 13 04:28:09 2022] PUT /identity/v3/projects/f7fa6871e2d54542bc285456ead4ac27/users/3cc02f2a741b41d7a6deeeb502f79624/roles/ef55fbd94e1040be83695754075a7ed5 => generated 0 bytes in 36 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 13 04:28:09.698115 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-160318bc-b99f-4896-a724-3858aa0934bf None None] REQUEST_METHOD: `POST` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:28:09.698434 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-160318bc-b99f-4896-a724-3858aa0934bf None None] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:28:09.698704 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-160318bc-b99f-4896-a724-3858aa0934bf None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:28:09.732827 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.auth.core [None req-160318bc-b99f-4896-a724-3858aa0934bf None None] MFA Rules not processed for user `3cc02f2a741b41d7a6deeeb502f79624`. Rule list: `[]` (Enabled: `True`). {{(pid=81408) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 13 04:28:09.734072 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.fernet_utils [None req-160318bc-b99f-4896-a724-3858aa0934bf None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81408) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:28:09.800669 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 1089/2207] 10.222.0.27 () {58 vars in 958 bytes} [Thu Oct 13 04:28:09 2022] POST /identity/v3/auth/tokens => generated 3309 bytes in 104 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 13 04:28:09.812110 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e7ac4ee3-ffc9-4490-bfff-31bf0d5fe3ad admin admin] Authenticating user token {{(pid=81409) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:28:09.813847 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.fernet_utils [None req-e7ac4ee3-ffc9-4490-bfff-31bf0d5fe3ad admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81409) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:28:09.871748 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e7ac4ee3-ffc9-4490-bfff-31bf0d5fe3ad admin admin] Validating token access rules against request {{(pid=81409) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:28:09.875413 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-63aa46a8-c301-4bc4-94b6-dc9b73e301c1 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6141b8a6c3d54e179d6879bb019f4d12', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dcec0431fb6849499d6aec35a234daa6', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81409) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:28:09.876008 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63aa46a8-c301-4bc4-94b6-dc9b73e301c1 service cinder] REQUEST_METHOD: `GET` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:28:09.876282 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63aa46a8-c301-4bc4-94b6-dc9b73e301c1 service cinder] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:28:09.876550 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63aa46a8-c301-4bc4-94b6-dc9b73e301c1 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:28:09.886113 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-63aa46a8-c301-4bc4-94b6-dc9b73e301c1 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:28:09.887558 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-63aa46a8-c301-4bc4-94b6-dc9b73e301c1 service cinder] RBAC: Authorization granted {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:28:09.896841 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 1119/2208] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Oct 13 04:28:09 2022] GET /identity/v3/auth/tokens => generated 3309 bytes in 85 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 13 04:28:10.306424 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-160318bc-b99f-4896-a724-3858aa0934bf None None] Authenticating user token {{(pid=81408) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:28:10.368199 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-160318bc-b99f-4896-a724-3858aa0934bf None None] Validating token access rules against request {{(pid=81408) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:28:10.372662 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5c8197bc-4b3a-4876-a12e-ccdfb04a9f7d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8ba823f313144da3b683bacef8fd62e0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81408) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:28:10.373285 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c8197bc-4b3a-4876-a12e-ccdfb04a9f7d admin admin] REQUEST_METHOD: `DELETE` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:28:10.373559 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c8197bc-4b3a-4876-a12e-ccdfb04a9f7d admin admin] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:28:10.373830 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c8197bc-4b3a-4876-a12e-ccdfb04a9f7d admin admin] PATH_INFO: `/v3/users/0d7c8742c38444edb9c3d8932d617a92` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:28:10.374993 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5c8197bc-4b3a-4876-a12e-ccdfb04a9f7d admin admin] RBAC: Authorizing `identity:delete_user(user_id=0d7c8742c38444edb9c3d8932d617a92)` {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:28:10.376927 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5c8197bc-4b3a-4876-a12e-ccdfb04a9f7d admin admin] RBAC: Authorization granted {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:28:10.399981 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG dbcounter [-] [81408] Writing DB stats keystone:SELECT=97,keystone:DELETE=3 {{(pid=81408) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:28:10.427030 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.notifications [None req-5c8197bc-4b3a-4876-a12e-ccdfb04a9f7d admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '0d7c8742c38444edb9c3d8932d617a92'} {{(pid=81408) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 13 04:28:10.434112 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.notifications [None req-5c8197bc-4b3a-4876-a12e-ccdfb04a9f7d admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '0d7c8742c38444edb9c3d8932d617a92'} {{(pid=81408) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 13 04:28:10.434797 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.notifications [None req-5c8197bc-4b3a-4876-a12e-ccdfb04a9f7d admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '0d7c8742c38444edb9c3d8932d617a92'} {{(pid=81408) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 13 04:28:10.435411 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.notifications [None req-5c8197bc-4b3a-4876-a12e-ccdfb04a9f7d admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '0d7c8742c38444edb9c3d8932d617a92'} {{(pid=81408) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 13 04:28:10.442657 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.notifications [None req-5c8197bc-4b3a-4876-a12e-ccdfb04a9f7d admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '0d7c8742c38444edb9c3d8932d617a92'} {{(pid=81408) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 13 04:28:10.460069 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 1090/2209] 10.222.0.27 () {58 vars in 1224 bytes} [Thu Oct 13 04:28:10 2022] DELETE /identity/v3/users/0d7c8742c38444edb9c3d8932d617a92 => generated 0 bytes in 154 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 13 04:28:10.465901 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e15b460-3eea-4b2b-914d-e7ae56d94f75 None None] REQUEST_METHOD: `POST` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:28:10.466402 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e15b460-3eea-4b2b-914d-e7ae56d94f75 None None] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:28:10.466534 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e15b460-3eea-4b2b-914d-e7ae56d94f75 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:28:10.485942 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.auth.core [None req-5e15b460-3eea-4b2b-914d-e7ae56d94f75 None None] MFA Rules not processed for user `0e2d979563d6421995451a0e86bf3022`. Rule list: `[]` (Enabled: `True`). {{(pid=81409) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 13 04:28:10.487111 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.fernet_utils [None req-5e15b460-3eea-4b2b-914d-e7ae56d94f75 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81409) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:28:10.545870 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 1120/2210] 10.222.0.27 () {58 vars in 958 bytes} [Thu Oct 13 04:28:10 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 81 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 13 04:28:10.555653 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5c8197bc-4b3a-4876-a12e-ccdfb04a9f7d admin admin] Authenticating user token {{(pid=81408) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:28:10.557053 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.fernet_utils [None req-5c8197bc-4b3a-4876-a12e-ccdfb04a9f7d admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81408) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:28:10.614573 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5c8197bc-4b3a-4876-a12e-ccdfb04a9f7d admin admin] Validating token access rules against request {{(pid=81408) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:28:10.619198 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ddfcd412-5f5f-4295-b2ef-ae1087d00ab9 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd8812c3974f84871bb2d761cf0016257', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dcec0431fb6849499d6aec35a234daa6', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81408) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:28:10.619796 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ddfcd412-5f5f-4295-b2ef-ae1087d00ab9 service neutron] REQUEST_METHOD: `GET` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:28:10.620071 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ddfcd412-5f5f-4295-b2ef-ae1087d00ab9 service neutron] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:28:10.620342 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ddfcd412-5f5f-4295-b2ef-ae1087d00ab9 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:28:10.629172 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ddfcd412-5f5f-4295-b2ef-ae1087d00ab9 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:28:10.630670 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ddfcd412-5f5f-4295-b2ef-ae1087d00ab9 service neutron] RBAC: Authorization granted {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:28:10.640259 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 1091/2211] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Oct 13 04:28:10 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 85 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 13 04:28:10.938946 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5e15b460-3eea-4b2b-914d-e7ae56d94f75 None None] Authenticating user token {{(pid=81409) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:28:10.940571 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.fernet_utils [None req-5e15b460-3eea-4b2b-914d-e7ae56d94f75 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81409) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:28:10.953925 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5e15b460-3eea-4b2b-914d-e7ae56d94f75 None None] Validating token access rules against request {{(pid=81409) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:28:10.959637 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-88685d10-beb6-4104-91f1-f324a2a3992c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8ba823f313144da3b683bacef8fd62e0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81409) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:28:10.960283 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88685d10-beb6-4104-91f1-f324a2a3992c admin admin] REQUEST_METHOD: `DELETE` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:28:10.960684 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88685d10-beb6-4104-91f1-f324a2a3992c admin admin] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:28:10.961012 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88685d10-beb6-4104-91f1-f324a2a3992c admin admin] PATH_INFO: `/v3/projects/2180a790f75442e2b2c230c7f247103b` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:28:10.962223 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-88685d10-beb6-4104-91f1-f324a2a3992c admin admin] RBAC: Authorizing `identity:delete_project(project_id=2180a790f75442e2b2c230c7f247103b)` {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:28:10.964380 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-88685d10-beb6-4104-91f1-f324a2a3992c admin admin] RBAC: Authorization granted {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:28:11.012753 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.notifications [None req-88685d10-beb6-4104-91f1-f324a2a3992c admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '2180a790f75442e2b2c230c7f247103b', 'request_id': 'req-88685d10-beb6-4104-91f1-f324a2a3992c'} {{(pid=81409) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 13 04:28:11.020179 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.notifications [None req-88685d10-beb6-4104-91f1-f324a2a3992c admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '2180a790f75442e2b2c230c7f247103b', 'request_id': 'req-88685d10-beb6-4104-91f1-f324a2a3992c'} {{(pid=81409) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 13 04:28:11.027619 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.notifications [None req-88685d10-beb6-4104-91f1-f324a2a3992c admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '2180a790f75442e2b2c230c7f247103b', 'request_id': 'req-88685d10-beb6-4104-91f1-f324a2a3992c'} {{(pid=81409) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 13 04:28:11.038376 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.notifications [None req-88685d10-beb6-4104-91f1-f324a2a3992c admin admin] The token cache is being invalidate because project 2180a790f75442e2b2c230c7f247103b was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81409) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 13 04:28:11.038668 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.notifications [None req-88685d10-beb6-4104-91f1-f324a2a3992c admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81409) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 13 04:28:11.040033 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.notifications [None req-88685d10-beb6-4104-91f1-f324a2a3992c admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81409) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 13 04:28:11.042194 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 1121/2212] 10.222.0.27 () {58 vars in 1233 bytes} [Thu Oct 13 04:28:10 2022] DELETE /identity/v3/projects/2180a790f75442e2b2c230c7f247103b => generated 0 bytes in 104 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 13 04:28:11.046653 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ddfcd412-5f5f-4295-b2ef-ae1087d00ab9 service neutron] Authenticating user token {{(pid=81408) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:28:11.056353 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ddfcd412-5f5f-4295-b2ef-ae1087d00ab9 service neutron] Validating token access rules against request {{(pid=81408) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:28:11.120459 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-de1b6379-55b3-467b-a01c-ba04e3f7064f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8ba823f313144da3b683bacef8fd62e0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81408) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:28:11.120859 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de1b6379-55b3-467b-a01c-ba04e3f7064f admin admin] REQUEST_METHOD: `DELETE` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:28:11.121149 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de1b6379-55b3-467b-a01c-ba04e3f7064f admin admin] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:28:11.121457 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de1b6379-55b3-467b-a01c-ba04e3f7064f admin admin] PATH_INFO: `/v3/users/3cc02f2a741b41d7a6deeeb502f79624` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:28:11.122787 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-de1b6379-55b3-467b-a01c-ba04e3f7064f admin admin] RBAC: Authorizing `identity:delete_user(user_id=3cc02f2a741b41d7a6deeeb502f79624)` {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:28:11.125039 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-de1b6379-55b3-467b-a01c-ba04e3f7064f admin admin] RBAC: Authorization granted {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:28:11.173086 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG dbcounter [-] [81408] Writing DB stats keystone:DELETE=13,keystone:SELECT=86,keystone:INSERT=1 {{(pid=81408) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:28:11.178166 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.notifications [None req-de1b6379-55b3-467b-a01c-ba04e3f7064f admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '3cc02f2a741b41d7a6deeeb502f79624'} {{(pid=81408) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 13 04:28:11.188210 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.notifications [None req-de1b6379-55b3-467b-a01c-ba04e3f7064f admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '3cc02f2a741b41d7a6deeeb502f79624'} {{(pid=81408) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 13 04:28:11.189001 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.notifications [None req-de1b6379-55b3-467b-a01c-ba04e3f7064f admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '3cc02f2a741b41d7a6deeeb502f79624'} {{(pid=81408) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 13 04:28:11.189692 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.notifications [None req-de1b6379-55b3-467b-a01c-ba04e3f7064f admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '3cc02f2a741b41d7a6deeeb502f79624'} {{(pid=81408) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 13 04:28:11.196903 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.notifications [None req-de1b6379-55b3-467b-a01c-ba04e3f7064f admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '3cc02f2a741b41d7a6deeeb502f79624'} {{(pid=81408) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 13 04:28:11.216025 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 1092/2213] 10.222.0.27 () {58 vars in 1224 bytes} [Thu Oct 13 04:28:11 2022] DELETE /identity/v3/users/3cc02f2a741b41d7a6deeeb502f79624 => generated 0 bytes in 170 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 13 04:28:11.497392 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-88685d10-beb6-4104-91f1-f324a2a3992c admin admin] Authenticating user token {{(pid=81409) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:28:11.498972 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.fernet_utils [None req-88685d10-beb6-4104-91f1-f324a2a3992c admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81409) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:28:11.511951 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-88685d10-beb6-4104-91f1-f324a2a3992c admin admin] Validating token access rules against request {{(pid=81409) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:28:11.529670 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG dbcounter [-] [81409] Writing DB stats keystone:INSERT=2,keystone:SELECT=91,keystone:DELETE=7 {{(pid=81409) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:28:11.572494 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cd39673d-d9de-4854-b96d-619fa784871e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8ba823f313144da3b683bacef8fd62e0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81409) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:28:11.573117 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd39673d-d9de-4854-b96d-619fa784871e admin admin] REQUEST_METHOD: `DELETE` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:28:11.573388 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd39673d-d9de-4854-b96d-619fa784871e admin admin] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:28:11.573653 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd39673d-d9de-4854-b96d-619fa784871e admin admin] PATH_INFO: `/v3/projects/f7fa6871e2d54542bc285456ead4ac27` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:28:11.574773 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cd39673d-d9de-4854-b96d-619fa784871e admin admin] RBAC: Authorizing `identity:delete_project(project_id=f7fa6871e2d54542bc285456ead4ac27)` {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:28:11.576703 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cd39673d-d9de-4854-b96d-619fa784871e admin admin] RBAC: Authorization granted {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:28:11.622914 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.notifications [None req-cd39673d-d9de-4854-b96d-619fa784871e admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'f7fa6871e2d54542bc285456ead4ac27', 'request_id': 'req-cd39673d-d9de-4854-b96d-619fa784871e'} {{(pid=81409) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 13 04:28:11.630410 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.notifications [None req-cd39673d-d9de-4854-b96d-619fa784871e admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'f7fa6871e2d54542bc285456ead4ac27', 'request_id': 'req-cd39673d-d9de-4854-b96d-619fa784871e'} {{(pid=81409) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 13 04:28:11.637050 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.notifications [None req-cd39673d-d9de-4854-b96d-619fa784871e admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'f7fa6871e2d54542bc285456ead4ac27', 'request_id': 'req-cd39673d-d9de-4854-b96d-619fa784871e'} {{(pid=81409) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 13 04:28:11.647198 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.notifications [None req-cd39673d-d9de-4854-b96d-619fa784871e admin admin] The token cache is being invalidate because project f7fa6871e2d54542bc285456ead4ac27 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81409) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 13 04:28:11.647478 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.notifications [None req-cd39673d-d9de-4854-b96d-619fa784871e admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81409) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 13 04:28:11.648100 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.notifications [None req-cd39673d-d9de-4854-b96d-619fa784871e admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81409) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 13 04:28:11.650316 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 1122/2214] 10.222.0.27 () {58 vars in 1233 bytes} [Thu Oct 13 04:28:11 2022] DELETE /identity/v3/projects/f7fa6871e2d54542bc285456ead4ac27 => generated 0 bytes in 154 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 13 04:28:11.655131 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27851c99-2142-45a3-9300-6dd94ac4aea5 None None] REQUEST_METHOD: `POST` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:28:11.655619 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27851c99-2142-45a3-9300-6dd94ac4aea5 None None] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:28:11.655733 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27851c99-2142-45a3-9300-6dd94ac4aea5 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:28:11.675965 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.auth.core [None req-27851c99-2142-45a3-9300-6dd94ac4aea5 None None] MFA Rules not processed for user `0e2d979563d6421995451a0e86bf3022`. Rule list: `[]` (Enabled: `True`). {{(pid=81408) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 13 04:28:11.677160 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.fernet_utils [None req-27851c99-2142-45a3-9300-6dd94ac4aea5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81408) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:28:11.736608 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 1093/2215] 10.222.0.27 () {58 vars in 958 bytes} [Thu Oct 13 04:28:11 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 82 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 13 04:28:11.744579 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef87b37c-e9e8-46ea-9e67-473f12aae026 None None] REQUEST_METHOD: `POST` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:28:11.744859 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef87b37c-e9e8-46ea-9e67-473f12aae026 None None] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:28:11.745132 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef87b37c-e9e8-46ea-9e67-473f12aae026 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:28:11.764914 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.auth.core [None req-ef87b37c-e9e8-46ea-9e67-473f12aae026 None None] MFA Rules not processed for user `0e2d979563d6421995451a0e86bf3022`. Rule list: `[]` (Enabled: `True`). {{(pid=81409) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 13 04:28:11.766037 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.fernet_utils [None req-ef87b37c-e9e8-46ea-9e67-473f12aae026 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81409) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:28:11.774406 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 1123/2216] 10.222.0.27 () {58 vars in 958 bytes} [Thu Oct 13 04:28:11 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 31 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 13 04:28:11.778506 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-27851c99-2142-45a3-9300-6dd94ac4aea5 None None] Authenticating user token {{(pid=81408) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:28:11.788181 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-27851c99-2142-45a3-9300-6dd94ac4aea5 None None] Validating token access rules against request {{(pid=81408) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:28:11.794131 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-701dab4d-9c8e-4780-806f-ed2ece6a4f62 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8ba823f313144da3b683bacef8fd62e0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81408) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:28:11.794301 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-701dab4d-9c8e-4780-806f-ed2ece6a4f62 admin admin] REQUEST_METHOD: `GET` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:28:11.797654 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-701dab4d-9c8e-4780-806f-ed2ece6a4f62 admin admin] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:28:11.797654 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-701dab4d-9c8e-4780-806f-ed2ece6a4f62 admin admin] PATH_INFO: `/v3/domains` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:28:11.797654 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-701dab4d-9c8e-4780-806f-ed2ece6a4f62 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81408) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 13 04:28:11.797654 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-701dab4d-9c8e-4780-806f-ed2ece6a4f62 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:28:11.797654 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-701dab4d-9c8e-4780-806f-ed2ece6a4f62 admin admin] RBAC: Authorization granted {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:28:11.810874 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 1094/2217] 10.222.0.27 () {58 vars in 1153 bytes} [Thu Oct 13 04:28:11 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 33 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 13 04:28:11.815254 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ef87b37c-e9e8-46ea-9e67-473f12aae026 None None] Authenticating user token {{(pid=81409) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:28:11.825089 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ef87b37c-e9e8-46ea-9e67-473f12aae026 None None] Validating token access rules against request {{(pid=81409) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:28:11.829667 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fbe4a34d-969d-410c-8843-2739aa37c227 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8ba823f313144da3b683bacef8fd62e0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81409) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:28:11.830286 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fbe4a34d-969d-410c-8843-2739aa37c227 admin admin] REQUEST_METHOD: `POST` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:28:11.830564 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fbe4a34d-969d-410c-8843-2739aa37c227 admin admin] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:28:11.830833 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fbe4a34d-969d-410c-8843-2739aa37c227 admin admin] PATH_INFO: `/v3/projects` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:28:11.831459 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fbe4a34d-969d-410c-8843-2739aa37c227 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:28:11.833359 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fbe4a34d-969d-410c-8843-2739aa37c227 admin admin] RBAC: Authorization granted {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:28:11.863245 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 1124/2218] 10.222.0.27 () {60 vars in 1153 bytes} [Thu Oct 13 04:28:11 2022] POST /identity/v3/projects => generated 365 bytes in 48 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 13 04:28:11.867749 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-701dab4d-9c8e-4780-806f-ed2ece6a4f62 admin admin] Authenticating user token {{(pid=81408) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:28:11.877667 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-701dab4d-9c8e-4780-806f-ed2ece6a4f62 admin admin] Validating token access rules against request {{(pid=81408) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:28:11.934293 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cb8647b0-c160-470c-b009-56e1e41d5bfa admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8ba823f313144da3b683bacef8fd62e0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81408) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:28:11.934987 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb8647b0-c160-470c-b009-56e1e41d5bfa admin admin] REQUEST_METHOD: `POST` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:28:11.935291 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb8647b0-c160-470c-b009-56e1e41d5bfa admin admin] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:28:11.935624 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb8647b0-c160-470c-b009-56e1e41d5bfa admin admin] PATH_INFO: `/v3/users` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:28:11.936292 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cb8647b0-c160-470c-b009-56e1e41d5bfa admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:28:11.938264 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cb8647b0-c160-470c-b009-56e1e41d5bfa admin admin] RBAC: Authorization granted {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:28:11.960119 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 1095/2219] 10.222.0.27 () {60 vars in 1144 bytes} [Thu Oct 13 04:28:11 2022] POST /identity/v3/users => generated 283 bytes in 93 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 13 04:28:11.965841 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fbe4a34d-969d-410c-8843-2739aa37c227 admin admin] Authenticating user token {{(pid=81409) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:28:11.977301 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fbe4a34d-969d-410c-8843-2739aa37c227 admin admin] Validating token access rules against request {{(pid=81409) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:28:11.982171 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c8aa2357-33e9-425e-a858-cbd02a87b6c3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8ba823f313144da3b683bacef8fd62e0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81409) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:28:11.982805 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8aa2357-33e9-425e-a858-cbd02a87b6c3 admin admin] REQUEST_METHOD: `GET` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:28:11.983072 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8aa2357-33e9-425e-a858-cbd02a87b6c3 admin admin] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:28:11.983336 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8aa2357-33e9-425e-a858-cbd02a87b6c3 admin admin] PATH_INFO: `/v3/roles` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:28:11.983976 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c8aa2357-33e9-425e-a858-cbd02a87b6c3 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:28:11.985873 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c8aa2357-33e9-425e-a858-cbd02a87b6c3 admin admin] RBAC: Authorization granted {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:28:11.998475 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 1125/2220] 10.222.0.27 () {58 vars in 1122 bytes} [Thu Oct 13 04:28:11 2022] GET /identity/v3/roles => generated 1420 bytes in 33 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 13 04:28:12.002773 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cb8647b0-c160-470c-b009-56e1e41d5bfa admin admin] Authenticating user token {{(pid=81408) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:28:12.013538 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cb8647b0-c160-470c-b009-56e1e41d5bfa admin admin] Validating token access rules against request {{(pid=81408) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:28:12.018476 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e6fa833c-137b-4917-a8d8-e84ccdf5181a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8ba823f313144da3b683bacef8fd62e0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81408) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:28:12.019024 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6fa833c-137b-4917-a8d8-e84ccdf5181a admin admin] REQUEST_METHOD: `GET` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:28:12.019304 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6fa833c-137b-4917-a8d8-e84ccdf5181a admin admin] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:28:12.019580 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6fa833c-137b-4917-a8d8-e84ccdf5181a admin admin] PATH_INFO: `/v3/roles` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:28:12.020223 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e6fa833c-137b-4917-a8d8-e84ccdf5181a admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:28:12.021965 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e6fa833c-137b-4917-a8d8-e84ccdf5181a admin admin] RBAC: Authorization granted {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:28:12.033316 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 1096/2221] 10.222.0.27 () {58 vars in 1122 bytes} [Thu Oct 13 04:28:12 2022] GET /identity/v3/roles => generated 1420 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Oct 13 04:28:12.037530 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c8aa2357-33e9-425e-a858-cbd02a87b6c3 admin admin] Authenticating user token {{(pid=81409) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:28:12.049065 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c8aa2357-33e9-425e-a858-cbd02a87b6c3 admin admin] Validating token access rules against request {{(pid=81409) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:28:12.054067 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1bb65799-3824-4598-9457-13aa808cf76f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8ba823f313144da3b683bacef8fd62e0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81409) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:28:12.054806 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1bb65799-3824-4598-9457-13aa808cf76f admin admin] REQUEST_METHOD: `PUT` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:28:12.055081 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1bb65799-3824-4598-9457-13aa808cf76f admin admin] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:28:12.055358 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1bb65799-3824-4598-9457-13aa808cf76f admin admin] PATH_INFO: `/v3/projects/41783935852540cbaa3dfc839d4f2929/users/f978c4b824b2424c817c63eac684773e/roles/cc56c4ae13a7424ea771c15da4b153c1` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:28:12.072133 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1bb65799-3824-4598-9457-13aa808cf76f admin admin] RBAC: Authorizing `identity:create_grant(project_id=41783935852540cbaa3dfc839d4f2929, user_id=f978c4b824b2424c817c63eac684773e, role_id=cc56c4ae13a7424ea771c15da4b153c1)` {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:28:12.074844 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1bb65799-3824-4598-9457-13aa808cf76f admin admin] RBAC: Authorization granted {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:28:12.094528 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 1126/2222] 10.222.0.27 () {60 vars in 1483 bytes} [Thu Oct 13 04:28:12 2022] PUT /identity/v3/projects/41783935852540cbaa3dfc839d4f2929/users/f978c4b824b2424c817c63eac684773e/roles/cc56c4ae13a7424ea771c15da4b153c1 => generated 0 bytes in 57 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 13 04:28:12.107085 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a215c7bb-2d9e-4bbd-b350-1419a9a7c32b None None] REQUEST_METHOD: `POST` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:28:12.107367 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a215c7bb-2d9e-4bbd-b350-1419a9a7c32b None None] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:28:12.107684 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a215c7bb-2d9e-4bbd-b350-1419a9a7c32b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:28:12.137546 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.auth.core [None req-a215c7bb-2d9e-4bbd-b350-1419a9a7c32b None None] MFA Rules not processed for user `f978c4b824b2424c817c63eac684773e`. Rule list: `[]` (Enabled: `True`). {{(pid=81408) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 13 04:28:12.138798 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.fernet_utils [None req-a215c7bb-2d9e-4bbd-b350-1419a9a7c32b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81408) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:28:12.152313 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG dbcounter [-] [81408] Writing DB stats keystone:DELETE=4,keystone:SELECT=92,keystone:INSERT=4 {{(pid=81408) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:28:12.205374 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 1097/2223] 10.222.0.27 () {58 vars in 958 bytes} [Thu Oct 13 04:28:12 2022] POST /identity/v3/auth/tokens => generated 3242 bytes in 100 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 13 04:28:12.209816 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1bb65799-3824-4598-9457-13aa808cf76f admin admin] Authenticating user token {{(pid=81409) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:28:12.258831 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG dbcounter [-] [81409] Writing DB stats keystone:SELECT=90,keystone:DELETE=7,keystone:INSERT=3 {{(pid=81409) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:28:12.275437 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1bb65799-3824-4598-9457-13aa808cf76f admin admin] Validating token access rules against request {{(pid=81409) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:28:12.280054 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-544bd618-35bc-4fcf-a6cc-ab24d7fac30a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8ba823f313144da3b683bacef8fd62e0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81409) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:28:12.280652 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-544bd618-35bc-4fcf-a6cc-ab24d7fac30a admin admin] REQUEST_METHOD: `POST` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:28:12.280927 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-544bd618-35bc-4fcf-a6cc-ab24d7fac30a admin admin] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:28:12.281191 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-544bd618-35bc-4fcf-a6cc-ab24d7fac30a admin admin] PATH_INFO: `/v3/projects` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:28:12.281808 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-544bd618-35bc-4fcf-a6cc-ab24d7fac30a admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:28:12.283738 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-544bd618-35bc-4fcf-a6cc-ab24d7fac30a admin admin] RBAC: Authorization granted {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:28:12.316575 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 1127/2224] 10.222.0.27 () {60 vars in 1153 bytes} [Thu Oct 13 04:28:12 2022] POST /identity/v3/projects => generated 365 bytes in 104 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 13 04:28:12.317921 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a215c7bb-2d9e-4bbd-b350-1419a9a7c32b None None] Authenticating user token {{(pid=81408) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:28:12.328563 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a215c7bb-2d9e-4bbd-b350-1419a9a7c32b None None] Validating token access rules against request {{(pid=81408) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:28:12.387548 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0ce8b664-c69e-474c-9d80-0a752246052b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8ba823f313144da3b683bacef8fd62e0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81408) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:28:12.388153 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ce8b664-c69e-474c-9d80-0a752246052b admin admin] REQUEST_METHOD: `POST` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:28:12.388429 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ce8b664-c69e-474c-9d80-0a752246052b admin admin] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:28:12.388697 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ce8b664-c69e-474c-9d80-0a752246052b admin admin] PATH_INFO: `/v3/users` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:28:12.389340 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0ce8b664-c69e-474c-9d80-0a752246052b admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:28:12.392084 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0ce8b664-c69e-474c-9d80-0a752246052b admin admin] RBAC: Authorization granted {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:28:12.422613 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 1098/2225] 10.222.0.27 () {60 vars in 1144 bytes} [Thu Oct 13 04:28:12 2022] POST /identity/v3/users => generated 289 bytes in 105 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 13 04:28:12.426614 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-544bd618-35bc-4fcf-a6cc-ab24d7fac30a admin admin] Authenticating user token {{(pid=81409) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:28:12.437775 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-544bd618-35bc-4fcf-a6cc-ab24d7fac30a admin admin] Validating token access rules against request {{(pid=81409) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:28:12.442654 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-15fd54f8-0599-417c-ab23-40b94689bd8c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8ba823f313144da3b683bacef8fd62e0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81409) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:28:12.443256 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15fd54f8-0599-417c-ab23-40b94689bd8c admin admin] REQUEST_METHOD: `GET` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:28:12.443580 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15fd54f8-0599-417c-ab23-40b94689bd8c admin admin] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:28:12.443853 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15fd54f8-0599-417c-ab23-40b94689bd8c admin admin] PATH_INFO: `/v3/roles` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:28:12.444496 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-15fd54f8-0599-417c-ab23-40b94689bd8c admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:28:12.446697 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-15fd54f8-0599-417c-ab23-40b94689bd8c admin admin] RBAC: Authorization granted {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:28:12.460602 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 1128/2226] 10.222.0.27 () {58 vars in 1122 bytes} [Thu Oct 13 04:28:12 2022] GET /identity/v3/roles => generated 1420 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Oct 13 04:28:12.462281 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0ce8b664-c69e-474c-9d80-0a752246052b admin admin] Authenticating user token {{(pid=81408) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:28:12.473011 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0ce8b664-c69e-474c-9d80-0a752246052b admin admin] Validating token access rules against request {{(pid=81408) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:28:12.477935 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2a7c30c1-6517-40b3-aad2-09735fbfd981 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8ba823f313144da3b683bacef8fd62e0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81408) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:28:12.478778 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a7c30c1-6517-40b3-aad2-09735fbfd981 admin admin] REQUEST_METHOD: `PUT` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:28:12.488314 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a7c30c1-6517-40b3-aad2-09735fbfd981 admin admin] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:28:12.488314 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a7c30c1-6517-40b3-aad2-09735fbfd981 admin admin] PATH_INFO: `/v3/projects/9af87bc785d0492c81a2c8821fb979ff/users/f61f8418116f4fa09bc5690a4dd45c5b/roles/ef55fbd94e1040be83695754075a7ed5` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:28:12.496369 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2a7c30c1-6517-40b3-aad2-09735fbfd981 admin admin] RBAC: Authorizing `identity:create_grant(project_id=9af87bc785d0492c81a2c8821fb979ff, user_id=f61f8418116f4fa09bc5690a4dd45c5b, role_id=ef55fbd94e1040be83695754075a7ed5)` {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:28:12.498404 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2a7c30c1-6517-40b3-aad2-09735fbfd981 admin admin] RBAC: Authorization granted {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:28:12.510817 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 1099/2227] 10.222.0.27 () {60 vars in 1483 bytes} [Thu Oct 13 04:28:12 2022] PUT /identity/v3/projects/9af87bc785d0492c81a2c8821fb979ff/users/f61f8418116f4fa09bc5690a4dd45c5b/roles/ef55fbd94e1040be83695754075a7ed5 => generated 0 bytes in 49 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 13 04:28:12.514950 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-15fd54f8-0599-417c-ab23-40b94689bd8c admin admin] Authenticating user token {{(pid=81409) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:28:12.526050 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-15fd54f8-0599-417c-ab23-40b94689bd8c admin admin] Validating token access rules against request {{(pid=81409) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:28:12.581271 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-203954be-9e96-4b6c-8cd1-3000068ed285 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8ba823f313144da3b683bacef8fd62e0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81409) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:28:12.581869 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-203954be-9e96-4b6c-8cd1-3000068ed285 admin admin] REQUEST_METHOD: `GET` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:28:12.582145 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-203954be-9e96-4b6c-8cd1-3000068ed285 admin admin] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:28:12.582443 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-203954be-9e96-4b6c-8cd1-3000068ed285 admin admin] PATH_INFO: `/v3/roles` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:28:12.583074 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-203954be-9e96-4b6c-8cd1-3000068ed285 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:28:12.584669 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-203954be-9e96-4b6c-8cd1-3000068ed285 admin admin] RBAC: Authorization granted {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:28:12.594769 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 1129/2228] 10.222.0.27 () {58 vars in 1122 bytes} [Thu Oct 13 04:28:12 2022] GET /identity/v3/roles => generated 1420 bytes in 80 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 13 04:28:12.599003 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2a7c30c1-6517-40b3-aad2-09735fbfd981 admin admin] Authenticating user token {{(pid=81408) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:28:12.609366 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2a7c30c1-6517-40b3-aad2-09735fbfd981 admin admin] Validating token access rules against request {{(pid=81408) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:28:12.613891 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4d5460a4-97ce-461b-9dd5-9193598212ef admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8ba823f313144da3b683bacef8fd62e0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81408) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:28:12.614563 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d5460a4-97ce-461b-9dd5-9193598212ef admin admin] REQUEST_METHOD: `PUT` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:28:12.614854 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d5460a4-97ce-461b-9dd5-9193598212ef admin admin] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:28:12.615114 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d5460a4-97ce-461b-9dd5-9193598212ef admin admin] PATH_INFO: `/v3/projects/9af87bc785d0492c81a2c8821fb979ff/users/f61f8418116f4fa09bc5690a4dd45c5b/roles/ef55fbd94e1040be83695754075a7ed5` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:28:12.617351 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4d5460a4-97ce-461b-9dd5-9193598212ef admin admin] RBAC: Authorizing `identity:create_grant(project_id=9af87bc785d0492c81a2c8821fb979ff, user_id=f61f8418116f4fa09bc5690a4dd45c5b, role_id=ef55fbd94e1040be83695754075a7ed5)` {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:28:12.620127 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4d5460a4-97ce-461b-9dd5-9193598212ef admin admin] RBAC: Authorization granted {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:28:12.630019 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 1100/2229] 10.222.0.27 () {60 vars in 1483 bytes} [Thu Oct 13 04:28:12 2022] PUT /identity/v3/projects/9af87bc785d0492c81a2c8821fb979ff/users/f61f8418116f4fa09bc5690a4dd45c5b/roles/ef55fbd94e1040be83695754075a7ed5 => generated 0 bytes in 32 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 13 04:28:12.640674 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50e54d94-7a75-4117-8d55-a963304d284f None None] REQUEST_METHOD: `POST` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:28:12.640959 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50e54d94-7a75-4117-8d55-a963304d284f None None] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:28:12.641236 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50e54d94-7a75-4117-8d55-a963304d284f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:28:12.669135 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.auth.core [None req-50e54d94-7a75-4117-8d55-a963304d284f None None] MFA Rules not processed for user `f61f8418116f4fa09bc5690a4dd45c5b`. Rule list: `[]` (Enabled: `True`). {{(pid=81409) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 13 04:28:12.670378 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.fernet_utils [None req-50e54d94-7a75-4117-8d55-a963304d284f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81409) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:28:12.744619 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 1130/2230] 10.222.0.27 () {58 vars in 958 bytes} [Thu Oct 13 04:28:12 2022] POST /identity/v3/auth/tokens => generated 3309 bytes in 105 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 13 04:28:12.756733 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4d5460a4-97ce-461b-9dd5-9193598212ef admin admin] Authenticating user token {{(pid=81408) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:28:12.758872 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.fernet_utils [None req-4d5460a4-97ce-461b-9dd5-9193598212ef admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81408) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:28:12.820737 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4d5460a4-97ce-461b-9dd5-9193598212ef admin admin] Validating token access rules against request {{(pid=81408) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:28:12.824314 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b69586b6-9d6e-421a-a9af-ea654cedb424 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6141b8a6c3d54e179d6879bb019f4d12', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dcec0431fb6849499d6aec35a234daa6', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81408) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:28:12.824929 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b69586b6-9d6e-421a-a9af-ea654cedb424 service cinder] REQUEST_METHOD: `GET` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:28:12.825201 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b69586b6-9d6e-421a-a9af-ea654cedb424 service cinder] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:28:12.825468 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b69586b6-9d6e-421a-a9af-ea654cedb424 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:28:12.835306 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b69586b6-9d6e-421a-a9af-ea654cedb424 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:28:12.836753 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b69586b6-9d6e-421a-a9af-ea654cedb424 service cinder] RBAC: Authorization granted {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:28:12.847243 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 1101/2231] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Oct 13 04:28:12 2022] GET /identity/v3/auth/tokens => generated 3309 bytes in 91 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 13 04:28:12.928913 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-50e54d94-7a75-4117-8d55-a963304d284f None None] Authenticating user token {{(pid=81409) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:28:12.939296 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-50e54d94-7a75-4117-8d55-a963304d284f None None] Validating token access rules against request {{(pid=81409) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:28:12.942991 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-08246344-c06e-49b9-97aa-d4626a5b958c service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6141b8a6c3d54e179d6879bb019f4d12', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dcec0431fb6849499d6aec35a234daa6', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81409) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:28:12.943818 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08246344-c06e-49b9-97aa-d4626a5b958c service cinder] REQUEST_METHOD: `GET` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:28:12.944094 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08246344-c06e-49b9-97aa-d4626a5b958c service cinder] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:28:12.944364 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08246344-c06e-49b9-97aa-d4626a5b958c service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:28:12.987266 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG dbcounter [-] [81409] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81409) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:28:13.005438 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-08246344-c06e-49b9-97aa-d4626a5b958c service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:28:13.006967 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-08246344-c06e-49b9-97aa-d4626a5b958c service cinder] RBAC: Authorization granted {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:28:13.016320 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 1131/2232] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Oct 13 04:28:12 2022] GET /identity/v3/auth/tokens => generated 3242 bytes in 88 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 13 04:28:15.985536 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b69586b6-9d6e-421a-a9af-ea654cedb424 service cinder] Authenticating user token {{(pid=81408) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:28:15.999960 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG dbcounter [-] [81408] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=81408) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:28:16.063417 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b69586b6-9d6e-421a-a9af-ea654cedb424 service cinder] Validating token access rules against request {{(pid=81408) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:28:16.068276 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cd8e6ea3-aaba-4b58-831e-c55b64facc3b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8ba823f313144da3b683bacef8fd62e0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81408) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:28:16.068938 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd8e6ea3-aaba-4b58-831e-c55b64facc3b admin admin] REQUEST_METHOD: `DELETE` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:28:16.069281 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd8e6ea3-aaba-4b58-831e-c55b64facc3b admin admin] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:28:16.069623 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd8e6ea3-aaba-4b58-831e-c55b64facc3b admin admin] PATH_INFO: `/v3/users/f978c4b824b2424c817c63eac684773e` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:28:16.071711 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cd8e6ea3-aaba-4b58-831e-c55b64facc3b admin admin] RBAC: Authorizing `identity:delete_user(user_id=f978c4b824b2424c817c63eac684773e)` {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:28:16.073716 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cd8e6ea3-aaba-4b58-831e-c55b64facc3b admin admin] RBAC: Authorization granted {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:28:16.128075 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.notifications [None req-cd8e6ea3-aaba-4b58-831e-c55b64facc3b admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'f978c4b824b2424c817c63eac684773e'} {{(pid=81408) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 13 04:28:16.142029 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.notifications [None req-cd8e6ea3-aaba-4b58-831e-c55b64facc3b admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'f978c4b824b2424c817c63eac684773e'} {{(pid=81408) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 13 04:28:16.142789 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.notifications [None req-cd8e6ea3-aaba-4b58-831e-c55b64facc3b admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'f978c4b824b2424c817c63eac684773e'} {{(pid=81408) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 13 04:28:16.143391 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.notifications [None req-cd8e6ea3-aaba-4b58-831e-c55b64facc3b admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'f978c4b824b2424c817c63eac684773e'} {{(pid=81408) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 13 04:28:16.161573 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.notifications [None req-cd8e6ea3-aaba-4b58-831e-c55b64facc3b admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'f978c4b824b2424c817c63eac684773e'} {{(pid=81408) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 13 04:28:16.181791 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 1102/2233] 10.222.0.27 () {58 vars in 1224 bytes} [Thu Oct 13 04:28:15 2022] DELETE /identity/v3/users/f978c4b824b2424c817c63eac684773e => generated 0 bytes in 197 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 13 04:28:16.186501 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2b24567-cc3e-4adf-ac4d-b21ed933bb67 None None] REQUEST_METHOD: `POST` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:28:16.186987 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2b24567-cc3e-4adf-ac4d-b21ed933bb67 None None] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:28:16.187085 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2b24567-cc3e-4adf-ac4d-b21ed933bb67 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:28:16.207012 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.auth.core [None req-b2b24567-cc3e-4adf-ac4d-b21ed933bb67 None None] MFA Rules not processed for user `0e2d979563d6421995451a0e86bf3022`. Rule list: `[]` (Enabled: `True`). {{(pid=81409) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 13 04:28:16.208313 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.fernet_utils [None req-b2b24567-cc3e-4adf-ac4d-b21ed933bb67 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81409) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:28:16.269522 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 1132/2234] 10.222.0.27 () {58 vars in 958 bytes} [Thu Oct 13 04:28:16 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 84 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 13 04:28:16.280221 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cd8e6ea3-aaba-4b58-831e-c55b64facc3b admin admin] Authenticating user token {{(pid=81408) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:28:16.281644 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.fernet_utils [None req-cd8e6ea3-aaba-4b58-831e-c55b64facc3b admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81408) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:28:16.349061 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cd8e6ea3-aaba-4b58-831e-c55b64facc3b admin admin] Validating token access rules against request {{(pid=81408) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:28:16.353004 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-88a89363-49d0-46c6-b764-94a2d22a703b service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd8812c3974f84871bb2d761cf0016257', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dcec0431fb6849499d6aec35a234daa6', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81408) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:28:16.353683 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88a89363-49d0-46c6-b764-94a2d22a703b service neutron] REQUEST_METHOD: `GET` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:28:16.354032 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88a89363-49d0-46c6-b764-94a2d22a703b service neutron] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:28:16.354402 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88a89363-49d0-46c6-b764-94a2d22a703b service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:28:16.364205 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-88a89363-49d0-46c6-b764-94a2d22a703b service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:28:16.365673 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-88a89363-49d0-46c6-b764-94a2d22a703b service neutron] RBAC: Authorization granted {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:28:16.375714 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 1103/2235] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Oct 13 04:28:16 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 96 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Oct 13 04:28:16.666328 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b2b24567-cc3e-4adf-ac4d-b21ed933bb67 None None] Authenticating user token {{(pid=81409) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:28:16.667813 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.fernet_utils [None req-b2b24567-cc3e-4adf-ac4d-b21ed933bb67 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81409) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:28:16.698694 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b2b24567-cc3e-4adf-ac4d-b21ed933bb67 None None] Validating token access rules against request {{(pid=81409) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:28:16.705019 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3a216847-611b-46fa-9ebc-39cd73175ed7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8ba823f313144da3b683bacef8fd62e0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81409) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:28:16.705655 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a216847-611b-46fa-9ebc-39cd73175ed7 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:28:16.705925 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a216847-611b-46fa-9ebc-39cd73175ed7 admin admin] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:28:16.706263 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a216847-611b-46fa-9ebc-39cd73175ed7 admin admin] PATH_INFO: `/v3/projects/41783935852540cbaa3dfc839d4f2929` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:28:16.707332 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3a216847-611b-46fa-9ebc-39cd73175ed7 admin admin] RBAC: Authorizing `identity:delete_project(project_id=41783935852540cbaa3dfc839d4f2929)` {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:28:16.709291 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3a216847-611b-46fa-9ebc-39cd73175ed7 admin admin] RBAC: Authorization granted {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:28:16.768717 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.notifications [None req-3a216847-611b-46fa-9ebc-39cd73175ed7 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '41783935852540cbaa3dfc839d4f2929', 'request_id': 'req-3a216847-611b-46fa-9ebc-39cd73175ed7'} {{(pid=81409) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 13 04:28:16.776179 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.notifications [None req-3a216847-611b-46fa-9ebc-39cd73175ed7 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '41783935852540cbaa3dfc839d4f2929', 'request_id': 'req-3a216847-611b-46fa-9ebc-39cd73175ed7'} {{(pid=81409) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 13 04:28:16.782598 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.notifications [None req-3a216847-611b-46fa-9ebc-39cd73175ed7 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '41783935852540cbaa3dfc839d4f2929', 'request_id': 'req-3a216847-611b-46fa-9ebc-39cd73175ed7'} {{(pid=81409) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 13 04:28:16.793086 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.notifications [None req-3a216847-611b-46fa-9ebc-39cd73175ed7 admin admin] The token cache is being invalidate because project 41783935852540cbaa3dfc839d4f2929 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81409) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 13 04:28:16.793390 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.notifications [None req-3a216847-611b-46fa-9ebc-39cd73175ed7 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81409) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 13 04:28:16.794070 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.notifications [None req-3a216847-611b-46fa-9ebc-39cd73175ed7 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81409) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 13 04:28:16.795648 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 1133/2236] 10.222.0.27 () {58 vars in 1233 bytes} [Thu Oct 13 04:28:16 2022] DELETE /identity/v3/projects/41783935852540cbaa3dfc839d4f2929 => generated 0 bytes in 130 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 13 04:28:16.799812 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-88a89363-49d0-46c6-b764-94a2d22a703b service neutron] Authenticating user token {{(pid=81408) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:28:16.809950 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-88a89363-49d0-46c6-b764-94a2d22a703b service neutron] Validating token access rules against request {{(pid=81408) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:28:16.841342 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG dbcounter [-] [81408] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81408) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:28:16.872605 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-46c37b7f-bf41-4e83-a8fd-a4f753d3c9f2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8ba823f313144da3b683bacef8fd62e0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81408) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:28:16.873291 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46c37b7f-bf41-4e83-a8fd-a4f753d3c9f2 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:28:16.873574 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46c37b7f-bf41-4e83-a8fd-a4f753d3c9f2 admin admin] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:28:16.873842 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46c37b7f-bf41-4e83-a8fd-a4f753d3c9f2 admin admin] PATH_INFO: `/v3/users/f61f8418116f4fa09bc5690a4dd45c5b` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:28:16.875014 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-46c37b7f-bf41-4e83-a8fd-a4f753d3c9f2 admin admin] RBAC: Authorizing `identity:delete_user(user_id=f61f8418116f4fa09bc5690a4dd45c5b)` {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:28:16.877107 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-46c37b7f-bf41-4e83-a8fd-a4f753d3c9f2 admin admin] RBAC: Authorization granted {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:28:16.923865 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.notifications [None req-46c37b7f-bf41-4e83-a8fd-a4f753d3c9f2 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'f61f8418116f4fa09bc5690a4dd45c5b'} {{(pid=81408) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 13 04:28:16.931620 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.notifications [None req-46c37b7f-bf41-4e83-a8fd-a4f753d3c9f2 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'f61f8418116f4fa09bc5690a4dd45c5b'} {{(pid=81408) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 13 04:28:16.932583 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.notifications [None req-46c37b7f-bf41-4e83-a8fd-a4f753d3c9f2 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'f61f8418116f4fa09bc5690a4dd45c5b'} {{(pid=81408) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 13 04:28:16.933286 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.notifications [None req-46c37b7f-bf41-4e83-a8fd-a4f753d3c9f2 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'f61f8418116f4fa09bc5690a4dd45c5b'} {{(pid=81408) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 13 04:28:16.940354 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.notifications [None req-46c37b7f-bf41-4e83-a8fd-a4f753d3c9f2 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'f61f8418116f4fa09bc5690a4dd45c5b'} {{(pid=81408) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 13 04:28:16.962857 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 1104/2237] 10.222.0.27 () {58 vars in 1224 bytes} [Thu Oct 13 04:28:16 2022] DELETE /identity/v3/users/f61f8418116f4fa09bc5690a4dd45c5b => generated 0 bytes in 164 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 13 04:28:17.269974 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3a216847-611b-46fa-9ebc-39cd73175ed7 admin admin] Authenticating user token {{(pid=81409) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:28:17.271770 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.fernet_utils [None req-3a216847-611b-46fa-9ebc-39cd73175ed7 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81409) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:28:17.292978 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3a216847-611b-46fa-9ebc-39cd73175ed7 admin admin] Validating token access rules against request {{(pid=81409) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:28:17.354603 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-691549f3-76d7-418c-bb44-ee2eee176c76 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8ba823f313144da3b683bacef8fd62e0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81409) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:28:17.355239 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-691549f3-76d7-418c-bb44-ee2eee176c76 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:28:17.355511 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-691549f3-76d7-418c-bb44-ee2eee176c76 admin admin] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:28:17.355775 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-691549f3-76d7-418c-bb44-ee2eee176c76 admin admin] PATH_INFO: `/v3/projects/9af87bc785d0492c81a2c8821fb979ff` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:28:17.357140 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-691549f3-76d7-418c-bb44-ee2eee176c76 admin admin] RBAC: Authorizing `identity:delete_project(project_id=9af87bc785d0492c81a2c8821fb979ff)` {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:28:17.359099 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-691549f3-76d7-418c-bb44-ee2eee176c76 admin admin] RBAC: Authorization granted {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:28:17.365703 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG dbcounter [-] [81409] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=81409) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:28:17.412011 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.notifications [None req-691549f3-76d7-418c-bb44-ee2eee176c76 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '9af87bc785d0492c81a2c8821fb979ff', 'request_id': 'req-691549f3-76d7-418c-bb44-ee2eee176c76'} {{(pid=81409) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 13 04:28:17.422889 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.notifications [None req-691549f3-76d7-418c-bb44-ee2eee176c76 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '9af87bc785d0492c81a2c8821fb979ff', 'request_id': 'req-691549f3-76d7-418c-bb44-ee2eee176c76'} {{(pid=81409) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 13 04:28:17.430824 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.notifications [None req-691549f3-76d7-418c-bb44-ee2eee176c76 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '9af87bc785d0492c81a2c8821fb979ff', 'request_id': 'req-691549f3-76d7-418c-bb44-ee2eee176c76'} {{(pid=81409) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 13 04:28:17.443527 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.notifications [None req-691549f3-76d7-418c-bb44-ee2eee176c76 admin admin] The token cache is being invalidate because project 9af87bc785d0492c81a2c8821fb979ff was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81409) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 13 04:28:17.444045 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.notifications [None req-691549f3-76d7-418c-bb44-ee2eee176c76 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81409) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 13 04:28:17.444789 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.notifications [None req-691549f3-76d7-418c-bb44-ee2eee176c76 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81409) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 13 04:28:17.446658 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 1134/2238] 10.222.0.27 () {58 vars in 1233 bytes} [Thu Oct 13 04:28:17 2022] DELETE /identity/v3/projects/9af87bc785d0492c81a2c8821fb979ff => generated 0 bytes in 177 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 13 04:28:17.451673 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e42aaba7-533a-4b5f-9798-ff628eae3bd3 None None] REQUEST_METHOD: `POST` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:28:17.452160 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e42aaba7-533a-4b5f-9798-ff628eae3bd3 None None] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:28:17.452398 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e42aaba7-533a-4b5f-9798-ff628eae3bd3 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:28:17.472610 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.auth.core [None req-e42aaba7-533a-4b5f-9798-ff628eae3bd3 None None] MFA Rules not processed for user `0e2d979563d6421995451a0e86bf3022`. Rule list: `[]` (Enabled: `True`). {{(pid=81408) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 13 04:28:17.473928 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.fernet_utils [None req-e42aaba7-533a-4b5f-9798-ff628eae3bd3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81408) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:28:17.546700 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 1105/2239] 10.222.0.27 () {58 vars in 958 bytes} [Thu Oct 13 04:28:17 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 96 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 13 04:28:17.555359 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b25506cc-0d5d-45bd-8229-4a6db4ea533d None None] REQUEST_METHOD: `POST` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:28:17.555538 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b25506cc-0d5d-45bd-8229-4a6db4ea533d None None] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:28:17.555935 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b25506cc-0d5d-45bd-8229-4a6db4ea533d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:28:17.575632 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.auth.core [None req-b25506cc-0d5d-45bd-8229-4a6db4ea533d None None] MFA Rules not processed for user `0e2d979563d6421995451a0e86bf3022`. Rule list: `[]` (Enabled: `True`). {{(pid=81409) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 13 04:28:17.576826 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.fernet_utils [None req-b25506cc-0d5d-45bd-8229-4a6db4ea533d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81409) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:28:17.585166 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 1135/2240] 10.222.0.27 () {58 vars in 958 bytes} [Thu Oct 13 04:28:17 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 32 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 13 04:28:17.590415 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e42aaba7-533a-4b5f-9798-ff628eae3bd3 None None] Authenticating user token {{(pid=81408) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:28:17.600803 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e42aaba7-533a-4b5f-9798-ff628eae3bd3 None None] Validating token access rules against request {{(pid=81408) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:28:17.605714 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-11bcaa3b-9fef-464c-a996-1a35b8989cba admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8ba823f313144da3b683bacef8fd62e0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81408) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:28:17.606715 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11bcaa3b-9fef-464c-a996-1a35b8989cba admin admin] REQUEST_METHOD: `GET` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:28:17.606989 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11bcaa3b-9fef-464c-a996-1a35b8989cba admin admin] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:28:17.607256 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11bcaa3b-9fef-464c-a996-1a35b8989cba admin admin] PATH_INFO: `/v3/domains` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:28:17.607902 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-11bcaa3b-9fef-464c-a996-1a35b8989cba admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81408) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 13 04:28:17.608205 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-11bcaa3b-9fef-464c-a996-1a35b8989cba admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:28:17.609809 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-11bcaa3b-9fef-464c-a996-1a35b8989cba admin admin] RBAC: Authorization granted {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:28:17.622920 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 1106/2241] 10.222.0.27 () {58 vars in 1153 bytes} [Thu Oct 13 04:28:17 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 33 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 13 04:28:17.627394 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b25506cc-0d5d-45bd-8229-4a6db4ea533d None None] Authenticating user token {{(pid=81409) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:28:17.637228 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b25506cc-0d5d-45bd-8229-4a6db4ea533d None None] Validating token access rules against request {{(pid=81409) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:28:17.641834 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be73084f-045d-4869-9446-f30a911b3e2e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8ba823f313144da3b683bacef8fd62e0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81409) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:28:17.642456 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be73084f-045d-4869-9446-f30a911b3e2e admin admin] REQUEST_METHOD: `POST` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:28:17.642729 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be73084f-045d-4869-9446-f30a911b3e2e admin admin] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:28:17.642994 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be73084f-045d-4869-9446-f30a911b3e2e admin admin] PATH_INFO: `/v3/projects` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:28:17.643621 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-be73084f-045d-4869-9446-f30a911b3e2e admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:28:17.645655 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-be73084f-045d-4869-9446-f30a911b3e2e admin admin] RBAC: Authorization granted {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:28:17.682294 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 1136/2242] 10.222.0.27 () {60 vars in 1153 bytes} [Thu Oct 13 04:28:17 2022] POST /identity/v3/projects => generated 365 bytes in 51 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 13 04:28:17.683028 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-11bcaa3b-9fef-464c-a996-1a35b8989cba admin admin] Authenticating user token {{(pid=81408) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:28:17.692785 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-11bcaa3b-9fef-464c-a996-1a35b8989cba admin admin] Validating token access rules against request {{(pid=81408) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:28:17.733380 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG dbcounter [-] [81408] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81408) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:28:17.756299 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3664bdd7-98b9-4bdf-b8b7-e6da97e62f19 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8ba823f313144da3b683bacef8fd62e0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81408) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:28:17.756930 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3664bdd7-98b9-4bdf-b8b7-e6da97e62f19 admin admin] REQUEST_METHOD: `POST` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:28:17.757221 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3664bdd7-98b9-4bdf-b8b7-e6da97e62f19 admin admin] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:28:17.757575 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3664bdd7-98b9-4bdf-b8b7-e6da97e62f19 admin admin] PATH_INFO: `/v3/users` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:28:17.758247 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3664bdd7-98b9-4bdf-b8b7-e6da97e62f19 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:28:17.760398 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3664bdd7-98b9-4bdf-b8b7-e6da97e62f19 admin admin] RBAC: Authorization granted {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:28:17.789639 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 1107/2243] 10.222.0.27 () {60 vars in 1144 bytes} [Thu Oct 13 04:28:17 2022] POST /identity/v3/users => generated 283 bytes in 104 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 13 04:28:17.791397 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be73084f-045d-4869-9446-f30a911b3e2e admin admin] Authenticating user token {{(pid=81409) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:28:17.802454 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be73084f-045d-4869-9446-f30a911b3e2e admin admin] Validating token access rules against request {{(pid=81409) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:28:17.807571 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8af92eaf-f420-4a74-8ead-ef4108bbc24e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8ba823f313144da3b683bacef8fd62e0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81409) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:28:17.808348 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8af92eaf-f420-4a74-8ead-ef4108bbc24e admin admin] REQUEST_METHOD: `GET` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:28:17.808627 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8af92eaf-f420-4a74-8ead-ef4108bbc24e admin admin] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:28:17.808899 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8af92eaf-f420-4a74-8ead-ef4108bbc24e admin admin] PATH_INFO: `/v3/roles` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:28:17.809548 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8af92eaf-f420-4a74-8ead-ef4108bbc24e admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:28:17.811210 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8af92eaf-f420-4a74-8ead-ef4108bbc24e admin admin] RBAC: Authorization granted {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:28:17.822872 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 1137/2244] 10.222.0.27 () {58 vars in 1122 bytes} [Thu Oct 13 04:28:17 2022] GET /identity/v3/roles => generated 1420 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 13 04:28:17.826769 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3664bdd7-98b9-4bdf-b8b7-e6da97e62f19 admin admin] Authenticating user token {{(pid=81408) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:28:17.837643 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3664bdd7-98b9-4bdf-b8b7-e6da97e62f19 admin admin] Validating token access rules against request {{(pid=81408) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:28:17.842424 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3fc8371c-3211-4338-ae12-6116114d14df admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8ba823f313144da3b683bacef8fd62e0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81408) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:28:17.843030 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3fc8371c-3211-4338-ae12-6116114d14df admin admin] REQUEST_METHOD: `GET` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:28:17.843308 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3fc8371c-3211-4338-ae12-6116114d14df admin admin] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:28:17.843580 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3fc8371c-3211-4338-ae12-6116114d14df admin admin] PATH_INFO: `/v3/roles` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:28:17.844221 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3fc8371c-3211-4338-ae12-6116114d14df admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:28:17.845824 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3fc8371c-3211-4338-ae12-6116114d14df admin admin] RBAC: Authorization granted {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:28:17.858777 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 1108/2245] 10.222.0.27 () {58 vars in 1122 bytes} [Thu Oct 13 04:28:17 2022] GET /identity/v3/roles => generated 1420 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 13 04:28:17.862970 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8af92eaf-f420-4a74-8ead-ef4108bbc24e admin admin] Authenticating user token {{(pid=81409) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:28:17.873492 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8af92eaf-f420-4a74-8ead-ef4108bbc24e admin admin] Validating token access rules against request {{(pid=81409) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:28:17.878318 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5568558b-08b5-4bc1-9b60-cb7bacba8d81 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8ba823f313144da3b683bacef8fd62e0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81409) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:28:17.878969 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5568558b-08b5-4bc1-9b60-cb7bacba8d81 admin admin] REQUEST_METHOD: `PUT` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:28:17.879239 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5568558b-08b5-4bc1-9b60-cb7bacba8d81 admin admin] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:28:17.879518 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5568558b-08b5-4bc1-9b60-cb7bacba8d81 admin admin] PATH_INFO: `/v3/projects/d6745f9d885b4c87a1d5971e04b33e5b/users/d8ab1ceba0304f07857da1d20ef78e18/roles/cc56c4ae13a7424ea771c15da4b153c1` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:28:17.895387 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5568558b-08b5-4bc1-9b60-cb7bacba8d81 admin admin] RBAC: Authorizing `identity:create_grant(project_id=d6745f9d885b4c87a1d5971e04b33e5b, user_id=d8ab1ceba0304f07857da1d20ef78e18, role_id=cc56c4ae13a7424ea771c15da4b153c1)` {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:28:17.898462 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5568558b-08b5-4bc1-9b60-cb7bacba8d81 admin admin] RBAC: Authorization granted {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:28:17.912883 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 1138/2246] 10.222.0.27 () {60 vars in 1483 bytes} [Thu Oct 13 04:28:17 2022] PUT /identity/v3/projects/d6745f9d885b4c87a1d5971e04b33e5b/users/d8ab1ceba0304f07857da1d20ef78e18/roles/cc56c4ae13a7424ea771c15da4b153c1 => generated 0 bytes in 50 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 13 04:28:17.925666 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-828bfee2-2484-4032-b7ef-a26f2cfa1fdf None None] REQUEST_METHOD: `POST` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:28:17.925952 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-828bfee2-2484-4032-b7ef-a26f2cfa1fdf None None] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:28:17.926256 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-828bfee2-2484-4032-b7ef-a26f2cfa1fdf None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:28:17.956228 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.auth.core [None req-828bfee2-2484-4032-b7ef-a26f2cfa1fdf None None] MFA Rules not processed for user `d8ab1ceba0304f07857da1d20ef78e18`. Rule list: `[]` (Enabled: `True`). {{(pid=81408) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 13 04:28:17.957647 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.fernet_utils [None req-828bfee2-2484-4032-b7ef-a26f2cfa1fdf None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81408) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:28:18.025213 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 1109/2247] 10.222.0.27 () {58 vars in 958 bytes} [Thu Oct 13 04:28:17 2022] POST /identity/v3/auth/tokens => generated 3242 bytes in 101 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 13 04:28:18.029864 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5568558b-08b5-4bc1-9b60-cb7bacba8d81 admin admin] Authenticating user token {{(pid=81409) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:28:18.092742 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5568558b-08b5-4bc1-9b60-cb7bacba8d81 admin admin] Validating token access rules against request {{(pid=81409) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:28:18.098153 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-085c152a-6f2c-4043-909a-aee25a21c191 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8ba823f313144da3b683bacef8fd62e0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81409) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:28:18.098776 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-085c152a-6f2c-4043-909a-aee25a21c191 admin admin] REQUEST_METHOD: `POST` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:28:18.099072 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-085c152a-6f2c-4043-909a-aee25a21c191 admin admin] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:28:18.099340 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-085c152a-6f2c-4043-909a-aee25a21c191 admin admin] PATH_INFO: `/v3/projects` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:28:18.100035 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-085c152a-6f2c-4043-909a-aee25a21c191 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:28:18.102448 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-085c152a-6f2c-4043-909a-aee25a21c191 admin admin] RBAC: Authorization granted {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:28:18.129904 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 1139/2248] 10.222.0.27 () {60 vars in 1153 bytes} [Thu Oct 13 04:28:18 2022] POST /identity/v3/projects => generated 365 bytes in 100 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 13 04:28:18.135300 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-828bfee2-2484-4032-b7ef-a26f2cfa1fdf None None] Authenticating user token {{(pid=81408) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:28:18.147860 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-828bfee2-2484-4032-b7ef-a26f2cfa1fdf None None] Validating token access rules against request {{(pid=81408) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:28:18.208146 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c1647daf-32a4-4dba-859a-1da1e6454169 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8ba823f313144da3b683bacef8fd62e0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81408) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:28:18.208758 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1647daf-32a4-4dba-859a-1da1e6454169 admin admin] REQUEST_METHOD: `POST` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:28:18.209036 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1647daf-32a4-4dba-859a-1da1e6454169 admin admin] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:28:18.209304 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1647daf-32a4-4dba-859a-1da1e6454169 admin admin] PATH_INFO: `/v3/users` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:28:18.209942 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c1647daf-32a4-4dba-859a-1da1e6454169 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:28:18.212194 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c1647daf-32a4-4dba-859a-1da1e6454169 admin admin] RBAC: Authorization granted {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:28:18.237412 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 1110/2249] 10.222.0.27 () {60 vars in 1144 bytes} [Thu Oct 13 04:28:18 2022] POST /identity/v3/users => generated 289 bytes in 100 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 13 04:28:18.238756 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-085c152a-6f2c-4043-909a-aee25a21c191 admin admin] Authenticating user token {{(pid=81409) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:28:18.250877 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-085c152a-6f2c-4043-909a-aee25a21c191 admin admin] Validating token access rules against request {{(pid=81409) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:28:18.255475 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6af02d04-0bc4-4ba7-b38f-f363dac5248d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8ba823f313144da3b683bacef8fd62e0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81409) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:28:18.256068 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6af02d04-0bc4-4ba7-b38f-f363dac5248d admin admin] REQUEST_METHOD: `GET` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:28:18.256341 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6af02d04-0bc4-4ba7-b38f-f363dac5248d admin admin] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:28:18.256606 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6af02d04-0bc4-4ba7-b38f-f363dac5248d admin admin] PATH_INFO: `/v3/roles` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:28:18.257234 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6af02d04-0bc4-4ba7-b38f-f363dac5248d admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:28:18.259799 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6af02d04-0bc4-4ba7-b38f-f363dac5248d admin admin] RBAC: Authorization granted {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:28:18.270825 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 1140/2250] 10.222.0.27 () {58 vars in 1122 bytes} [Thu Oct 13 04:28:18 2022] GET /identity/v3/roles => generated 1420 bytes in 33 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 13 04:28:18.275317 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c1647daf-32a4-4dba-859a-1da1e6454169 admin admin] Authenticating user token {{(pid=81408) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:28:18.286100 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c1647daf-32a4-4dba-859a-1da1e6454169 admin admin] Validating token access rules against request {{(pid=81408) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:28:18.290855 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ce172465-0888-4b2c-a895-4123ca6eff0e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8ba823f313144da3b683bacef8fd62e0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81408) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:28:18.291498 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce172465-0888-4b2c-a895-4123ca6eff0e admin admin] REQUEST_METHOD: `PUT` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:28:18.291769 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce172465-0888-4b2c-a895-4123ca6eff0e admin admin] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:28:18.292040 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce172465-0888-4b2c-a895-4123ca6eff0e admin admin] PATH_INFO: `/v3/projects/a5131be56034418f997a2e501ec3e8c1/users/aa1a855b44b246bebffea3f215e4b186/roles/ef55fbd94e1040be83695754075a7ed5` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:28:18.307758 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ce172465-0888-4b2c-a895-4123ca6eff0e admin admin] RBAC: Authorizing `identity:create_grant(project_id=a5131be56034418f997a2e501ec3e8c1, user_id=aa1a855b44b246bebffea3f215e4b186, role_id=ef55fbd94e1040be83695754075a7ed5)` {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:28:18.311057 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ce172465-0888-4b2c-a895-4123ca6eff0e admin admin] RBAC: Authorization granted {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:28:18.328727 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 1111/2251] 10.222.0.27 () {60 vars in 1483 bytes} [Thu Oct 13 04:28:18 2022] PUT /identity/v3/projects/a5131be56034418f997a2e501ec3e8c1/users/aa1a855b44b246bebffea3f215e4b186/roles/ef55fbd94e1040be83695754075a7ed5 => generated 0 bytes in 54 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 13 04:28:18.332844 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6af02d04-0bc4-4ba7-b38f-f363dac5248d admin admin] Authenticating user token {{(pid=81409) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:28:18.339996 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG dbcounter [-] [81409] Writing DB stats keystone:SELECT=89,keystone:DELETE=7,keystone:INSERT=4 {{(pid=81409) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:28:18.344211 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6af02d04-0bc4-4ba7-b38f-f363dac5248d admin admin] Validating token access rules against request {{(pid=81409) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:28:18.402153 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-557d2a88-2290-4424-9018-ae3566b4da81 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8ba823f313144da3b683bacef8fd62e0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81409) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:28:18.402722 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-557d2a88-2290-4424-9018-ae3566b4da81 admin admin] REQUEST_METHOD: `GET` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:28:18.402995 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-557d2a88-2290-4424-9018-ae3566b4da81 admin admin] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:28:18.403258 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-557d2a88-2290-4424-9018-ae3566b4da81 admin admin] PATH_INFO: `/v3/roles` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:28:18.403890 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-557d2a88-2290-4424-9018-ae3566b4da81 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:28:18.405478 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-557d2a88-2290-4424-9018-ae3566b4da81 admin admin] RBAC: Authorization granted {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:28:18.416378 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 1141/2252] 10.222.0.27 () {58 vars in 1122 bytes} [Thu Oct 13 04:28:18 2022] GET /identity/v3/roles => generated 1420 bytes in 84 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 13 04:28:18.420525 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ce172465-0888-4b2c-a895-4123ca6eff0e admin admin] Authenticating user token {{(pid=81408) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:28:18.430720 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ce172465-0888-4b2c-a895-4123ca6eff0e admin admin] Validating token access rules against request {{(pid=81408) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:28:18.435382 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-93b738a4-4104-4b45-b9e1-79ca485c0ed6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8ba823f313144da3b683bacef8fd62e0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81408) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:28:18.436020 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93b738a4-4104-4b45-b9e1-79ca485c0ed6 admin admin] REQUEST_METHOD: `PUT` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:28:18.436293 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93b738a4-4104-4b45-b9e1-79ca485c0ed6 admin admin] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:28:18.436558 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93b738a4-4104-4b45-b9e1-79ca485c0ed6 admin admin] PATH_INFO: `/v3/projects/a5131be56034418f997a2e501ec3e8c1/users/aa1a855b44b246bebffea3f215e4b186/roles/ef55fbd94e1040be83695754075a7ed5` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:28:18.438500 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-93b738a4-4104-4b45-b9e1-79ca485c0ed6 admin admin] RBAC: Authorizing `identity:create_grant(project_id=a5131be56034418f997a2e501ec3e8c1, user_id=aa1a855b44b246bebffea3f215e4b186, role_id=ef55fbd94e1040be83695754075a7ed5)` {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:28:18.441502 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-93b738a4-4104-4b45-b9e1-79ca485c0ed6 admin admin] RBAC: Authorization granted {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:28:18.448753 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG dbcounter [-] [81408] Writing DB stats keystone:SELECT=92,keystone:INSERT=8 {{(pid=81408) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:28:18.453983 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 1112/2253] 10.222.0.27 () {60 vars in 1483 bytes} [Thu Oct 13 04:28:18 2022] PUT /identity/v3/projects/a5131be56034418f997a2e501ec3e8c1/users/aa1a855b44b246bebffea3f215e4b186/roles/ef55fbd94e1040be83695754075a7ed5 => generated 0 bytes in 34 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 13 04:28:18.466100 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f8692b0-46f7-47be-9596-d7db604189e7 None None] REQUEST_METHOD: `POST` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:28:18.466420 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f8692b0-46f7-47be-9596-d7db604189e7 None None] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:28:18.466810 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f8692b0-46f7-47be-9596-d7db604189e7 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:28:18.495037 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.auth.core [None req-3f8692b0-46f7-47be-9596-d7db604189e7 None None] MFA Rules not processed for user `aa1a855b44b246bebffea3f215e4b186`. Rule list: `[]` (Enabled: `True`). {{(pid=81409) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 13 04:28:18.496243 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.fernet_utils [None req-3f8692b0-46f7-47be-9596-d7db604189e7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81409) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:28:18.564009 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 1142/2254] 10.222.0.27 () {58 vars in 958 bytes} [Thu Oct 13 04:28:18 2022] POST /identity/v3/auth/tokens => generated 3309 bytes in 99 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 13 04:28:18.575136 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-93b738a4-4104-4b45-b9e1-79ca485c0ed6 admin admin] Authenticating user token {{(pid=81408) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:28:18.576811 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.fernet_utils [None req-93b738a4-4104-4b45-b9e1-79ca485c0ed6 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81408) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:28:18.635116 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-93b738a4-4104-4b45-b9e1-79ca485c0ed6 admin admin] Validating token access rules against request {{(pid=81408) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:28:18.638742 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-31c75bd1-a5ac-4188-a3cd-6d695b86fc84 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6141b8a6c3d54e179d6879bb019f4d12', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dcec0431fb6849499d6aec35a234daa6', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81408) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:28:18.639336 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31c75bd1-a5ac-4188-a3cd-6d695b86fc84 service cinder] REQUEST_METHOD: `GET` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:28:18.639611 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31c75bd1-a5ac-4188-a3cd-6d695b86fc84 service cinder] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:28:18.639877 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31c75bd1-a5ac-4188-a3cd-6d695b86fc84 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:28:18.648857 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-31c75bd1-a5ac-4188-a3cd-6d695b86fc84 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:28:18.650311 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-31c75bd1-a5ac-4188-a3cd-6d695b86fc84 service cinder] RBAC: Authorization granted {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:28:18.659898 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 1113/2255] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Oct 13 04:28:18 2022] GET /identity/v3/auth/tokens => generated 3309 bytes in 85 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 13 04:28:19.080009 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3f8692b0-46f7-47be-9596-d7db604189e7 None None] Authenticating user token {{(pid=81409) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:28:19.081456 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.fernet_utils [None req-3f8692b0-46f7-47be-9596-d7db604189e7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81409) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:28:19.149632 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3f8692b0-46f7-47be-9596-d7db604189e7 None None] Validating token access rules against request {{(pid=81409) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:28:19.153234 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f315f220-d034-48d5-ada0-0628063a29c8 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd8812c3974f84871bb2d761cf0016257', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dcec0431fb6849499d6aec35a234daa6', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81409) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:28:19.153860 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f315f220-d034-48d5-ada0-0628063a29c8 service neutron] REQUEST_METHOD: `GET` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:28:19.154136 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f315f220-d034-48d5-ada0-0628063a29c8 service neutron] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:28:19.154434 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f315f220-d034-48d5-ada0-0628063a29c8 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:28:19.156113 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.fernet_utils [None req-f315f220-d034-48d5-ada0-0628063a29c8 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81409) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:28:19.201708 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG dbcounter [-] [81409] Writing DB stats keystone:SELECT=100 {{(pid=81409) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:28:19.225333 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f315f220-d034-48d5-ada0-0628063a29c8 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:28:19.226961 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f315f220-d034-48d5-ada0-0628063a29c8 service neutron] RBAC: Authorization granted {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:28:19.237340 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 1143/2256] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Oct 13 04:28:19 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 158 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 13 04:28:20.088402 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-31c75bd1-a5ac-4188-a3cd-6d695b86fc84 service cinder] Authenticating user token {{(pid=81408) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:28:20.100244 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-31c75bd1-a5ac-4188-a3cd-6d695b86fc84 service cinder] Validating token access rules against request {{(pid=81408) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:28:20.105671 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c8c110d7-514a-431c-bc39-034960feb826 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8ba823f313144da3b683bacef8fd62e0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81408) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:28:20.106755 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8c110d7-514a-431c-bc39-034960feb826 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:28:20.107124 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8c110d7-514a-431c-bc39-034960feb826 admin admin] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:28:20.107564 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8c110d7-514a-431c-bc39-034960feb826 admin admin] PATH_INFO: `/v3/users/d8ab1ceba0304f07857da1d20ef78e18` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:28:20.108952 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c8c110d7-514a-431c-bc39-034960feb826 admin admin] RBAC: Authorizing `identity:delete_user(user_id=d8ab1ceba0304f07857da1d20ef78e18)` {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:28:20.111365 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c8c110d7-514a-431c-bc39-034960feb826 admin admin] RBAC: Authorization granted {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:28:20.177229 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.notifications [None req-c8c110d7-514a-431c-bc39-034960feb826 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'd8ab1ceba0304f07857da1d20ef78e18'} {{(pid=81408) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 13 04:28:20.184318 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.notifications [None req-c8c110d7-514a-431c-bc39-034960feb826 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'd8ab1ceba0304f07857da1d20ef78e18'} {{(pid=81408) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 13 04:28:20.184912 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.notifications [None req-c8c110d7-514a-431c-bc39-034960feb826 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'd8ab1ceba0304f07857da1d20ef78e18'} {{(pid=81408) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 13 04:28:20.185514 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.notifications [None req-c8c110d7-514a-431c-bc39-034960feb826 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'd8ab1ceba0304f07857da1d20ef78e18'} {{(pid=81408) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 13 04:28:20.193656 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.notifications [None req-c8c110d7-514a-431c-bc39-034960feb826 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'd8ab1ceba0304f07857da1d20ef78e18'} {{(pid=81408) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 13 04:28:20.214162 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 1114/2257] 10.222.0.27 () {58 vars in 1224 bytes} [Thu Oct 13 04:28:20 2022] DELETE /identity/v3/users/d8ab1ceba0304f07857da1d20ef78e18 => generated 0 bytes in 126 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 13 04:28:20.218825 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cddfdc7d-5ca5-47e5-af34-08c4659ebed5 None None] REQUEST_METHOD: `POST` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:28:20.219327 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cddfdc7d-5ca5-47e5-af34-08c4659ebed5 None None] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:28:20.219527 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cddfdc7d-5ca5-47e5-af34-08c4659ebed5 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:28:20.239083 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.auth.core [None req-cddfdc7d-5ca5-47e5-af34-08c4659ebed5 None None] MFA Rules not processed for user `0e2d979563d6421995451a0e86bf3022`. Rule list: `[]` (Enabled: `True`). {{(pid=81409) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 13 04:28:20.240294 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.fernet_utils [None req-cddfdc7d-5ca5-47e5-af34-08c4659ebed5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81409) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:28:20.301793 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 1144/2258] 10.222.0.27 () {58 vars in 958 bytes} [Thu Oct 13 04:28:20 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 84 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 13 04:28:20.315676 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c8c110d7-514a-431c-bc39-034960feb826 admin admin] Authenticating user token {{(pid=81408) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:28:20.317224 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.fernet_utils [None req-c8c110d7-514a-431c-bc39-034960feb826 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81408) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:28:20.383258 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c8c110d7-514a-431c-bc39-034960feb826 admin admin] Validating token access rules against request {{(pid=81408) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:28:20.386943 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cda15a71-cf10-42b0-886e-cf74592d9af1 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd8812c3974f84871bb2d761cf0016257', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dcec0431fb6849499d6aec35a234daa6', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81408) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:28:20.387548 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cda15a71-cf10-42b0-886e-cf74592d9af1 service neutron] REQUEST_METHOD: `GET` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:28:20.387825 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cda15a71-cf10-42b0-886e-cf74592d9af1 service neutron] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:28:20.388093 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cda15a71-cf10-42b0-886e-cf74592d9af1 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:28:20.397564 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cda15a71-cf10-42b0-886e-cf74592d9af1 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:28:20.399162 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cda15a71-cf10-42b0-886e-cf74592d9af1 service neutron] RBAC: Authorization granted {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:28:20.410939 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 1115/2259] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Oct 13 04:28:20 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 96 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 13 04:28:20.926431 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cddfdc7d-5ca5-47e5-af34-08c4659ebed5 None None] Authenticating user token {{(pid=81409) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:28:20.930238 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.fernet_utils [None req-cddfdc7d-5ca5-47e5-af34-08c4659ebed5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81409) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:28:20.959855 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cddfdc7d-5ca5-47e5-af34-08c4659ebed5 None None] Validating token access rules against request {{(pid=81409) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:28:20.970086 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-62704541-6180-4eff-a9d7-6dbc4d3cc45f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8ba823f313144da3b683bacef8fd62e0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81409) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:28:20.971022 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62704541-6180-4eff-a9d7-6dbc4d3cc45f admin admin] REQUEST_METHOD: `DELETE` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:28:20.971291 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62704541-6180-4eff-a9d7-6dbc4d3cc45f admin admin] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:28:20.971878 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62704541-6180-4eff-a9d7-6dbc4d3cc45f admin admin] PATH_INFO: `/v3/projects/d6745f9d885b4c87a1d5971e04b33e5b` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:28:20.973429 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-62704541-6180-4eff-a9d7-6dbc4d3cc45f admin admin] RBAC: Authorizing `identity:delete_project(project_id=d6745f9d885b4c87a1d5971e04b33e5b)` {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:28:20.976480 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-62704541-6180-4eff-a9d7-6dbc4d3cc45f admin admin] RBAC: Authorization granted {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:28:21.037427 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.notifications [None req-62704541-6180-4eff-a9d7-6dbc4d3cc45f admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'd6745f9d885b4c87a1d5971e04b33e5b', 'request_id': 'req-62704541-6180-4eff-a9d7-6dbc4d3cc45f'} {{(pid=81409) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 13 04:28:21.047390 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.notifications [None req-62704541-6180-4eff-a9d7-6dbc4d3cc45f admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'd6745f9d885b4c87a1d5971e04b33e5b', 'request_id': 'req-62704541-6180-4eff-a9d7-6dbc4d3cc45f'} {{(pid=81409) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 13 04:28:21.054569 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.notifications [None req-62704541-6180-4eff-a9d7-6dbc4d3cc45f admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'd6745f9d885b4c87a1d5971e04b33e5b', 'request_id': 'req-62704541-6180-4eff-a9d7-6dbc4d3cc45f'} {{(pid=81409) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 13 04:28:21.066018 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.notifications [None req-62704541-6180-4eff-a9d7-6dbc4d3cc45f admin admin] The token cache is being invalidate because project d6745f9d885b4c87a1d5971e04b33e5b was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81409) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 13 04:28:21.066440 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.notifications [None req-62704541-6180-4eff-a9d7-6dbc4d3cc45f admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81409) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 13 04:28:21.067093 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.notifications [None req-62704541-6180-4eff-a9d7-6dbc4d3cc45f admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81409) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 13 04:28:21.068933 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 1145/2260] 10.222.0.27 () {58 vars in 1233 bytes} [Thu Oct 13 04:28:20 2022] DELETE /identity/v3/projects/d6745f9d885b4c87a1d5971e04b33e5b => generated 0 bytes in 143 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 13 04:28:21.073804 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cda15a71-cf10-42b0-886e-cf74592d9af1 service neutron] Authenticating user token {{(pid=81408) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:28:21.085117 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cda15a71-cf10-42b0-886e-cf74592d9af1 service neutron] Validating token access rules against request {{(pid=81408) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:28:21.111990 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG dbcounter [-] [81408] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81408) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:28:21.244827 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-706152c4-9cda-4311-9297-31a6631cf8e1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8ba823f313144da3b683bacef8fd62e0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81408) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:28:21.245750 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-706152c4-9cda-4311-9297-31a6631cf8e1 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:28:21.246185 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-706152c4-9cda-4311-9297-31a6631cf8e1 admin admin] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:28:21.247331 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-706152c4-9cda-4311-9297-31a6631cf8e1 admin admin] PATH_INFO: `/v3/users/aa1a855b44b246bebffea3f215e4b186` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:28:21.249648 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-706152c4-9cda-4311-9297-31a6631cf8e1 admin admin] RBAC: Authorizing `identity:delete_user(user_id=aa1a855b44b246bebffea3f215e4b186)` {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:28:21.251832 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-706152c4-9cda-4311-9297-31a6631cf8e1 admin admin] RBAC: Authorization granted {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:28:21.340799 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-62704541-6180-4eff-a9d7-6dbc4d3cc45f admin admin] Authenticating user token {{(pid=81409) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:28:21.343291 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.fernet_utils [None req-62704541-6180-4eff-a9d7-6dbc4d3cc45f admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81409) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:28:21.352395 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.notifications [None req-706152c4-9cda-4311-9297-31a6631cf8e1 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'aa1a855b44b246bebffea3f215e4b186'} {{(pid=81408) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 13 04:28:21.362191 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.notifications [None req-706152c4-9cda-4311-9297-31a6631cf8e1 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'aa1a855b44b246bebffea3f215e4b186'} {{(pid=81408) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 13 04:28:21.363098 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.notifications [None req-706152c4-9cda-4311-9297-31a6631cf8e1 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'aa1a855b44b246bebffea3f215e4b186'} {{(pid=81408) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 13 04:28:21.363864 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.notifications [None req-706152c4-9cda-4311-9297-31a6631cf8e1 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'aa1a855b44b246bebffea3f215e4b186'} {{(pid=81408) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 13 04:28:21.374728 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.notifications [None req-706152c4-9cda-4311-9297-31a6631cf8e1 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'aa1a855b44b246bebffea3f215e4b186'} {{(pid=81408) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 13 04:28:21.402884 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 1116/2261] 10.222.0.27 () {58 vars in 1224 bytes} [Thu Oct 13 04:28:21 2022] DELETE /identity/v3/users/aa1a855b44b246bebffea3f215e4b186 => generated 0 bytes in 330 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 13 04:28:21.449925 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-62704541-6180-4eff-a9d7-6dbc4d3cc45f admin admin] Validating token access rules against request {{(pid=81409) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:28:21.466102 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG dbcounter [-] [81409] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=81409) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:28:21.553522 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bd089404-3d62-46c4-96ca-46d8f1b4f1d4 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd8812c3974f84871bb2d761cf0016257', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dcec0431fb6849499d6aec35a234daa6', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81409) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:28:21.555403 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd089404-3d62-46c4-96ca-46d8f1b4f1d4 service neutron] REQUEST_METHOD: `GET` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:28:21.555980 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd089404-3d62-46c4-96ca-46d8f1b4f1d4 service neutron] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:28:21.556620 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd089404-3d62-46c4-96ca-46d8f1b4f1d4 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:28:21.558800 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.fernet_utils [None req-bd089404-3d62-46c4-96ca-46d8f1b4f1d4 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81409) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:28:21.670859 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bd089404-3d62-46c4-96ca-46d8f1b4f1d4 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:28:21.672816 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bd089404-3d62-46c4-96ca-46d8f1b4f1d4 service neutron] RBAC: Authorization granted {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:28:21.690540 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 1146/2262] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Oct 13 04:28:21 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 350 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 13 04:28:22.016003 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-706152c4-9cda-4311-9297-31a6631cf8e1 admin admin] Authenticating user token {{(pid=81408) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:28:22.017471 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.fernet_utils [None req-706152c4-9cda-4311-9297-31a6631cf8e1 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81408) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:28:22.032422 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-706152c4-9cda-4311-9297-31a6631cf8e1 admin admin] Validating token access rules against request {{(pid=81408) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:28:22.039352 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8a0928ad-3a71-46c5-8362-b4a1354c3bea admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8ba823f313144da3b683bacef8fd62e0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81408) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:28:22.041243 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a0928ad-3a71-46c5-8362-b4a1354c3bea admin admin] REQUEST_METHOD: `DELETE` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:28:22.041721 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a0928ad-3a71-46c5-8362-b4a1354c3bea admin admin] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:28:22.042161 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a0928ad-3a71-46c5-8362-b4a1354c3bea admin admin] PATH_INFO: `/v3/projects/a5131be56034418f997a2e501ec3e8c1` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:28:22.043593 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8a0928ad-3a71-46c5-8362-b4a1354c3bea admin admin] RBAC: Authorizing `identity:delete_project(project_id=a5131be56034418f997a2e501ec3e8c1)` {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:28:22.045816 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8a0928ad-3a71-46c5-8362-b4a1354c3bea admin admin] RBAC: Authorization granted {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:28:22.106109 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.notifications [None req-8a0928ad-3a71-46c5-8362-b4a1354c3bea admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'a5131be56034418f997a2e501ec3e8c1', 'request_id': 'req-8a0928ad-3a71-46c5-8362-b4a1354c3bea'} {{(pid=81408) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 13 04:28:22.114883 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.notifications [None req-8a0928ad-3a71-46c5-8362-b4a1354c3bea admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'a5131be56034418f997a2e501ec3e8c1', 'request_id': 'req-8a0928ad-3a71-46c5-8362-b4a1354c3bea'} {{(pid=81408) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 13 04:28:22.123277 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.notifications [None req-8a0928ad-3a71-46c5-8362-b4a1354c3bea admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'a5131be56034418f997a2e501ec3e8c1', 'request_id': 'req-8a0928ad-3a71-46c5-8362-b4a1354c3bea'} {{(pid=81408) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 13 04:28:22.138887 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.notifications [None req-8a0928ad-3a71-46c5-8362-b4a1354c3bea admin admin] The token cache is being invalidate because project a5131be56034418f997a2e501ec3e8c1 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81408) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 13 04:28:22.139177 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.notifications [None req-8a0928ad-3a71-46c5-8362-b4a1354c3bea admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81408) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 13 04:28:22.139897 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.notifications [None req-8a0928ad-3a71-46c5-8362-b4a1354c3bea admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81408) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 13 04:28:22.142445 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 1117/2263] 10.222.0.27 () {58 vars in 1233 bytes} [Thu Oct 13 04:28:22 2022] DELETE /identity/v3/projects/a5131be56034418f997a2e501ec3e8c1 => generated 0 bytes in 127 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 13 04:28:22.147334 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37e9970b-bddc-4fe4-b21e-03af73d8f29c None None] REQUEST_METHOD: `POST` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:28:22.147675 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37e9970b-bddc-4fe4-b21e-03af73d8f29c None None] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:28:22.148667 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37e9970b-bddc-4fe4-b21e-03af73d8f29c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:28:22.173610 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.auth.core [None req-37e9970b-bddc-4fe4-b21e-03af73d8f29c None None] MFA Rules not processed for user `0e2d979563d6421995451a0e86bf3022`. Rule list: `[]` (Enabled: `True`). {{(pid=81409) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 13 04:28:22.174859 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.fernet_utils [None req-37e9970b-bddc-4fe4-b21e-03af73d8f29c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81409) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:28:22.274795 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 1147/2264] 10.222.0.27 () {58 vars in 958 bytes} [Thu Oct 13 04:28:22 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 129 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 13 04:28:22.283710 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1bbf298-914e-4fb8-836c-38dc155e28ae None None] REQUEST_METHOD: `POST` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:28:22.284203 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1bbf298-914e-4fb8-836c-38dc155e28ae None None] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:28:22.284632 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1bbf298-914e-4fb8-836c-38dc155e28ae None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:28:22.323326 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.auth.core [None req-c1bbf298-914e-4fb8-836c-38dc155e28ae None None] MFA Rules not processed for user `0e2d979563d6421995451a0e86bf3022`. Rule list: `[]` (Enabled: `True`). {{(pid=81408) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 13 04:28:22.324617 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.fernet_utils [None req-c1bbf298-914e-4fb8-836c-38dc155e28ae None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81408) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:28:22.351452 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-37e9970b-bddc-4fe4-b21e-03af73d8f29c None None] Authenticating user token {{(pid=81409) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:28:22.352452 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 1118/2265] 10.222.0.27 () {58 vars in 958 bytes} [Thu Oct 13 04:28:22 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 70 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 13 04:28:22.376077 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-37e9970b-bddc-4fe4-b21e-03af73d8f29c None None] Validating token access rules against request {{(pid=81409) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:28:22.389492 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b3d8b6ed-2848-4b9c-8738-e6673e87618f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8ba823f313144da3b683bacef8fd62e0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81409) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:28:22.390465 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3d8b6ed-2848-4b9c-8738-e6673e87618f admin admin] REQUEST_METHOD: `GET` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:28:22.391444 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3d8b6ed-2848-4b9c-8738-e6673e87618f admin admin] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:28:22.391824 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3d8b6ed-2848-4b9c-8738-e6673e87618f admin admin] PATH_INFO: `/v3/domains` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:28:22.392592 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b3d8b6ed-2848-4b9c-8738-e6673e87618f admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81409) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 13 04:28:22.393022 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b3d8b6ed-2848-4b9c-8738-e6673e87618f admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:28:22.394760 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b3d8b6ed-2848-4b9c-8738-e6673e87618f admin admin] RBAC: Authorization granted {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:28:22.414709 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 1148/2266] 10.222.0.27 () {58 vars in 1153 bytes} [Thu Oct 13 04:28:22 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 64 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 13 04:28:22.418896 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c1bbf298-914e-4fb8-836c-38dc155e28ae None None] Authenticating user token {{(pid=81408) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:28:22.432761 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c1bbf298-914e-4fb8-836c-38dc155e28ae None None] Validating token access rules against request {{(pid=81408) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:28:22.438046 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-528391f9-4d64-4e77-8505-e26342b0a4e2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8ba823f313144da3b683bacef8fd62e0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81408) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:28:22.439034 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-528391f9-4d64-4e77-8505-e26342b0a4e2 admin admin] REQUEST_METHOD: `POST` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:28:22.439275 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-528391f9-4d64-4e77-8505-e26342b0a4e2 admin admin] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:28:22.439594 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-528391f9-4d64-4e77-8505-e26342b0a4e2 admin admin] PATH_INFO: `/v3/projects` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:28:22.440315 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-528391f9-4d64-4e77-8505-e26342b0a4e2 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:28:22.442490 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-528391f9-4d64-4e77-8505-e26342b0a4e2 admin admin] RBAC: Authorization granted {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:28:22.455580 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG dbcounter [-] [81408] Writing DB stats keystone:SELECT=81,keystone:DELETE=17,keystone:INSERT=2 {{(pid=81408) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:28:22.477351 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 1119/2267] 10.222.0.27 () {60 vars in 1153 bytes} [Thu Oct 13 04:28:22 2022] POST /identity/v3/projects => generated 387 bytes in 59 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 13 04:28:22.481780 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b3d8b6ed-2848-4b9c-8738-e6673e87618f admin admin] Authenticating user token {{(pid=81409) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:28:22.494388 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b3d8b6ed-2848-4b9c-8738-e6673e87618f admin admin] Validating token access rules against request {{(pid=81409) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:28:22.563256 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG dbcounter [-] [81409] Writing DB stats keystone:SELECT=100 {{(pid=81409) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:28:22.580966 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8f377e55-19b7-4af6-9b66-2651691551e5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8ba823f313144da3b683bacef8fd62e0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81409) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:28:22.581920 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f377e55-19b7-4af6-9b66-2651691551e5 admin admin] REQUEST_METHOD: `POST` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:28:22.581920 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f377e55-19b7-4af6-9b66-2651691551e5 admin admin] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:28:22.582141 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f377e55-19b7-4af6-9b66-2651691551e5 admin admin] PATH_INFO: `/v3/users` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:28:22.582811 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8f377e55-19b7-4af6-9b66-2651691551e5 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:28:22.584755 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8f377e55-19b7-4af6-9b66-2651691551e5 admin admin] RBAC: Authorization granted {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:28:22.616168 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 1149/2268] 10.222.0.27 () {60 vars in 1144 bytes} [Thu Oct 13 04:28:22 2022] POST /identity/v3/users => generated 294 bytes in 135 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 13 04:28:22.620243 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-528391f9-4d64-4e77-8505-e26342b0a4e2 admin admin] Authenticating user token {{(pid=81408) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:28:22.631669 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-528391f9-4d64-4e77-8505-e26342b0a4e2 admin admin] Validating token access rules against request {{(pid=81408) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:28:22.637972 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6009e9a6-6f73-48b5-9b84-910c8bfafcec admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8ba823f313144da3b683bacef8fd62e0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81408) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:28:22.638644 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6009e9a6-6f73-48b5-9b84-910c8bfafcec admin admin] REQUEST_METHOD: `GET` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:28:22.638925 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6009e9a6-6f73-48b5-9b84-910c8bfafcec admin admin] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:28:22.639199 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6009e9a6-6f73-48b5-9b84-910c8bfafcec admin admin] PATH_INFO: `/v3/roles` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:28:22.639851 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6009e9a6-6f73-48b5-9b84-910c8bfafcec admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:28:22.641465 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6009e9a6-6f73-48b5-9b84-910c8bfafcec admin admin] RBAC: Authorization granted {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:28:22.655307 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 1120/2269] 10.222.0.27 () {58 vars in 1122 bytes} [Thu Oct 13 04:28:22 2022] GET /identity/v3/roles => generated 1420 bytes in 35 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 13 04:28:22.659059 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8f377e55-19b7-4af6-9b66-2651691551e5 admin admin] Authenticating user token {{(pid=81409) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:28:22.670021 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8f377e55-19b7-4af6-9b66-2651691551e5 admin admin] Validating token access rules against request {{(pid=81409) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:28:22.680554 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-51b694ca-eb62-4ed1-98d1-a365a453cf65 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8ba823f313144da3b683bacef8fd62e0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81409) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:28:22.681187 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51b694ca-eb62-4ed1-98d1-a365a453cf65 admin admin] REQUEST_METHOD: `GET` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:28:22.681484 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51b694ca-eb62-4ed1-98d1-a365a453cf65 admin admin] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:28:22.681742 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51b694ca-eb62-4ed1-98d1-a365a453cf65 admin admin] PATH_INFO: `/v3/roles` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:28:22.682440 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-51b694ca-eb62-4ed1-98d1-a365a453cf65 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:28:22.684253 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-51b694ca-eb62-4ed1-98d1-a365a453cf65 admin admin] RBAC: Authorization granted {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:28:22.697444 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 1150/2270] 10.222.0.27 () {58 vars in 1122 bytes} [Thu Oct 13 04:28:22 2022] GET /identity/v3/roles => generated 1420 bytes in 39 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 13 04:28:22.701795 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6009e9a6-6f73-48b5-9b84-910c8bfafcec admin admin] Authenticating user token {{(pid=81408) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:28:22.721746 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6009e9a6-6f73-48b5-9b84-910c8bfafcec admin admin] Validating token access rules against request {{(pid=81408) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:28:22.727360 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-38290971-ed19-4047-a205-999c25c0c669 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8ba823f313144da3b683bacef8fd62e0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81408) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:28:22.728029 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38290971-ed19-4047-a205-999c25c0c669 admin admin] REQUEST_METHOD: `PUT` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:28:22.728312 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38290971-ed19-4047-a205-999c25c0c669 admin admin] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:28:22.728575 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38290971-ed19-4047-a205-999c25c0c669 admin admin] PATH_INFO: `/v3/projects/f640b40145d346029612622767058c24/users/ef67d77b942346d4a877bcc1ed4faca1/roles/cc56c4ae13a7424ea771c15da4b153c1` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:28:22.746998 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-38290971-ed19-4047-a205-999c25c0c669 admin admin] RBAC: Authorizing `identity:create_grant(project_id=f640b40145d346029612622767058c24, user_id=ef67d77b942346d4a877bcc1ed4faca1, role_id=cc56c4ae13a7424ea771c15da4b153c1)` {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:28:22.749910 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-38290971-ed19-4047-a205-999c25c0c669 admin admin] RBAC: Authorization granted {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:28:22.769191 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 1121/2271] 10.222.0.27 () {60 vars in 1483 bytes} [Thu Oct 13 04:28:22 2022] PUT /identity/v3/projects/f640b40145d346029612622767058c24/users/ef67d77b942346d4a877bcc1ed4faca1/roles/cc56c4ae13a7424ea771c15da4b153c1 => generated 0 bytes in 68 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 13 04:28:22.779619 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9ea8d5a-51ec-45a4-b491-1eb097c29b9e None None] REQUEST_METHOD: `POST` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:28:22.779903 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9ea8d5a-51ec-45a4-b491-1eb097c29b9e None None] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:28:22.780204 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9ea8d5a-51ec-45a4-b491-1eb097c29b9e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:28:22.812128 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.auth.core [None req-f9ea8d5a-51ec-45a4-b491-1eb097c29b9e None None] MFA Rules not processed for user `ef67d77b942346d4a877bcc1ed4faca1`. Rule list: `[]` (Enabled: `True`). {{(pid=81409) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 13 04:28:22.813431 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.fernet_utils [None req-f9ea8d5a-51ec-45a4-b491-1eb097c29b9e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81409) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:28:22.897478 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 1151/2272] 10.222.0.27 () {58 vars in 958 bytes} [Thu Oct 13 04:28:22 2022] POST /identity/v3/auth/tokens => generated 3264 bytes in 119 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 13 04:28:22.902480 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-38290971-ed19-4047-a205-999c25c0c669 admin admin] Authenticating user token {{(pid=81408) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:28:22.972375 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-38290971-ed19-4047-a205-999c25c0c669 admin admin] Validating token access rules against request {{(pid=81408) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:28:22.978666 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-940dc5fa-b7e4-42ec-89a9-3d98fee35207 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8ba823f313144da3b683bacef8fd62e0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81408) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:28:22.979821 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-940dc5fa-b7e4-42ec-89a9-3d98fee35207 admin admin] REQUEST_METHOD: `POST` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:28:22.980264 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-940dc5fa-b7e4-42ec-89a9-3d98fee35207 admin admin] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:28:22.980710 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-940dc5fa-b7e4-42ec-89a9-3d98fee35207 admin admin] PATH_INFO: `/v3/projects` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:28:22.981763 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-940dc5fa-b7e4-42ec-89a9-3d98fee35207 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:28:22.983885 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-940dc5fa-b7e4-42ec-89a9-3d98fee35207 admin admin] RBAC: Authorization granted {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:28:23.022450 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 1122/2273] 10.222.0.27 () {60 vars in 1153 bytes} [Thu Oct 13 04:28:22 2022] POST /identity/v3/projects => generated 385 bytes in 118 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 13 04:28:23.025195 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f9ea8d5a-51ec-45a4-b491-1eb097c29b9e None None] Authenticating user token {{(pid=81409) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:28:23.036720 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f9ea8d5a-51ec-45a4-b491-1eb097c29b9e None None] Validating token access rules against request {{(pid=81409) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:28:23.138109 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ed087894-71c8-4bd9-bd14-1b73fe9479c2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8ba823f313144da3b683bacef8fd62e0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81409) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:28:23.138888 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed087894-71c8-4bd9-bd14-1b73fe9479c2 admin admin] REQUEST_METHOD: `POST` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:28:23.139170 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed087894-71c8-4bd9-bd14-1b73fe9479c2 admin admin] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:28:23.139632 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed087894-71c8-4bd9-bd14-1b73fe9479c2 admin admin] PATH_INFO: `/v3/users` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:28:23.140095 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ed087894-71c8-4bd9-bd14-1b73fe9479c2 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:28:23.142851 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ed087894-71c8-4bd9-bd14-1b73fe9479c2 admin admin] RBAC: Authorization granted {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:28:23.178706 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 1152/2274] 10.222.0.27 () {60 vars in 1144 bytes} [Thu Oct 13 04:28:23 2022] POST /identity/v3/users => generated 299 bytes in 153 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 13 04:28:23.180383 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-940dc5fa-b7e4-42ec-89a9-3d98fee35207 admin admin] Authenticating user token {{(pid=81408) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:28:23.191782 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-940dc5fa-b7e4-42ec-89a9-3d98fee35207 admin admin] Validating token access rules against request {{(pid=81408) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:28:23.197492 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bbb5fc13-eebd-4e13-84a9-3e898227f48b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8ba823f313144da3b683bacef8fd62e0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81408) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:28:23.198251 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bbb5fc13-eebd-4e13-84a9-3e898227f48b admin admin] REQUEST_METHOD: `GET` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:28:23.198609 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bbb5fc13-eebd-4e13-84a9-3e898227f48b admin admin] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:28:23.199132 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bbb5fc13-eebd-4e13-84a9-3e898227f48b admin admin] PATH_INFO: `/v3/roles` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:28:23.199860 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bbb5fc13-eebd-4e13-84a9-3e898227f48b admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:28:23.201663 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bbb5fc13-eebd-4e13-84a9-3e898227f48b admin admin] RBAC: Authorization granted {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:28:23.215005 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 1123/2275] 10.222.0.27 () {58 vars in 1122 bytes} [Thu Oct 13 04:28:23 2022] GET /identity/v3/roles => generated 1420 bytes in 35 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 13 04:28:23.219451 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ed087894-71c8-4bd9-bd14-1b73fe9479c2 admin admin] Authenticating user token {{(pid=81409) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:28:23.231244 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ed087894-71c8-4bd9-bd14-1b73fe9479c2 admin admin] Validating token access rules against request {{(pid=81409) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:28:23.242299 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cd41eb41-e92f-41cb-ad19-0cd82cf4a9fa admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8ba823f313144da3b683bacef8fd62e0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81409) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:28:23.243107 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd41eb41-e92f-41cb-ad19-0cd82cf4a9fa admin admin] REQUEST_METHOD: `PUT` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:28:23.243500 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd41eb41-e92f-41cb-ad19-0cd82cf4a9fa admin admin] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:28:23.243840 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd41eb41-e92f-41cb-ad19-0cd82cf4a9fa admin admin] PATH_INFO: `/v3/projects/9172fb3ffa2847bca7f56a6630200775/users/d90a2efa8b2041ae8625c8c5aa83f6c0/roles/ef55fbd94e1040be83695754075a7ed5` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:28:23.262849 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cd41eb41-e92f-41cb-ad19-0cd82cf4a9fa admin admin] RBAC: Authorizing `identity:create_grant(project_id=9172fb3ffa2847bca7f56a6630200775, user_id=d90a2efa8b2041ae8625c8c5aa83f6c0, role_id=ef55fbd94e1040be83695754075a7ed5)` {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:28:23.265826 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cd41eb41-e92f-41cb-ad19-0cd82cf4a9fa admin admin] RBAC: Authorization granted {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:28:23.282291 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 1153/2276] 10.222.0.27 () {60 vars in 1483 bytes} [Thu Oct 13 04:28:23 2022] PUT /identity/v3/projects/9172fb3ffa2847bca7f56a6630200775/users/d90a2efa8b2041ae8625c8c5aa83f6c0/roles/ef55fbd94e1040be83695754075a7ed5 => generated 0 bytes in 63 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 13 04:28:23.286585 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bbb5fc13-eebd-4e13-84a9-3e898227f48b admin admin] Authenticating user token {{(pid=81408) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:28:23.298150 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bbb5fc13-eebd-4e13-84a9-3e898227f48b admin admin] Validating token access rules against request {{(pid=81408) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:28:23.365200 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6744153a-3576-40f8-aa15-dcc198a0331b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8ba823f313144da3b683bacef8fd62e0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81408) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:28:23.365812 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6744153a-3576-40f8-aa15-dcc198a0331b admin admin] REQUEST_METHOD: `GET` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:28:23.366091 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6744153a-3576-40f8-aa15-dcc198a0331b admin admin] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:28:23.366402 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6744153a-3576-40f8-aa15-dcc198a0331b admin admin] PATH_INFO: `/v3/roles` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:28:23.367895 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6744153a-3576-40f8-aa15-dcc198a0331b admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:28:23.369540 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6744153a-3576-40f8-aa15-dcc198a0331b admin admin] RBAC: Authorization granted {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:28:23.379670 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 1124/2277] 10.222.0.27 () {58 vars in 1122 bytes} [Thu Oct 13 04:28:23 2022] GET /identity/v3/roles => generated 1420 bytes in 94 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 13 04:28:23.387946 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cd41eb41-e92f-41cb-ad19-0cd82cf4a9fa admin admin] Authenticating user token {{(pid=81409) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:28:23.400675 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cd41eb41-e92f-41cb-ad19-0cd82cf4a9fa admin admin] Validating token access rules against request {{(pid=81409) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:28:23.410238 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-05da343b-747f-4434-8ada-1fae3c78994d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8ba823f313144da3b683bacef8fd62e0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81409) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:28:23.410238 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05da343b-747f-4434-8ada-1fae3c78994d admin admin] REQUEST_METHOD: `PUT` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:28:23.410238 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05da343b-747f-4434-8ada-1fae3c78994d admin admin] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:28:23.410238 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05da343b-747f-4434-8ada-1fae3c78994d admin admin] PATH_INFO: `/v3/projects/9172fb3ffa2847bca7f56a6630200775/users/d90a2efa8b2041ae8625c8c5aa83f6c0/roles/ef55fbd94e1040be83695754075a7ed5` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:28:23.411054 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-05da343b-747f-4434-8ada-1fae3c78994d admin admin] RBAC: Authorizing `identity:create_grant(project_id=9172fb3ffa2847bca7f56a6630200775, user_id=d90a2efa8b2041ae8625c8c5aa83f6c0, role_id=ef55fbd94e1040be83695754075a7ed5)` {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:28:23.414035 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-05da343b-747f-4434-8ada-1fae3c78994d admin admin] RBAC: Authorization granted {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:28:23.425340 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 1154/2278] 10.222.0.27 () {60 vars in 1483 bytes} [Thu Oct 13 04:28:23 2022] PUT /identity/v3/projects/9172fb3ffa2847bca7f56a6630200775/users/d90a2efa8b2041ae8625c8c5aa83f6c0/roles/ef55fbd94e1040be83695754075a7ed5 => generated 0 bytes in 38 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 13 04:28:23.435825 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e69300b-010c-4fb2-95e6-371d12a0375b None None] REQUEST_METHOD: `POST` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:28:23.436111 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e69300b-010c-4fb2-95e6-371d12a0375b None None] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:28:23.436384 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e69300b-010c-4fb2-95e6-371d12a0375b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:28:23.466081 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.auth.core [None req-4e69300b-010c-4fb2-95e6-371d12a0375b None None] MFA Rules not processed for user `d90a2efa8b2041ae8625c8c5aa83f6c0`. Rule list: `[]` (Enabled: `True`). {{(pid=81408) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 13 04:28:23.467332 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.fernet_utils [None req-4e69300b-010c-4fb2-95e6-371d12a0375b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81408) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:28:23.481355 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG dbcounter [-] [81408] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81408) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:28:23.535411 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-05da343b-747f-4434-8ada-1fae3c78994d admin admin] Authenticating user token {{(pid=81409) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:28:23.539288 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.fernet_utils [None req-05da343b-747f-4434-8ada-1fae3c78994d admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81409) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:28:23.558421 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 1125/2279] 10.222.0.27 () {58 vars in 958 bytes} [Thu Oct 13 04:28:23 2022] POST /identity/v3/auth/tokens => generated 3329 bytes in 123 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 13 04:28:23.570075 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4e69300b-010c-4fb2-95e6-371d12a0375b None None] Authenticating user token {{(pid=81408) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:28:23.571510 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.fernet_utils [None req-4e69300b-010c-4fb2-95e6-371d12a0375b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81408) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:28:23.574621 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG dbcounter [-] [81409] Writing DB stats keystone:SELECT=92,keystone:INSERT=8 {{(pid=81409) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:28:23.654450 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-05da343b-747f-4434-8ada-1fae3c78994d admin admin] Validating token access rules against request {{(pid=81409) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:28:23.661938 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d5acd8a5-cb41-4f72-b0da-b1c1c61bd49f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8ba823f313144da3b683bacef8fd62e0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81409) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:28:23.662810 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5acd8a5-cb41-4f72-b0da-b1c1c61bd49f admin admin] REQUEST_METHOD: `DELETE` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:28:23.663898 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5acd8a5-cb41-4f72-b0da-b1c1c61bd49f admin admin] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:28:23.664364 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5acd8a5-cb41-4f72-b0da-b1c1c61bd49f admin admin] PATH_INFO: `/v3/users/35a058c849104a9699ecfcd0c81d398e` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:28:23.665779 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d5acd8a5-cb41-4f72-b0da-b1c1c61bd49f admin admin] RBAC: Authorizing `identity:delete_user(user_id=35a058c849104a9699ecfcd0c81d398e)` {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:28:23.667776 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d5acd8a5-cb41-4f72-b0da-b1c1c61bd49f admin admin] RBAC: Authorization granted {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:28:23.670786 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4e69300b-010c-4fb2-95e6-371d12a0375b None None] Validating token access rules against request {{(pid=81408) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:28:23.675904 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ef1d2093-0ff9-479d-9cc3-e1fa75da77f3 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6141b8a6c3d54e179d6879bb019f4d12', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dcec0431fb6849499d6aec35a234daa6', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81408) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:28:23.676655 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef1d2093-0ff9-479d-9cc3-e1fa75da77f3 service cinder] REQUEST_METHOD: `GET` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:28:23.676934 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef1d2093-0ff9-479d-9cc3-e1fa75da77f3 service cinder] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:28:23.677203 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef1d2093-0ff9-479d-9cc3-e1fa75da77f3 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:28:23.690105 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ef1d2093-0ff9-479d-9cc3-e1fa75da77f3 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:28:23.691576 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ef1d2093-0ff9-479d-9cc3-e1fa75da77f3 service cinder] RBAC: Authorization granted {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:28:23.707078 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 1126/2280] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Oct 13 04:28:23 2022] GET /identity/v3/auth/tokens => generated 3329 bytes in 137 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Oct 13 04:28:23.735871 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.notifications [None req-d5acd8a5-cb41-4f72-b0da-b1c1c61bd49f admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '35a058c849104a9699ecfcd0c81d398e'} {{(pid=81409) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 13 04:28:23.736646 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.notifications [None req-d5acd8a5-cb41-4f72-b0da-b1c1c61bd49f admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '35a058c849104a9699ecfcd0c81d398e'} {{(pid=81409) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 13 04:28:23.742530 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ef1d2093-0ff9-479d-9cc3-e1fa75da77f3 service cinder] Authenticating user token {{(pid=81408) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:28:23.746852 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.notifications [None req-d5acd8a5-cb41-4f72-b0da-b1c1c61bd49f admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '35a058c849104a9699ecfcd0c81d398e'} {{(pid=81409) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 13 04:28:23.747774 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.notifications [None req-d5acd8a5-cb41-4f72-b0da-b1c1c61bd49f admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '35a058c849104a9699ecfcd0c81d398e'} {{(pid=81409) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 13 04:28:23.756104 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ef1d2093-0ff9-479d-9cc3-e1fa75da77f3 service cinder] Validating token access rules against request {{(pid=81408) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:28:23.760035 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.notifications [None req-d5acd8a5-cb41-4f72-b0da-b1c1c61bd49f admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '35a058c849104a9699ecfcd0c81d398e'} {{(pid=81409) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 13 04:28:23.763027 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0cb2e083-922c-46d9-94d3-3305c4971daf admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8ba823f313144da3b683bacef8fd62e0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81408) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:28:23.764092 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0cb2e083-922c-46d9-94d3-3305c4971daf admin admin] REQUEST_METHOD: `DELETE` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:28:23.764377 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0cb2e083-922c-46d9-94d3-3305c4971daf admin admin] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:28:23.764647 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0cb2e083-922c-46d9-94d3-3305c4971daf admin admin] PATH_INFO: `/v3/users/ef67d77b942346d4a877bcc1ed4faca1` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:28:23.765811 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0cb2e083-922c-46d9-94d3-3305c4971daf admin admin] RBAC: Authorizing `identity:delete_user(user_id=ef67d77b942346d4a877bcc1ed4faca1)` {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:28:23.768100 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0cb2e083-922c-46d9-94d3-3305c4971daf admin admin] RBAC: Authorization granted {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:28:23.787280 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 1155/2281] 10.222.0.27 () {58 vars in 1224 bytes} [Thu Oct 13 04:28:23 2022] DELETE /identity/v3/users/35a058c849104a9699ecfcd0c81d398e => generated 0 bytes in 253 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 13 04:28:23.835575 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.notifications [None req-0cb2e083-922c-46d9-94d3-3305c4971daf admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'ef67d77b942346d4a877bcc1ed4faca1'} {{(pid=81408) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 13 04:28:23.845611 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.notifications [None req-0cb2e083-922c-46d9-94d3-3305c4971daf admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'ef67d77b942346d4a877bcc1ed4faca1'} {{(pid=81408) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 13 04:28:23.847179 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.notifications [None req-0cb2e083-922c-46d9-94d3-3305c4971daf admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'ef67d77b942346d4a877bcc1ed4faca1'} {{(pid=81408) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 13 04:28:23.847808 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.notifications [None req-0cb2e083-922c-46d9-94d3-3305c4971daf admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'ef67d77b942346d4a877bcc1ed4faca1'} {{(pid=81408) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 13 04:28:23.862577 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.notifications [None req-0cb2e083-922c-46d9-94d3-3305c4971daf admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'ef67d77b942346d4a877bcc1ed4faca1'} {{(pid=81408) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 13 04:28:23.883010 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 1127/2282] 10.222.0.27 () {58 vars in 1224 bytes} [Thu Oct 13 04:28:23 2022] DELETE /identity/v3/users/ef67d77b942346d4a877bcc1ed4faca1 => generated 0 bytes in 142 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 13 04:28:23.888708 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e411fad5-e2b6-4048-8e45-4e073dd480aa None None] REQUEST_METHOD: `POST` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:28:23.892382 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e411fad5-e2b6-4048-8e45-4e073dd480aa None None] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:28:23.892752 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e411fad5-e2b6-4048-8e45-4e073dd480aa None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:28:23.926234 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.auth.core [None req-e411fad5-e2b6-4048-8e45-4e073dd480aa None None] MFA Rules not processed for user `0e2d979563d6421995451a0e86bf3022`. Rule list: `[]` (Enabled: `True`). {{(pid=81409) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 13 04:28:23.927335 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.fernet_utils [None req-e411fad5-e2b6-4048-8e45-4e073dd480aa None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81409) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:28:24.049727 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0cb2e083-922c-46d9-94d3-3305c4971daf admin admin] Authenticating user token {{(pid=81408) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:28:24.052081 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.fernet_utils [None req-0cb2e083-922c-46d9-94d3-3305c4971daf admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81408) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:28:24.060394 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 1156/2283] 10.222.0.27 () {58 vars in 958 bytes} [Thu Oct 13 04:28:23 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 174 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 13 04:28:24.074174 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e411fad5-e2b6-4048-8e45-4e073dd480aa None None] Authenticating user token {{(pid=81409) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:28:24.080398 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.fernet_utils [None req-e411fad5-e2b6-4048-8e45-4e073dd480aa None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81409) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:28:24.083890 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0cb2e083-922c-46d9-94d3-3305c4971daf admin admin] Validating token access rules against request {{(pid=81408) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:28:24.101710 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1325dc9c-2ea6-4024-83e6-d1ff7d684307 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8ba823f313144da3b683bacef8fd62e0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81408) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:28:24.102423 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1325dc9c-2ea6-4024-83e6-d1ff7d684307 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:28:24.102807 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1325dc9c-2ea6-4024-83e6-d1ff7d684307 admin admin] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:28:24.103084 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1325dc9c-2ea6-4024-83e6-d1ff7d684307 admin admin] PATH_INFO: `/v3/projects/75387c76ea384e5c96a336751c613e6e` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:28:24.105319 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1325dc9c-2ea6-4024-83e6-d1ff7d684307 admin admin] RBAC: Authorizing `identity:delete_project(project_id=75387c76ea384e5c96a336751c613e6e)` {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:28:24.107572 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1325dc9c-2ea6-4024-83e6-d1ff7d684307 admin admin] RBAC: Authorization granted {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:28:24.138150 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG dbcounter [-] [81409] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81409) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:28:24.173301 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e411fad5-e2b6-4048-8e45-4e073dd480aa None None] Validating token access rules against request {{(pid=81409) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:28:24.179818 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-08f27f0d-0208-43f5-9533-5794f0d0224a service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd8812c3974f84871bb2d761cf0016257', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dcec0431fb6849499d6aec35a234daa6', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81409) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:28:24.180556 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08f27f0d-0208-43f5-9533-5794f0d0224a service neutron] REQUEST_METHOD: `GET` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:28:24.181275 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG dbcounter [-] [81408] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81408) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:28:24.182080 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08f27f0d-0208-43f5-9533-5794f0d0224a service neutron] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:28:24.182695 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08f27f0d-0208-43f5-9533-5794f0d0224a service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:28:24.200468 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-08f27f0d-0208-43f5-9533-5794f0d0224a service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:28:24.203231 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-08f27f0d-0208-43f5-9533-5794f0d0224a service neutron] RBAC: Authorization granted {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:28:24.233672 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 1157/2284] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Oct 13 04:28:24 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 158 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 13 04:28:24.245782 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.notifications [None req-1325dc9c-2ea6-4024-83e6-d1ff7d684307 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '75387c76ea384e5c96a336751c613e6e', 'request_id': 'req-1325dc9c-2ea6-4024-83e6-d1ff7d684307'} {{(pid=81408) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 13 04:28:24.262351 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.notifications [None req-1325dc9c-2ea6-4024-83e6-d1ff7d684307 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '75387c76ea384e5c96a336751c613e6e', 'request_id': 'req-1325dc9c-2ea6-4024-83e6-d1ff7d684307'} {{(pid=81408) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 13 04:28:24.269810 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.notifications [None req-1325dc9c-2ea6-4024-83e6-d1ff7d684307 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '75387c76ea384e5c96a336751c613e6e', 'request_id': 'req-1325dc9c-2ea6-4024-83e6-d1ff7d684307'} {{(pid=81408) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 13 04:28:24.295656 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.notifications [None req-1325dc9c-2ea6-4024-83e6-d1ff7d684307 admin admin] The token cache is being invalidate because project 75387c76ea384e5c96a336751c613e6e was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81408) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 13 04:28:24.295996 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.notifications [None req-1325dc9c-2ea6-4024-83e6-d1ff7d684307 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81408) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 13 04:28:24.296643 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.notifications [None req-1325dc9c-2ea6-4024-83e6-d1ff7d684307 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81408) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 13 04:28:24.298194 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 1128/2285] 10.222.0.27 () {58 vars in 1233 bytes} [Thu Oct 13 04:28:24 2022] DELETE /identity/v3/projects/75387c76ea384e5c96a336751c613e6e => generated 0 bytes in 250 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 13 04:28:24.303632 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58415732-1b6b-47bd-91f7-7f987df1273f None None] REQUEST_METHOD: `POST` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:28:24.303917 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58415732-1b6b-47bd-91f7-7f987df1273f None None] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:28:24.304251 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58415732-1b6b-47bd-91f7-7f987df1273f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:28:24.348211 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.auth.core [None req-58415732-1b6b-47bd-91f7-7f987df1273f None None] MFA Rules not processed for user `0e2d979563d6421995451a0e86bf3022`. Rule list: `[]` (Enabled: `True`). {{(pid=81409) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 13 04:28:24.349555 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.fernet_utils [None req-58415732-1b6b-47bd-91f7-7f987df1273f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81409) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:28:24.496011 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 1158/2286] 10.222.0.27 () {58 vars in 958 bytes} [Thu Oct 13 04:28:24 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 193 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 13 04:28:24.503511 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6047665-a387-4138-b42b-23845cb4f382 None None] REQUEST_METHOD: `POST` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:28:24.503807 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6047665-a387-4138-b42b-23845cb4f382 None None] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:28:24.504089 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6047665-a387-4138-b42b-23845cb4f382 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:28:24.530053 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.auth.core [None req-f6047665-a387-4138-b42b-23845cb4f382 None None] MFA Rules not processed for user `0e2d979563d6421995451a0e86bf3022`. Rule list: `[]` (Enabled: `True`). {{(pid=81408) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 13 04:28:24.531534 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.fernet_utils [None req-f6047665-a387-4138-b42b-23845cb4f382 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81408) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:28:24.540633 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 1129/2287] 10.222.0.27 () {58 vars in 958 bytes} [Thu Oct 13 04:28:24 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 38 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 13 04:28:24.545410 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-58415732-1b6b-47bd-91f7-7f987df1273f None None] Authenticating user token {{(pid=81409) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:28:24.556086 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-58415732-1b6b-47bd-91f7-7f987df1273f None None] Validating token access rules against request {{(pid=81409) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:28:24.561411 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b7fe9df5-479f-4924-b33e-8a2f63a619be admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8ba823f313144da3b683bacef8fd62e0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81409) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:28:24.561935 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7fe9df5-479f-4924-b33e-8a2f63a619be admin admin] REQUEST_METHOD: `GET` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:28:24.562233 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7fe9df5-479f-4924-b33e-8a2f63a619be admin admin] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:28:24.562498 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7fe9df5-479f-4924-b33e-8a2f63a619be admin admin] PATH_INFO: `/v3/domains` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:28:24.563166 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b7fe9df5-479f-4924-b33e-8a2f63a619be admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81409) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 13 04:28:24.563476 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b7fe9df5-479f-4924-b33e-8a2f63a619be admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:28:24.565319 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b7fe9df5-479f-4924-b33e-8a2f63a619be admin admin] RBAC: Authorization granted {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:28:24.584723 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 1159/2288] 10.222.0.27 () {58 vars in 1153 bytes} [Thu Oct 13 04:28:24 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 40 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 13 04:28:24.589778 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f6047665-a387-4138-b42b-23845cb4f382 None None] Authenticating user token {{(pid=81408) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:28:24.599701 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f6047665-a387-4138-b42b-23845cb4f382 None None] Validating token access rules against request {{(pid=81408) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:28:24.606872 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9c1faa33-9d7f-4d58-b387-8d24fbf07ae9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8ba823f313144da3b683bacef8fd62e0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81408) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:28:24.607563 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c1faa33-9d7f-4d58-b387-8d24fbf07ae9 admin admin] REQUEST_METHOD: `POST` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:28:24.607904 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c1faa33-9d7f-4d58-b387-8d24fbf07ae9 admin admin] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:28:24.608725 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c1faa33-9d7f-4d58-b387-8d24fbf07ae9 admin admin] PATH_INFO: `/v3/projects` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:28:24.608893 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9c1faa33-9d7f-4d58-b387-8d24fbf07ae9 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:28:24.610985 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9c1faa33-9d7f-4d58-b387-8d24fbf07ae9 admin admin] RBAC: Authorization granted {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:28:24.646675 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b7fe9df5-479f-4924-b33e-8a2f63a619be admin admin] Authenticating user token {{(pid=81409) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:28:24.652415 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 1130/2289] 10.222.0.27 () {60 vars in 1153 bytes} [Thu Oct 13 04:28:24 2022] POST /identity/v3/projects => generated 385 bytes in 58 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 13 04:28:24.657659 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b7fe9df5-479f-4924-b33e-8a2f63a619be admin admin] Validating token access rules against request {{(pid=81409) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:28:24.732574 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-89fc87f2-22ca-4e1f-a234-64b9bd0e27a7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8ba823f313144da3b683bacef8fd62e0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81409) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:28:24.733318 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89fc87f2-22ca-4e1f-a234-64b9bd0e27a7 admin admin] REQUEST_METHOD: `POST` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:28:24.733668 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89fc87f2-22ca-4e1f-a234-64b9bd0e27a7 admin admin] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:28:24.734042 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89fc87f2-22ca-4e1f-a234-64b9bd0e27a7 admin admin] PATH_INFO: `/v3/users` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:28:24.734843 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-89fc87f2-22ca-4e1f-a234-64b9bd0e27a7 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:28:24.740111 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-89fc87f2-22ca-4e1f-a234-64b9bd0e27a7 admin admin] RBAC: Authorization granted {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:28:24.763962 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9c1faa33-9d7f-4d58-b387-8d24fbf07ae9 admin admin] Authenticating user token {{(pid=81408) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:28:24.765987 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.fernet_utils [None req-9c1faa33-9d7f-4d58-b387-8d24fbf07ae9 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81408) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:28:24.773609 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 1160/2290] 10.222.0.27 () {60 vars in 1144 bytes} [Thu Oct 13 04:28:24 2022] POST /identity/v3/users => generated 293 bytes in 127 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 13 04:28:24.777831 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-89fc87f2-22ca-4e1f-a234-64b9bd0e27a7 admin admin] Authenticating user token {{(pid=81409) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:28:24.785505 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9c1faa33-9d7f-4d58-b387-8d24fbf07ae9 admin admin] Validating token access rules against request {{(pid=81408) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:28:24.793384 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-feadb396-ff6e-44c7-90ef-ad577d746674 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8ba823f313144da3b683bacef8fd62e0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81408) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:28:24.794095 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-feadb396-ff6e-44c7-90ef-ad577d746674 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:28:24.794388 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-feadb396-ff6e-44c7-90ef-ad577d746674 admin admin] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:28:24.794656 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-feadb396-ff6e-44c7-90ef-ad577d746674 admin admin] PATH_INFO: `/v3/projects/f640b40145d346029612622767058c24` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:28:24.795818 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-feadb396-ff6e-44c7-90ef-ad577d746674 admin admin] RBAC: Authorizing `identity:delete_project(project_id=f640b40145d346029612622767058c24)` {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:28:24.798445 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-feadb396-ff6e-44c7-90ef-ad577d746674 admin admin] RBAC: Authorization granted {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:28:24.814712 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-89fc87f2-22ca-4e1f-a234-64b9bd0e27a7 admin admin] Validating token access rules against request {{(pid=81409) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:28:24.825143 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-273d108b-4479-4ea3-9b3b-124ad117410e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8ba823f313144da3b683bacef8fd62e0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81409) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:28:24.826014 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-273d108b-4479-4ea3-9b3b-124ad117410e admin admin] REQUEST_METHOD: `GET` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:28:24.826440 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-273d108b-4479-4ea3-9b3b-124ad117410e admin admin] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:28:24.826899 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-273d108b-4479-4ea3-9b3b-124ad117410e admin admin] PATH_INFO: `/v3/roles` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:28:24.828051 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-273d108b-4479-4ea3-9b3b-124ad117410e admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:28:24.832628 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-273d108b-4479-4ea3-9b3b-124ad117410e admin admin] RBAC: Authorization granted {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:28:24.860752 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 1161/2291] 10.222.0.27 () {58 vars in 1122 bytes} [Thu Oct 13 04:28:24 2022] GET /identity/v3/roles => generated 1420 bytes in 83 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Oct 13 04:28:24.864962 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-273d108b-4479-4ea3-9b3b-124ad117410e admin admin] Authenticating user token {{(pid=81409) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:28:24.879129 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-273d108b-4479-4ea3-9b3b-124ad117410e admin admin] Validating token access rules against request {{(pid=81409) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:28:24.890403 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.notifications [None req-feadb396-ff6e-44c7-90ef-ad577d746674 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'f640b40145d346029612622767058c24', 'request_id': 'req-feadb396-ff6e-44c7-90ef-ad577d746674'} {{(pid=81408) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 13 04:28:24.902056 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.notifications [None req-feadb396-ff6e-44c7-90ef-ad577d746674 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'f640b40145d346029612622767058c24', 'request_id': 'req-feadb396-ff6e-44c7-90ef-ad577d746674'} {{(pid=81408) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 13 04:28:24.913961 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.notifications [None req-feadb396-ff6e-44c7-90ef-ad577d746674 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'f640b40145d346029612622767058c24', 'request_id': 'req-feadb396-ff6e-44c7-90ef-ad577d746674'} {{(pid=81408) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 13 04:28:24.926569 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.notifications [None req-feadb396-ff6e-44c7-90ef-ad577d746674 admin admin] The token cache is being invalidate because project f640b40145d346029612622767058c24 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81408) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 13 04:28:24.926864 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.notifications [None req-feadb396-ff6e-44c7-90ef-ad577d746674 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81408) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 13 04:28:24.927545 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.notifications [None req-feadb396-ff6e-44c7-90ef-ad577d746674 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81408) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 13 04:28:24.932359 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 1131/2292] 10.222.0.27 () {58 vars in 1233 bytes} [Thu Oct 13 04:28:24 2022] DELETE /identity/v3/projects/f640b40145d346029612622767058c24 => generated 0 bytes in 166 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 13 04:28:24.933408 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-feadb396-ff6e-44c7-90ef-ad577d746674 admin admin] Authenticating user token {{(pid=81408) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:28:24.934750 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG dbcounter [-] [81409] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81409) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:28:24.934900 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.fernet_utils [None req-feadb396-ff6e-44c7-90ef-ad577d746674 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81408) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:28:24.954501 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-97d23087-51ff-4b15-8286-669903160156 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8ba823f313144da3b683bacef8fd62e0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81409) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:28:24.955148 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97d23087-51ff-4b15-8286-669903160156 admin admin] REQUEST_METHOD: `GET` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:28:24.955555 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97d23087-51ff-4b15-8286-669903160156 admin admin] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:28:24.955832 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97d23087-51ff-4b15-8286-669903160156 admin admin] PATH_INFO: `/v3/roles` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:28:24.956604 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-97d23087-51ff-4b15-8286-669903160156 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:28:24.958054 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-97d23087-51ff-4b15-8286-669903160156 admin admin] RBAC: Authorization granted {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:28:24.969386 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 1162/2293] 10.222.0.27 () {58 vars in 1122 bytes} [Thu Oct 13 04:28:24 2022] GET /identity/v3/roles => generated 1420 bytes in 105 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 13 04:28:24.973326 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-97d23087-51ff-4b15-8286-669903160156 admin admin] Authenticating user token {{(pid=81409) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:28:24.975087 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.fernet_utils [None req-97d23087-51ff-4b15-8286-669903160156 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81409) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:28:24.989371 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-97d23087-51ff-4b15-8286-669903160156 admin admin] Validating token access rules against request {{(pid=81409) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:28:24.996291 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8fa156b9-92ce-40c7-8170-3c2aed38f9e9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8ba823f313144da3b683bacef8fd62e0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81409) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:28:24.997033 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8fa156b9-92ce-40c7-8170-3c2aed38f9e9 admin admin] REQUEST_METHOD: `PUT` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:28:24.997385 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8fa156b9-92ce-40c7-8170-3c2aed38f9e9 admin admin] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:28:24.997739 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8fa156b9-92ce-40c7-8170-3c2aed38f9e9 admin admin] PATH_INFO: `/v3/projects/36d9f45714674d75bdc9b7c155c1b666/users/2a7c9f30017f446bb9178c04357905d6/roles/cc56c4ae13a7424ea771c15da4b153c1` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:28:25.017047 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-feadb396-ff6e-44c7-90ef-ad577d746674 admin admin] Validating token access rules against request {{(pid=81408) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:28:25.020442 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8fa156b9-92ce-40c7-8170-3c2aed38f9e9 admin admin] RBAC: Authorizing `identity:create_grant(project_id=36d9f45714674d75bdc9b7c155c1b666, user_id=2a7c9f30017f446bb9178c04357905d6, role_id=cc56c4ae13a7424ea771c15da4b153c1)` {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:28:25.022303 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8634aeb1-2b7c-4d26-b627-3a2505861757 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8ba823f313144da3b683bacef8fd62e0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81408) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:28:25.023594 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8634aeb1-2b7c-4d26-b627-3a2505861757 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:28:25.023924 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8fa156b9-92ce-40c7-8170-3c2aed38f9e9 admin admin] RBAC: Authorization granted {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:28:25.024231 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8634aeb1-2b7c-4d26-b627-3a2505861757 admin admin] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:28:25.024603 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8634aeb1-2b7c-4d26-b627-3a2505861757 admin admin] PATH_INFO: `/v3/users/d90a2efa8b2041ae8625c8c5aa83f6c0` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:28:25.025867 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8634aeb1-2b7c-4d26-b627-3a2505861757 admin admin] RBAC: Authorizing `identity:delete_user(user_id=d90a2efa8b2041ae8625c8c5aa83f6c0)` {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:28:25.027984 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8634aeb1-2b7c-4d26-b627-3a2505861757 admin admin] RBAC: Authorization granted {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:28:25.033671 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG dbcounter [-] [81408] Writing DB stats keystone:DELETE=14,keystone:SELECT=83,keystone:INSERT=3 {{(pid=81408) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:28:25.039798 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 1163/2294] 10.222.0.27 () {60 vars in 1483 bytes} [Thu Oct 13 04:28:24 2022] PUT /identity/v3/projects/36d9f45714674d75bdc9b7c155c1b666/users/2a7c9f30017f446bb9178c04357905d6/roles/cc56c4ae13a7424ea771c15da4b153c1 => generated 0 bytes in 67 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 13 04:28:25.050408 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bff161db-e394-40ca-a1cd-3cf6a6566266 None None] REQUEST_METHOD: `POST` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:28:25.050840 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bff161db-e394-40ca-a1cd-3cf6a6566266 None None] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:28:25.051181 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bff161db-e394-40ca-a1cd-3cf6a6566266 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:28:25.092440 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.auth.core [None req-bff161db-e394-40ca-a1cd-3cf6a6566266 None None] MFA Rules not processed for user `2a7c9f30017f446bb9178c04357905d6`. Rule list: `[]` (Enabled: `True`). {{(pid=81409) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 13 04:28:25.093732 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.fernet_utils [None req-bff161db-e394-40ca-a1cd-3cf6a6566266 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81409) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:28:25.095791 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.notifications [None req-8634aeb1-2b7c-4d26-b627-3a2505861757 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'd90a2efa8b2041ae8625c8c5aa83f6c0'} {{(pid=81408) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 13 04:28:25.104774 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.notifications [None req-8634aeb1-2b7c-4d26-b627-3a2505861757 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'd90a2efa8b2041ae8625c8c5aa83f6c0'} {{(pid=81408) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 13 04:28:25.105566 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.notifications [None req-8634aeb1-2b7c-4d26-b627-3a2505861757 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'd90a2efa8b2041ae8625c8c5aa83f6c0'} {{(pid=81408) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 13 04:28:25.107343 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.notifications [None req-8634aeb1-2b7c-4d26-b627-3a2505861757 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'd90a2efa8b2041ae8625c8c5aa83f6c0'} {{(pid=81408) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 13 04:28:25.118989 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.notifications [None req-8634aeb1-2b7c-4d26-b627-3a2505861757 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'd90a2efa8b2041ae8625c8c5aa83f6c0'} {{(pid=81408) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 13 04:28:25.142977 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 1132/2295] 10.222.0.27 () {58 vars in 1224 bytes} [Thu Oct 13 04:28:24 2022] DELETE /identity/v3/users/d90a2efa8b2041ae8625c8c5aa83f6c0 => generated 0 bytes in 210 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 13 04:28:25.197175 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 1164/2296] 10.222.0.27 () {58 vars in 958 bytes} [Thu Oct 13 04:28:25 2022] POST /identity/v3/auth/tokens => generated 3262 bytes in 148 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 13 04:28:25.202706 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8634aeb1-2b7c-4d26-b627-3a2505861757 admin admin] Authenticating user token {{(pid=81408) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:28:25.207980 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.fernet_utils [None req-8634aeb1-2b7c-4d26-b627-3a2505861757 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81408) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:28:25.304711 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8634aeb1-2b7c-4d26-b627-3a2505861757 admin admin] Validating token access rules against request {{(pid=81408) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:28:25.312028 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c8055a42-3353-49c0-99b3-de5bf5d31caf admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8ba823f313144da3b683bacef8fd62e0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81408) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:28:25.313196 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8055a42-3353-49c0-99b3-de5bf5d31caf admin admin] REQUEST_METHOD: `POST` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:28:25.314008 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8055a42-3353-49c0-99b3-de5bf5d31caf admin admin] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:28:25.314482 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8055a42-3353-49c0-99b3-de5bf5d31caf admin admin] PATH_INFO: `/v3/projects` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:28:25.315194 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c8055a42-3353-49c0-99b3-de5bf5d31caf admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:28:25.318381 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c8055a42-3353-49c0-99b3-de5bf5d31caf admin admin] RBAC: Authorization granted {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:28:25.352800 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 1133/2297] 10.222.0.27 () {60 vars in 1153 bytes} [Thu Oct 13 04:28:25 2022] POST /identity/v3/projects => generated 385 bytes in 152 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 13 04:28:25.357152 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bff161db-e394-40ca-a1cd-3cf6a6566266 None None] Authenticating user token {{(pid=81409) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:28:25.434578 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bff161db-e394-40ca-a1cd-3cf6a6566266 None None] Validating token access rules against request {{(pid=81409) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:28:25.513646 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG dbcounter [-] [81409] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81409) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:28:25.525631 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a9fa7123-9e5b-422d-a042-9c7fc526ced6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8ba823f313144da3b683bacef8fd62e0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81409) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:28:25.526329 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9fa7123-9e5b-422d-a042-9c7fc526ced6 admin admin] REQUEST_METHOD: `POST` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:28:25.526992 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9fa7123-9e5b-422d-a042-9c7fc526ced6 admin admin] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:28:25.527460 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9fa7123-9e5b-422d-a042-9c7fc526ced6 admin admin] PATH_INFO: `/v3/users` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:28:25.528734 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a9fa7123-9e5b-422d-a042-9c7fc526ced6 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:28:25.530952 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a9fa7123-9e5b-422d-a042-9c7fc526ced6 admin admin] RBAC: Authorization granted {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:28:25.539996 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c8055a42-3353-49c0-99b3-de5bf5d31caf admin admin] Authenticating user token {{(pid=81408) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:28:25.542470 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.fernet_utils [None req-c8055a42-3353-49c0-99b3-de5bf5d31caf admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81408) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:28:25.573168 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c8055a42-3353-49c0-99b3-de5bf5d31caf admin admin] Validating token access rules against request {{(pid=81408) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:28:25.586109 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 1165/2298] 10.222.0.27 () {60 vars in 1144 bytes} [Thu Oct 13 04:28:25 2022] POST /identity/v3/users => generated 293 bytes in 229 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 13 04:28:25.588025 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-60310db4-4598-4488-a513-3b20ea74eca3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8ba823f313144da3b683bacef8fd62e0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81408) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:28:25.588805 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60310db4-4598-4488-a513-3b20ea74eca3 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:28:25.589031 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60310db4-4598-4488-a513-3b20ea74eca3 admin admin] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:28:25.589709 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60310db4-4598-4488-a513-3b20ea74eca3 admin admin] PATH_INFO: `/v3/projects/9172fb3ffa2847bca7f56a6630200775` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:28:25.591643 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-60310db4-4598-4488-a513-3b20ea74eca3 admin admin] RBAC: Authorizing `identity:delete_project(project_id=9172fb3ffa2847bca7f56a6630200775)` {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:28:25.592955 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a9fa7123-9e5b-422d-a042-9c7fc526ced6 admin admin] Authenticating user token {{(pid=81409) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:28:25.593660 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-60310db4-4598-4488-a513-3b20ea74eca3 admin admin] RBAC: Authorization granted {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:28:25.610883 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a9fa7123-9e5b-422d-a042-9c7fc526ced6 admin admin] Validating token access rules against request {{(pid=81409) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:28:25.616749 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f9918be5-6fac-40bd-916d-be8016295106 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8ba823f313144da3b683bacef8fd62e0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81409) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:28:25.617917 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9918be5-6fac-40bd-916d-be8016295106 admin admin] REQUEST_METHOD: `GET` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:28:25.618456 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9918be5-6fac-40bd-916d-be8016295106 admin admin] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:28:25.618948 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9918be5-6fac-40bd-916d-be8016295106 admin admin] PATH_INFO: `/v3/roles` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:28:25.619942 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f9918be5-6fac-40bd-916d-be8016295106 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:28:25.621813 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f9918be5-6fac-40bd-916d-be8016295106 admin admin] RBAC: Authorization granted {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:28:25.648745 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 1166/2299] 10.222.0.27 () {58 vars in 1122 bytes} [Thu Oct 13 04:28:25 2022] GET /identity/v3/roles => generated 1420 bytes in 56 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 13 04:28:25.649749 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f9918be5-6fac-40bd-916d-be8016295106 admin admin] Authenticating user token {{(pid=81409) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:28:25.661984 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f9918be5-6fac-40bd-916d-be8016295106 admin admin] Validating token access rules against request {{(pid=81409) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:28:25.680836 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.notifications [None req-60310db4-4598-4488-a513-3b20ea74eca3 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '9172fb3ffa2847bca7f56a6630200775', 'request_id': 'req-60310db4-4598-4488-a513-3b20ea74eca3'} {{(pid=81408) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 13 04:28:25.692946 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.notifications [None req-60310db4-4598-4488-a513-3b20ea74eca3 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '9172fb3ffa2847bca7f56a6630200775', 'request_id': 'req-60310db4-4598-4488-a513-3b20ea74eca3'} {{(pid=81408) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 13 04:28:25.705398 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.notifications [None req-60310db4-4598-4488-a513-3b20ea74eca3 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '9172fb3ffa2847bca7f56a6630200775', 'request_id': 'req-60310db4-4598-4488-a513-3b20ea74eca3'} {{(pid=81408) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 13 04:28:25.707760 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG dbcounter [-] [81408] Writing DB stats keystone:SELECT=81,keystone:DELETE=16,keystone:INSERT=3 {{(pid=81408) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:28:25.722409 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.notifications [None req-60310db4-4598-4488-a513-3b20ea74eca3 admin admin] The token cache is being invalidate because project 9172fb3ffa2847bca7f56a6630200775 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81408) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 13 04:28:25.722794 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.notifications [None req-60310db4-4598-4488-a513-3b20ea74eca3 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81408) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 13 04:28:25.723550 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.notifications [None req-60310db4-4598-4488-a513-3b20ea74eca3 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81408) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 13 04:28:25.726833 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 1134/2300] 10.222.0.27 () {58 vars in 1233 bytes} [Thu Oct 13 04:28:25 2022] DELETE /identity/v3/projects/9172fb3ffa2847bca7f56a6630200775 => generated 0 bytes in 187 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 13 04:28:25.732878 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d3e7f63-1d77-45fc-8a97-1cfd669ab220 None None] REQUEST_METHOD: `POST` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:28:25.733172 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d3e7f63-1d77-45fc-8a97-1cfd669ab220 None None] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:28:25.733447 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d3e7f63-1d77-45fc-8a97-1cfd669ab220 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:28:25.745350 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-14fcdea6-7311-4c10-bbe6-f5a804513718 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8ba823f313144da3b683bacef8fd62e0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81409) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:28:25.745971 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14fcdea6-7311-4c10-bbe6-f5a804513718 admin admin] REQUEST_METHOD: `GET` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:28:25.746291 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14fcdea6-7311-4c10-bbe6-f5a804513718 admin admin] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:28:25.746566 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14fcdea6-7311-4c10-bbe6-f5a804513718 admin admin] PATH_INFO: `/v3/roles` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:28:25.747217 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-14fcdea6-7311-4c10-bbe6-f5a804513718 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:28:25.749018 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-14fcdea6-7311-4c10-bbe6-f5a804513718 admin admin] RBAC: Authorization granted {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:28:25.765227 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 1167/2301] 10.222.0.27 () {58 vars in 1122 bytes} [Thu Oct 13 04:28:25 2022] GET /identity/v3/roles => generated 1420 bytes in 116 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 13 04:28:25.769208 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-14fcdea6-7311-4c10-bbe6-f5a804513718 admin admin] Authenticating user token {{(pid=81409) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:28:25.771009 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.fernet_utils [None req-14fcdea6-7311-4c10-bbe6-f5a804513718 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81409) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:28:25.775936 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.auth.core [None req-4d3e7f63-1d77-45fc-8a97-1cfd669ab220 None None] MFA Rules not processed for user `0e2d979563d6421995451a0e86bf3022`. Rule list: `[]` (Enabled: `True`). {{(pid=81408) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 13 04:28:25.777703 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.fernet_utils [None req-4d3e7f63-1d77-45fc-8a97-1cfd669ab220 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81408) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:28:25.788507 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 1135/2302] 10.222.0.27 () {58 vars in 958 bytes} [Thu Oct 13 04:28:25 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 56 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 13 04:28:25.789281 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-14fcdea6-7311-4c10-bbe6-f5a804513718 admin admin] Validating token access rules against request {{(pid=81409) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:28:25.794623 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e340bfd7-27bb-4e4b-b95a-1a1b40afef85 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8ba823f313144da3b683bacef8fd62e0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81409) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:28:25.795315 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e340bfd7-27bb-4e4b-b95a-1a1b40afef85 admin admin] REQUEST_METHOD: `PUT` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:28:25.795599 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e340bfd7-27bb-4e4b-b95a-1a1b40afef85 admin admin] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:28:25.796006 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e340bfd7-27bb-4e4b-b95a-1a1b40afef85 admin admin] PATH_INFO: `/v3/projects/797d67a0c1cc4d478291fef206129965/users/3c571c60a63c42a696b3ca12f9678618/roles/cc56c4ae13a7424ea771c15da4b153c1` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:28:25.797426 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40db6894-7eff-4ea4-b2a8-a37d80c130f5 None None] REQUEST_METHOD: `POST` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:28:25.797709 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40db6894-7eff-4ea4-b2a8-a37d80c130f5 None None] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:28:25.797982 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40db6894-7eff-4ea4-b2a8-a37d80c130f5 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:28:25.822658 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e340bfd7-27bb-4e4b-b95a-1a1b40afef85 admin admin] RBAC: Authorizing `identity:create_grant(project_id=797d67a0c1cc4d478291fef206129965, user_id=3c571c60a63c42a696b3ca12f9678618, role_id=cc56c4ae13a7424ea771c15da4b153c1)` {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:28:25.823737 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.auth.core [None req-40db6894-7eff-4ea4-b2a8-a37d80c130f5 None None] MFA Rules not processed for user `0e2d979563d6421995451a0e86bf3022`. Rule list: `[]` (Enabled: `True`). {{(pid=81408) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 13 04:28:25.825356 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.fernet_utils [None req-40db6894-7eff-4ea4-b2a8-a37d80c130f5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81408) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:28:25.825716 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e340bfd7-27bb-4e4b-b95a-1a1b40afef85 admin admin] RBAC: Authorization granted {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:28:25.840991 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 1136/2303] 10.222.0.27 () {58 vars in 958 bytes} [Thu Oct 13 04:28:25 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 44 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 13 04:28:25.845836 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-40db6894-7eff-4ea4-b2a8-a37d80c130f5 None None] Authenticating user token {{(pid=81408) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:28:25.854676 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 1168/2304] 10.222.0.27 () {60 vars in 1483 bytes} [Thu Oct 13 04:28:25 2022] PUT /identity/v3/projects/797d67a0c1cc4d478291fef206129965/users/3c571c60a63c42a696b3ca12f9678618/roles/cc56c4ae13a7424ea771c15da4b153c1 => generated 0 bytes in 78 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 13 04:28:25.857307 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-40db6894-7eff-4ea4-b2a8-a37d80c130f5 None None] Validating token access rules against request {{(pid=81408) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:28:25.859323 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52826d7e-6775-4001-ac84-efbf57283998 None None] REQUEST_METHOD: `POST` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:28:25.859608 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52826d7e-6775-4001-ac84-efbf57283998 None None] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:28:25.859882 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52826d7e-6775-4001-ac84-efbf57283998 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:28:25.917368 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.auth.core [None req-52826d7e-6775-4001-ac84-efbf57283998 None None] MFA Rules not processed for user `3c571c60a63c42a696b3ca12f9678618`. Rule list: `[]` (Enabled: `True`). {{(pid=81409) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 13 04:28:25.918642 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.fernet_utils [None req-52826d7e-6775-4001-ac84-efbf57283998 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81409) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:28:25.946186 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f6bbd46-93e8-49d9-a7b0-9b25bd1c738d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8ba823f313144da3b683bacef8fd62e0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81408) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:28:25.947729 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f6bbd46-93e8-49d9-a7b0-9b25bd1c738d admin admin] REQUEST_METHOD: `GET` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:28:25.947977 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f6bbd46-93e8-49d9-a7b0-9b25bd1c738d admin admin] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:28:25.948348 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f6bbd46-93e8-49d9-a7b0-9b25bd1c738d admin admin] PATH_INFO: `/v3/domains` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:28:25.949605 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5f6bbd46-93e8-49d9-a7b0-9b25bd1c738d admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81408) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 13 04:28:25.950014 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5f6bbd46-93e8-49d9-a7b0-9b25bd1c738d admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:28:25.951970 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5f6bbd46-93e8-49d9-a7b0-9b25bd1c738d admin admin] RBAC: Authorization granted {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:28:25.969617 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 1137/2305] 10.222.0.27 () {58 vars in 1153 bytes} [Thu Oct 13 04:28:25 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 121 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 13 04:28:25.970823 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f6bbd46-93e8-49d9-a7b0-9b25bd1c738d admin admin] Authenticating user token {{(pid=81408) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:28:25.983825 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f6bbd46-93e8-49d9-a7b0-9b25bd1c738d admin admin] Validating token access rules against request {{(pid=81408) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:28:25.990440 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-33dc9834-04a5-4238-a1ae-070807af697e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8ba823f313144da3b683bacef8fd62e0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81408) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:28:25.991200 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33dc9834-04a5-4238-a1ae-070807af697e admin admin] REQUEST_METHOD: `POST` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:28:25.992252 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33dc9834-04a5-4238-a1ae-070807af697e admin admin] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:28:25.992629 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33dc9834-04a5-4238-a1ae-070807af697e admin admin] PATH_INFO: `/v3/projects` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:28:25.993500 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-33dc9834-04a5-4238-a1ae-070807af697e admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:28:25.995906 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-33dc9834-04a5-4238-a1ae-070807af697e admin admin] RBAC: Authorization granted {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:28:26.027612 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 1169/2306] 10.222.0.27 () {58 vars in 958 bytes} [Thu Oct 13 04:28:25 2022] POST /identity/v3/auth/tokens => generated 3262 bytes in 169 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 13 04:28:26.032569 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-52826d7e-6775-4001-ac84-efbf57283998 None None] Authenticating user token {{(pid=81409) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:28:26.045559 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 1138/2307] 10.222.0.27 () {60 vars in 1153 bytes} [Thu Oct 13 04:28:25 2022] POST /identity/v3/projects => generated 375 bytes in 72 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 13 04:28:26.045559 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-33dc9834-04a5-4238-a1ae-070807af697e admin admin] Authenticating user token {{(pid=81408) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:28:26.048466 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.fernet_utils [None req-33dc9834-04a5-4238-a1ae-070807af697e admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81408) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:28:26.063397 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-52826d7e-6775-4001-ac84-efbf57283998 None None] Validating token access rules against request {{(pid=81409) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:28:26.101750 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG dbcounter [-] [81409] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=81409) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:28:26.138932 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-33dc9834-04a5-4238-a1ae-070807af697e admin admin] Validating token access rules against request {{(pid=81408) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:28:26.142992 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fba066e6-28cd-4d33-b97c-ceab9ae78d50 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8ba823f313144da3b683bacef8fd62e0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81409) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:28:26.143839 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3a8093ae-3854-42fa-945c-5c9832eadf3a service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd8812c3974f84871bb2d761cf0016257', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dcec0431fb6849499d6aec35a234daa6', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81408) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:28:26.144202 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fba066e6-28cd-4d33-b97c-ceab9ae78d50 admin admin] REQUEST_METHOD: `POST` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:28:26.144448 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a8093ae-3854-42fa-945c-5c9832eadf3a service neutron] REQUEST_METHOD: `GET` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:28:26.144812 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a8093ae-3854-42fa-945c-5c9832eadf3a service neutron] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:28:26.144980 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a8093ae-3854-42fa-945c-5c9832eadf3a service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:28:26.145694 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fba066e6-28cd-4d33-b97c-ceab9ae78d50 admin admin] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:28:26.146365 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fba066e6-28cd-4d33-b97c-ceab9ae78d50 admin admin] PATH_INFO: `/v3/projects` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:28:26.147004 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.fernet_utils [None req-3a8093ae-3854-42fa-945c-5c9832eadf3a service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81408) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:28:26.147431 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fba066e6-28cd-4d33-b97c-ceab9ae78d50 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:28:26.151032 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fba066e6-28cd-4d33-b97c-ceab9ae78d50 admin admin] RBAC: Authorization granted {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:28:26.200269 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 1170/2308] 10.222.0.27 () {60 vars in 1153 bytes} [Thu Oct 13 04:28:26 2022] POST /identity/v3/projects => generated 387 bytes in 165 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 13 04:28:26.203364 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fba066e6-28cd-4d33-b97c-ceab9ae78d50 admin admin] Authenticating user token {{(pid=81409) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:28:26.242903 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3a8093ae-3854-42fa-945c-5c9832eadf3a service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:28:26.246168 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3a8093ae-3854-42fa-945c-5c9832eadf3a service neutron] RBAC: Authorization granted {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:28:26.256463 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 1139/2309] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Oct 13 04:28:26 2022] GET /identity/v3/auth/tokens => generated 3248 bytes in 213 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 13 04:28:26.257833 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3a8093ae-3854-42fa-945c-5c9832eadf3a service neutron] Authenticating user token {{(pid=81408) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:28:26.265785 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG dbcounter [-] [81408] Writing DB stats keystone:SELECT=98,keystone:DELETE=1,keystone:INSERT=1 {{(pid=81408) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:28:26.270580 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3a8093ae-3854-42fa-945c-5c9832eadf3a service neutron] Validating token access rules against request {{(pid=81408) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:28:26.291772 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fba066e6-28cd-4d33-b97c-ceab9ae78d50 admin admin] Validating token access rules against request {{(pid=81409) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:28:26.302101 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c3fd9a90-a23c-4f2c-abd2-90c0102f7bf0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8ba823f313144da3b683bacef8fd62e0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81409) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:28:26.302817 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c3fd9a90-a23c-4f2c-abd2-90c0102f7bf0 admin admin] REQUEST_METHOD: `POST` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:28:26.303626 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c3fd9a90-a23c-4f2c-abd2-90c0102f7bf0 admin admin] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:28:26.303626 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c3fd9a90-a23c-4f2c-abd2-90c0102f7bf0 admin admin] PATH_INFO: `/v3/users` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:28:26.304809 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c3fd9a90-a23c-4f2c-abd2-90c0102f7bf0 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:28:26.307047 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c3fd9a90-a23c-4f2c-abd2-90c0102f7bf0 admin admin] RBAC: Authorization granted {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:28:26.338494 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 1171/2310] 10.222.0.27 () {60 vars in 1144 bytes} [Thu Oct 13 04:28:26 2022] POST /identity/v3/users => generated 288 bytes in 135 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 13 04:28:26.346688 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c3fd9a90-a23c-4f2c-abd2-90c0102f7bf0 admin admin] Authenticating user token {{(pid=81409) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:28:26.349759 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.fernet_utils [None req-c3fd9a90-a23c-4f2c-abd2-90c0102f7bf0 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81409) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:28:26.355588 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8d6117cc-ab80-4121-8689-a3168cd33918 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8ba823f313144da3b683bacef8fd62e0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81408) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:28:26.356555 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d6117cc-ab80-4121-8689-a3168cd33918 admin admin] REQUEST_METHOD: `POST` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:28:26.357165 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d6117cc-ab80-4121-8689-a3168cd33918 admin admin] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:28:26.357774 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d6117cc-ab80-4121-8689-a3168cd33918 admin admin] PATH_INFO: `/v3/users` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:28:26.358517 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8d6117cc-ab80-4121-8689-a3168cd33918 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:28:26.360556 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8d6117cc-ab80-4121-8689-a3168cd33918 admin admin] RBAC: Authorization granted {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:28:26.367469 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c3fd9a90-a23c-4f2c-abd2-90c0102f7bf0 admin admin] Validating token access rules against request {{(pid=81409) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:28:26.373169 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-db9853f2-0805-4d56-a525-4d0d3b9b4e81 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8ba823f313144da3b683bacef8fd62e0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81409) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:28:26.373862 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db9853f2-0805-4d56-a525-4d0d3b9b4e81 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:28:26.374159 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db9853f2-0805-4d56-a525-4d0d3b9b4e81 admin admin] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:28:26.374444 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db9853f2-0805-4d56-a525-4d0d3b9b4e81 admin admin] PATH_INFO: `/v3/users/1f6cf725908041ccaceb1ffa694ad161` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:28:26.375784 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-db9853f2-0805-4d56-a525-4d0d3b9b4e81 admin admin] RBAC: Authorizing `identity:delete_user(user_id=1f6cf725908041ccaceb1ffa694ad161)` {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:28:26.378450 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-db9853f2-0805-4d56-a525-4d0d3b9b4e81 admin admin] RBAC: Authorization granted {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:28:26.388805 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 1140/2311] 10.222.0.27 () {60 vars in 1144 bytes} [Thu Oct 13 04:28:26 2022] POST /identity/v3/users => generated 300 bytes in 129 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 13 04:28:26.394409 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8d6117cc-ab80-4121-8689-a3168cd33918 admin admin] Authenticating user token {{(pid=81408) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:28:26.406626 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8d6117cc-ab80-4121-8689-a3168cd33918 admin admin] Validating token access rules against request {{(pid=81408) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:28:26.412609 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5e7cad7b-fb9e-499d-ae31-49a2c317d9e8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8ba823f313144da3b683bacef8fd62e0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81408) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:28:26.413453 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e7cad7b-fb9e-499d-ae31-49a2c317d9e8 admin admin] REQUEST_METHOD: `GET` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:28:26.414040 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e7cad7b-fb9e-499d-ae31-49a2c317d9e8 admin admin] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:28:26.414452 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e7cad7b-fb9e-499d-ae31-49a2c317d9e8 admin admin] PATH_INFO: `/v3/roles` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:28:26.415225 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5e7cad7b-fb9e-499d-ae31-49a2c317d9e8 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:28:26.416983 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5e7cad7b-fb9e-499d-ae31-49a2c317d9e8 admin admin] RBAC: Authorization granted {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:28:26.429933 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 1141/2312] 10.222.0.27 () {58 vars in 1122 bytes} [Thu Oct 13 04:28:26 2022] GET /identity/v3/roles => generated 1420 bytes in 43 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 13 04:28:26.431336 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5e7cad7b-fb9e-499d-ae31-49a2c317d9e8 admin admin] Authenticating user token {{(pid=81408) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:28:26.447026 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.notifications [None req-db9853f2-0805-4d56-a525-4d0d3b9b4e81 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '1f6cf725908041ccaceb1ffa694ad161'} {{(pid=81409) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 13 04:28:26.447250 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5e7cad7b-fb9e-499d-ae31-49a2c317d9e8 admin admin] Validating token access rules against request {{(pid=81408) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:28:26.447832 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.notifications [None req-db9853f2-0805-4d56-a525-4d0d3b9b4e81 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '1f6cf725908041ccaceb1ffa694ad161'} {{(pid=81409) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 13 04:28:26.452890 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-67f3f24d-90a5-4eb8-ad20-2ba148ff1564 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8ba823f313144da3b683bacef8fd62e0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81408) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:28:26.453598 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67f3f24d-90a5-4eb8-ad20-2ba148ff1564 admin admin] REQUEST_METHOD: `GET` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:28:26.453949 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67f3f24d-90a5-4eb8-ad20-2ba148ff1564 admin admin] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:28:26.454349 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67f3f24d-90a5-4eb8-ad20-2ba148ff1564 admin admin] PATH_INFO: `/v3/roles` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:28:26.455072 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-67f3f24d-90a5-4eb8-ad20-2ba148ff1564 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:28:26.456593 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.notifications [None req-db9853f2-0805-4d56-a525-4d0d3b9b4e81 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '1f6cf725908041ccaceb1ffa694ad161'} {{(pid=81409) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 13 04:28:26.457121 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-67f3f24d-90a5-4eb8-ad20-2ba148ff1564 admin admin] RBAC: Authorization granted {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:28:26.457604 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.notifications [None req-db9853f2-0805-4d56-a525-4d0d3b9b4e81 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '1f6cf725908041ccaceb1ffa694ad161'} {{(pid=81409) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 13 04:28:26.472463 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 1142/2313] 10.222.0.27 () {58 vars in 1122 bytes} [Thu Oct 13 04:28:26 2022] GET /identity/v3/roles => generated 1420 bytes in 38 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 13 04:28:26.472463 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-67f3f24d-90a5-4eb8-ad20-2ba148ff1564 admin admin] Authenticating user token {{(pid=81408) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:28:26.472463 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.fernet_utils [None req-67f3f24d-90a5-4eb8-ad20-2ba148ff1564 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81408) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:28:26.473823 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.notifications [None req-db9853f2-0805-4d56-a525-4d0d3b9b4e81 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '1f6cf725908041ccaceb1ffa694ad161'} {{(pid=81409) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 13 04:28:26.486732 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-67f3f24d-90a5-4eb8-ad20-2ba148ff1564 admin admin] Validating token access rules against request {{(pid=81408) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:28:26.492550 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8a84b151-7d87-44e1-a48a-238cf6ad739c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8ba823f313144da3b683bacef8fd62e0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81408) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:28:26.493296 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a84b151-7d87-44e1-a48a-238cf6ad739c admin admin] REQUEST_METHOD: `GET` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:28:26.493642 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a84b151-7d87-44e1-a48a-238cf6ad739c admin admin] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:28:26.493977 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a84b151-7d87-44e1-a48a-238cf6ad739c admin admin] PATH_INFO: `/v3/roles` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:28:26.494743 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8a84b151-7d87-44e1-a48a-238cf6ad739c admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:28:26.496453 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8a84b151-7d87-44e1-a48a-238cf6ad739c admin admin] RBAC: Authorization granted {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:28:26.502271 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 1172/2314] 10.222.0.27 () {58 vars in 1224 bytes} [Thu Oct 13 04:28:26 2022] DELETE /identity/v3/users/1f6cf725908041ccaceb1ffa694ad161 => generated 0 bytes in 163 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 13 04:28:26.506233 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-db9853f2-0805-4d56-a525-4d0d3b9b4e81 admin admin] Authenticating user token {{(pid=81409) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:28:26.507942 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.fernet_utils [None req-db9853f2-0805-4d56-a525-4d0d3b9b4e81 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81409) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:28:26.511166 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 1143/2315] 10.222.0.27 () {58 vars in 1122 bytes} [Thu Oct 13 04:28:26 2022] GET /identity/v3/roles => generated 1420 bytes in 41 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 13 04:28:26.515386 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8a84b151-7d87-44e1-a48a-238cf6ad739c admin admin] Authenticating user token {{(pid=81408) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:28:26.528918 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8a84b151-7d87-44e1-a48a-238cf6ad739c admin admin] Validating token access rules against request {{(pid=81408) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:28:26.535869 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG dbcounter [-] [81409] Writing DB stats keystone:SELECT=85,keystone:INSERT=5,keystone:DELETE=10 {{(pid=81409) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:28:26.600063 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-db9853f2-0805-4d56-a525-4d0d3b9b4e81 admin admin] Validating token access rules against request {{(pid=81409) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:28:26.608625 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a268be01-9d1f-407d-9f86-e39a75363e09 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8ba823f313144da3b683bacef8fd62e0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81409) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:28:26.609465 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a268be01-9d1f-407d-9f86-e39a75363e09 admin admin] REQUEST_METHOD: `PUT` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:28:26.609838 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a268be01-9d1f-407d-9f86-e39a75363e09 admin admin] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:28:26.610279 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a268be01-9d1f-407d-9f86-e39a75363e09 admin admin] PATH_INFO: `/v3/projects/60d27631ff674889b323af891d908aae/users/d4aaf64993a44c309689022e70400951/roles/ef55fbd94e1040be83695754075a7ed5` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:28:26.619869 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8b8a14a2-b5c2-4174-abdc-ffbdd94c2cb7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8ba823f313144da3b683bacef8fd62e0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81408) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:28:26.621801 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b8a14a2-b5c2-4174-abdc-ffbdd94c2cb7 admin admin] REQUEST_METHOD: `PUT` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:28:26.622271 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b8a14a2-b5c2-4174-abdc-ffbdd94c2cb7 admin admin] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:28:26.622850 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b8a14a2-b5c2-4174-abdc-ffbdd94c2cb7 admin admin] PATH_INFO: `/v3/projects/4df9b6305d2d47bd98de154e629c596f/users/6382f287013d4cd4b8201ec3a9103d22/roles/cc56c4ae13a7424ea771c15da4b153c1` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:28:26.630842 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a268be01-9d1f-407d-9f86-e39a75363e09 admin admin] RBAC: Authorizing `identity:create_grant(project_id=60d27631ff674889b323af891d908aae, user_id=d4aaf64993a44c309689022e70400951, role_id=ef55fbd94e1040be83695754075a7ed5)` {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:28:26.633943 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a268be01-9d1f-407d-9f86-e39a75363e09 admin admin] RBAC: Authorization granted {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:28:26.647149 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8b8a14a2-b5c2-4174-abdc-ffbdd94c2cb7 admin admin] RBAC: Authorizing `identity:create_grant(project_id=4df9b6305d2d47bd98de154e629c596f, user_id=6382f287013d4cd4b8201ec3a9103d22, role_id=cc56c4ae13a7424ea771c15da4b153c1)` {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:28:26.651010 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8b8a14a2-b5c2-4174-abdc-ffbdd94c2cb7 admin admin] RBAC: Authorization granted {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:28:26.653051 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 1173/2316] 10.222.0.27 () {60 vars in 1483 bytes} [Thu Oct 13 04:28:26 2022] PUT /identity/v3/projects/60d27631ff674889b323af891d908aae/users/d4aaf64993a44c309689022e70400951/roles/ef55fbd94e1040be83695754075a7ed5 => generated 0 bytes in 150 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 13 04:28:26.657307 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a268be01-9d1f-407d-9f86-e39a75363e09 admin admin] Authenticating user token {{(pid=81409) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:28:26.674514 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 1144/2317] 10.222.0.27 () {60 vars in 1483 bytes} [Thu Oct 13 04:28:26 2022] PUT /identity/v3/projects/4df9b6305d2d47bd98de154e629c596f/users/6382f287013d4cd4b8201ec3a9103d22/roles/cc56c4ae13a7424ea771c15da4b153c1 => generated 0 bytes in 156 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 13 04:28:26.682699 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3a15e51-3003-4741-8cf6-a4d898b1b2e2 None None] REQUEST_METHOD: `POST` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:28:26.683206 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3a15e51-3003-4741-8cf6-a4d898b1b2e2 None None] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:28:26.683631 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3a15e51-3003-4741-8cf6-a4d898b1b2e2 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:28:26.719100 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.auth.core [None req-e3a15e51-3003-4741-8cf6-a4d898b1b2e2 None None] MFA Rules not processed for user `6382f287013d4cd4b8201ec3a9103d22`. Rule list: `[]` (Enabled: `True`). {{(pid=81408) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 13 04:28:26.720332 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.fernet_utils [None req-e3a15e51-3003-4741-8cf6-a4d898b1b2e2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81408) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:28:26.762595 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a268be01-9d1f-407d-9f86-e39a75363e09 admin admin] Validating token access rules against request {{(pid=81409) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:28:26.769393 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG dbcounter [-] [81408] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=81408) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:28:26.821702 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 1145/2318] 10.222.0.27 () {58 vars in 958 bytes} [Thu Oct 13 04:28:26 2022] POST /identity/v3/auth/tokens => generated 3252 bytes in 140 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 13 04:28:26.825882 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e3a15e51-3003-4741-8cf6-a4d898b1b2e2 None None] Authenticating user token {{(pid=81408) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:28:26.827444 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.fernet_utils [None req-e3a15e51-3003-4741-8cf6-a4d898b1b2e2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81408) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:28:26.836310 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fdd73f9a-6e14-4d9a-a3b7-f53565d2e2cd admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8ba823f313144da3b683bacef8fd62e0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81409) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:28:26.837230 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fdd73f9a-6e14-4d9a-a3b7-f53565d2e2cd admin admin] REQUEST_METHOD: `GET` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:28:26.837681 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fdd73f9a-6e14-4d9a-a3b7-f53565d2e2cd admin admin] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:28:26.837958 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fdd73f9a-6e14-4d9a-a3b7-f53565d2e2cd admin admin] PATH_INFO: `/v3/roles` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:28:26.839152 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fdd73f9a-6e14-4d9a-a3b7-f53565d2e2cd admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:28:26.841245 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fdd73f9a-6e14-4d9a-a3b7-f53565d2e2cd admin admin] RBAC: Authorization granted {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:28:26.856981 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 1174/2319] 10.222.0.27 () {58 vars in 1122 bytes} [Thu Oct 13 04:28:26 2022] GET /identity/v3/roles => generated 1420 bytes in 197 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Oct 13 04:28:26.856981 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fdd73f9a-6e14-4d9a-a3b7-f53565d2e2cd admin admin] Authenticating user token {{(pid=81409) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:28:26.858475 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.fernet_utils [None req-fdd73f9a-6e14-4d9a-a3b7-f53565d2e2cd admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81409) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:28:26.907771 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e3a15e51-3003-4741-8cf6-a4d898b1b2e2 None None] Validating token access rules against request {{(pid=81408) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:28:26.912387 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bd400bd8-7556-40c1-a9f4-c3bf53d53a3c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8ba823f313144da3b683bacef8fd62e0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81408) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:28:26.913014 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd400bd8-7556-40c1-a9f4-c3bf53d53a3c admin admin] REQUEST_METHOD: `DELETE` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:28:26.913324 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd400bd8-7556-40c1-a9f4-c3bf53d53a3c admin admin] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:28:26.913723 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd400bd8-7556-40c1-a9f4-c3bf53d53a3c admin admin] PATH_INFO: `/v3/projects/73d6c8ffc0354e9d889b1abc471602ac` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:28:26.915302 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bd400bd8-7556-40c1-a9f4-c3bf53d53a3c admin admin] RBAC: Authorizing `identity:delete_project(project_id=73d6c8ffc0354e9d889b1abc471602ac)` {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:28:26.916916 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG dbcounter [-] [81409] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81409) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:28:26.917937 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bd400bd8-7556-40c1-a9f4-c3bf53d53a3c admin admin] RBAC: Authorization granted {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:28:26.933271 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fdd73f9a-6e14-4d9a-a3b7-f53565d2e2cd admin admin] Validating token access rules against request {{(pid=81409) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:28:26.936970 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-de9ef6f9-0653-4859-a1a7-af32017e160b service glance] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a6700f49b86543f6a5b92cad8faaba42', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dcec0431fb6849499d6aec35a234daa6', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81409) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:28:26.937736 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de9ef6f9-0653-4859-a1a7-af32017e160b service glance] REQUEST_METHOD: `GET` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:28:26.938020 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de9ef6f9-0653-4859-a1a7-af32017e160b service glance] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:28:26.938376 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de9ef6f9-0653-4859-a1a7-af32017e160b service glance] PATH_INFO: `/v3/auth/tokens` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:28:26.950012 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-de9ef6f9-0653-4859-a1a7-af32017e160b service glance] RBAC: Authorizing `identity:validate_token()` {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:28:26.951499 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-de9ef6f9-0653-4859-a1a7-af32017e160b service glance] RBAC: Authorization granted {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:28:26.961761 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 1175/2320] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Oct 13 04:28:26 2022] GET /identity/v3/auth/tokens => generated 3252 bytes in 108 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 13 04:28:26.962603 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-de9ef6f9-0653-4859-a1a7-af32017e160b service glance] Authenticating user token {{(pid=81409) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:28:26.973758 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-de9ef6f9-0653-4859-a1a7-af32017e160b service glance] Validating token access rules against request {{(pid=81409) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:28:26.975567 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.notifications [None req-bd400bd8-7556-40c1-a9f4-c3bf53d53a3c admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '73d6c8ffc0354e9d889b1abc471602ac', 'request_id': 'req-bd400bd8-7556-40c1-a9f4-c3bf53d53a3c'} {{(pid=81408) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 13 04:28:26.990388 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.notifications [None req-bd400bd8-7556-40c1-a9f4-c3bf53d53a3c admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '73d6c8ffc0354e9d889b1abc471602ac', 'request_id': 'req-bd400bd8-7556-40c1-a9f4-c3bf53d53a3c'} {{(pid=81408) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 13 04:28:26.998369 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.notifications [None req-bd400bd8-7556-40c1-a9f4-c3bf53d53a3c admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '73d6c8ffc0354e9d889b1abc471602ac', 'request_id': 'req-bd400bd8-7556-40c1-a9f4-c3bf53d53a3c'} {{(pid=81408) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 13 04:28:27.012716 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.notifications [None req-bd400bd8-7556-40c1-a9f4-c3bf53d53a3c admin admin] The token cache is being invalidate because project 73d6c8ffc0354e9d889b1abc471602ac was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81408) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 13 04:28:27.013188 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.notifications [None req-bd400bd8-7556-40c1-a9f4-c3bf53d53a3c admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81408) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 13 04:28:27.014107 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.notifications [None req-bd400bd8-7556-40c1-a9f4-c3bf53d53a3c admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81408) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 13 04:28:27.016165 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 1146/2321] 10.222.0.27 () {58 vars in 1233 bytes} [Thu Oct 13 04:28:26 2022] DELETE /identity/v3/projects/73d6c8ffc0354e9d889b1abc471602ac => generated 0 bytes in 194 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 13 04:28:27.021047 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8248a3e-f4e8-414e-8724-09b1cf66abde None None] REQUEST_METHOD: `POST` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:28:27.021417 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8248a3e-f4e8-414e-8724-09b1cf66abde None None] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:28:27.021790 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8248a3e-f4e8-414e-8724-09b1cf66abde None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:28:27.052787 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.auth.core [None req-f8248a3e-f4e8-414e-8724-09b1cf66abde None None] MFA Rules not processed for user `0e2d979563d6421995451a0e86bf3022`. Rule list: `[]` (Enabled: `True`). {{(pid=81408) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 13 04:28:27.054734 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.fernet_utils [None req-f8248a3e-f4e8-414e-8724-09b1cf66abde None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81408) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:28:27.125252 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-95613797-dfde-40bf-a39a-7e537deeeccb admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8ba823f313144da3b683bacef8fd62e0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81409) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:28:27.125938 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95613797-dfde-40bf-a39a-7e537deeeccb admin admin] REQUEST_METHOD: `PUT` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:28:27.126236 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95613797-dfde-40bf-a39a-7e537deeeccb admin admin] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:28:27.126613 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95613797-dfde-40bf-a39a-7e537deeeccb admin admin] PATH_INFO: `/v3/projects/60d27631ff674889b323af891d908aae/users/d4aaf64993a44c309689022e70400951/roles/ef55fbd94e1040be83695754075a7ed5` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:28:27.128585 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-95613797-dfde-40bf-a39a-7e537deeeccb admin admin] RBAC: Authorizing `identity:create_grant(project_id=60d27631ff674889b323af891d908aae, user_id=d4aaf64993a44c309689022e70400951, role_id=ef55fbd94e1040be83695754075a7ed5)` {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:28:27.131461 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-95613797-dfde-40bf-a39a-7e537deeeccb admin admin] RBAC: Authorization granted {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:28:27.143438 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 1176/2322] 10.222.0.27 () {60 vars in 1483 bytes} [Thu Oct 13 04:28:26 2022] PUT /identity/v3/projects/60d27631ff674889b323af891d908aae/users/d4aaf64993a44c309689022e70400951/roles/ef55fbd94e1040be83695754075a7ed5 => generated 0 bytes in 181 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 13 04:28:27.143873 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 1147/2323] 10.222.0.27 () {58 vars in 958 bytes} [Thu Oct 13 04:28:27 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 124 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 13 04:28:27.154141 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d466d8d3-2cdb-4f5c-af81-ef5b74eea1a4 None None] REQUEST_METHOD: `POST` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:28:27.154473 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d466d8d3-2cdb-4f5c-af81-ef5b74eea1a4 None None] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:28:27.154743 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d466d8d3-2cdb-4f5c-af81-ef5b74eea1a4 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:28:27.155675 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-227eee67-a538-45a4-a084-1dda18220994 None None] REQUEST_METHOD: `POST` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:28:27.158443 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-227eee67-a538-45a4-a084-1dda18220994 None None] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:28:27.158757 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-227eee67-a538-45a4-a084-1dda18220994 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:28:27.179150 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.auth.core [None req-d466d8d3-2cdb-4f5c-af81-ef5b74eea1a4 None None] MFA Rules not processed for user `0e2d979563d6421995451a0e86bf3022`. Rule list: `[]` (Enabled: `True`). {{(pid=81409) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 13 04:28:27.180532 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.fernet_utils [None req-d466d8d3-2cdb-4f5c-af81-ef5b74eea1a4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81409) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:28:27.182542 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG dbcounter [-] [81408] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=81408) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:28:27.232563 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.auth.core [None req-227eee67-a538-45a4-a084-1dda18220994 None None] MFA Rules not processed for user `d4aaf64993a44c309689022e70400951`. Rule list: `[]` (Enabled: `True`). {{(pid=81408) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 13 04:28:27.234035 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.fernet_utils [None req-227eee67-a538-45a4-a084-1dda18220994 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81408) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:28:27.254760 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 1177/2324] 10.222.0.27 () {58 vars in 958 bytes} [Thu Oct 13 04:28:27 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 111 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 13 04:28:27.259250 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d466d8d3-2cdb-4f5c-af81-ef5b74eea1a4 None None] Authenticating user token {{(pid=81409) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:28:27.270060 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d466d8d3-2cdb-4f5c-af81-ef5b74eea1a4 None None] Validating token access rules against request {{(pid=81409) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:28:27.275784 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-29bcfdd3-7610-4508-ba11-c830e4e089bd admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8ba823f313144da3b683bacef8fd62e0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81409) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:28:27.276878 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29bcfdd3-7610-4508-ba11-c830e4e089bd admin admin] REQUEST_METHOD: `GET` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:28:27.277168 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29bcfdd3-7610-4508-ba11-c830e4e089bd admin admin] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:28:27.277444 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29bcfdd3-7610-4508-ba11-c830e4e089bd admin admin] PATH_INFO: `/v3/domains` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:28:27.278110 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-29bcfdd3-7610-4508-ba11-c830e4e089bd admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81409) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 13 04:28:27.278600 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-29bcfdd3-7610-4508-ba11-c830e4e089bd admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:28:27.280261 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-29bcfdd3-7610-4508-ba11-c830e4e089bd admin admin] RBAC: Authorization granted {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:28:27.298299 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 1178/2325] 10.222.0.27 () {58 vars in 1153 bytes} [Thu Oct 13 04:28:27 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 39 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 13 04:28:27.306111 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-29bcfdd3-7610-4508-ba11-c830e4e089bd admin admin] Authenticating user token {{(pid=81409) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:28:27.320402 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-29bcfdd3-7610-4508-ba11-c830e4e089bd admin admin] Validating token access rules against request {{(pid=81409) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:28:27.326985 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c5b7e825-a3e2-4619-b277-08eb47e34be6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8ba823f313144da3b683bacef8fd62e0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81409) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:28:27.327765 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5b7e825-a3e2-4619-b277-08eb47e34be6 admin admin] REQUEST_METHOD: `POST` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:28:27.328047 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5b7e825-a3e2-4619-b277-08eb47e34be6 admin admin] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:28:27.328357 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5b7e825-a3e2-4619-b277-08eb47e34be6 admin admin] PATH_INFO: `/v3/projects` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:28:27.329122 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c5b7e825-a3e2-4619-b277-08eb47e34be6 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:28:27.331778 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c5b7e825-a3e2-4619-b277-08eb47e34be6 admin admin] RBAC: Authorization granted {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:28:27.386167 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 1179/2326] 10.222.0.27 () {60 vars in 1153 bytes} [Thu Oct 13 04:28:27 2022] POST /identity/v3/projects => generated 371 bytes in 82 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 13 04:28:27.387056 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c5b7e825-a3e2-4619-b277-08eb47e34be6 admin admin] Authenticating user token {{(pid=81409) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:28:27.429857 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 1148/2327] 10.222.0.27 () {58 vars in 958 bytes} [Thu Oct 13 04:28:27 2022] POST /identity/v3/auth/tokens => generated 3331 bytes in 276 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 13 04:28:27.440553 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-227eee67-a538-45a4-a084-1dda18220994 None None] Authenticating user token {{(pid=81408) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:28:27.442498 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.fernet_utils [None req-227eee67-a538-45a4-a084-1dda18220994 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81408) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:28:27.471726 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG dbcounter [-] [81409] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=81409) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:28:27.514658 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c5b7e825-a3e2-4619-b277-08eb47e34be6 admin admin] Validating token access rules against request {{(pid=81409) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:28:27.520372 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d1c348c9-d427-42ed-a6ad-8ea6cc75f64f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8ba823f313144da3b683bacef8fd62e0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81409) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:28:27.521018 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1c348c9-d427-42ed-a6ad-8ea6cc75f64f admin admin] REQUEST_METHOD: `POST` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:28:27.521293 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1c348c9-d427-42ed-a6ad-8ea6cc75f64f admin admin] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:28:27.521609 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1c348c9-d427-42ed-a6ad-8ea6cc75f64f admin admin] PATH_INFO: `/v3/users` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:28:27.522270 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d1c348c9-d427-42ed-a6ad-8ea6cc75f64f admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:28:27.525206 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d1c348c9-d427-42ed-a6ad-8ea6cc75f64f admin admin] RBAC: Authorization granted {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:28:27.558270 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 1180/2328] 10.222.0.27 () {60 vars in 1144 bytes} [Thu Oct 13 04:28:27 2022] POST /identity/v3/users => generated 286 bytes in 172 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 13 04:28:27.562485 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d1c348c9-d427-42ed-a6ad-8ea6cc75f64f admin admin] Authenticating user token {{(pid=81409) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:28:27.564346 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-227eee67-a538-45a4-a084-1dda18220994 None None] Validating token access rules against request {{(pid=81408) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:28:27.577424 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d1c348c9-d427-42ed-a6ad-8ea6cc75f64f admin admin] Validating token access rules against request {{(pid=81409) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:28:27.583131 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9d57d6b0-bc36-4a07-8a9a-a9f6a763c8da admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8ba823f313144da3b683bacef8fd62e0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81409) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:28:27.583597 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d57d6b0-bc36-4a07-8a9a-a9f6a763c8da admin admin] REQUEST_METHOD: `GET` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:28:27.583873 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d57d6b0-bc36-4a07-8a9a-a9f6a763c8da admin admin] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:28:27.584138 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d57d6b0-bc36-4a07-8a9a-a9f6a763c8da admin admin] PATH_INFO: `/v3/roles` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:28:27.584786 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9d57d6b0-bc36-4a07-8a9a-a9f6a763c8da admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:28:27.586924 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9d57d6b0-bc36-4a07-8a9a-a9f6a763c8da admin admin] RBAC: Authorization granted {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:28:27.604725 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 1181/2329] 10.222.0.27 () {58 vars in 1122 bytes} [Thu Oct 13 04:28:27 2022] GET /identity/v3/roles => generated 1420 bytes in 42 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 13 04:28:27.606452 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9d57d6b0-bc36-4a07-8a9a-a9f6a763c8da admin admin] Authenticating user token {{(pid=81409) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:28:27.622595 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9d57d6b0-bc36-4a07-8a9a-a9f6a763c8da admin admin] Validating token access rules against request {{(pid=81409) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:28:27.629161 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7b1a5f4c-a383-42e2-9542-5502a9a63742 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8ba823f313144da3b683bacef8fd62e0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81409) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:28:27.629999 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b1a5f4c-a383-42e2-9542-5502a9a63742 admin admin] REQUEST_METHOD: `GET` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:28:27.630465 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b1a5f4c-a383-42e2-9542-5502a9a63742 admin admin] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:28:27.630902 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b1a5f4c-a383-42e2-9542-5502a9a63742 admin admin] PATH_INFO: `/v3/roles` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:28:27.631777 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7b1a5f4c-a383-42e2-9542-5502a9a63742 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:28:27.635492 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7b1a5f4c-a383-42e2-9542-5502a9a63742 admin admin] RBAC: Authorization granted {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:28:27.664532 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cac9e463-99e4-4810-ab43-8d4dafebdb64 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6141b8a6c3d54e179d6879bb019f4d12', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dcec0431fb6849499d6aec35a234daa6', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81408) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:28:27.665179 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cac9e463-99e4-4810-ab43-8d4dafebdb64 service cinder] REQUEST_METHOD: `GET` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:28:27.665462 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cac9e463-99e4-4810-ab43-8d4dafebdb64 service cinder] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:28:27.665737 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cac9e463-99e4-4810-ab43-8d4dafebdb64 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:28:27.670447 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 1182/2330] 10.222.0.27 () {58 vars in 1122 bytes} [Thu Oct 13 04:28:27 2022] GET /identity/v3/roles => generated 1420 bytes in 65 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 13 04:28:27.671880 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7b1a5f4c-a383-42e2-9542-5502a9a63742 admin admin] Authenticating user token {{(pid=81409) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:28:27.688949 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7b1a5f4c-a383-42e2-9542-5502a9a63742 admin admin] Validating token access rules against request {{(pid=81409) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:28:27.695135 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1ddea92e-4ccd-4dac-b22a-539d61f84c35 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8ba823f313144da3b683bacef8fd62e0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81409) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:28:27.696221 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ddea92e-4ccd-4dac-b22a-539d61f84c35 admin admin] REQUEST_METHOD: `PUT` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:28:27.696690 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ddea92e-4ccd-4dac-b22a-539d61f84c35 admin admin] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:28:27.697797 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ddea92e-4ccd-4dac-b22a-539d61f84c35 admin admin] PATH_INFO: `/v3/projects/728402e7d8a6419a8ebce7ccdbb965ef/users/778eb55039f94988853ee1f24b64336c/roles/cc56c4ae13a7424ea771c15da4b153c1` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:28:27.733016 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1ddea92e-4ccd-4dac-b22a-539d61f84c35 admin admin] RBAC: Authorizing `identity:create_grant(project_id=728402e7d8a6419a8ebce7ccdbb965ef, user_id=778eb55039f94988853ee1f24b64336c, role_id=cc56c4ae13a7424ea771c15da4b153c1)` {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:28:27.735919 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1ddea92e-4ccd-4dac-b22a-539d61f84c35 admin admin] RBAC: Authorization granted {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:28:27.751477 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG dbcounter [-] [81408] Writing DB stats keystone:SELECT=100 {{(pid=81408) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:28:27.758859 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 1183/2331] 10.222.0.27 () {60 vars in 1483 bytes} [Thu Oct 13 04:28:27 2022] PUT /identity/v3/projects/728402e7d8a6419a8ebce7ccdbb965ef/users/778eb55039f94988853ee1f24b64336c/roles/cc56c4ae13a7424ea771c15da4b153c1 => generated 0 bytes in 88 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 13 04:28:27.770228 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cac9e463-99e4-4810-ab43-8d4dafebdb64 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:28:27.774628 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cac9e463-99e4-4810-ab43-8d4dafebdb64 service cinder] RBAC: Authorization granted {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:28:27.777452 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc1bfa21-15be-4a61-998a-f2e2c47218b7 None None] REQUEST_METHOD: `POST` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:28:27.777739 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc1bfa21-15be-4a61-998a-f2e2c47218b7 None None] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:28:27.778011 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc1bfa21-15be-4a61-998a-f2e2c47218b7 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:28:27.790393 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 1149/2332] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Oct 13 04:28:27 2022] GET /identity/v3/auth/tokens => generated 3331 bytes in 350 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 13 04:28:27.841134 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.auth.core [None req-dc1bfa21-15be-4a61-998a-f2e2c47218b7 None None] MFA Rules not processed for user `778eb55039f94988853ee1f24b64336c`. Rule list: `[]` (Enabled: `True`). {{(pid=81409) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 13 04:28:27.842652 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.fernet_utils [None req-dc1bfa21-15be-4a61-998a-f2e2c47218b7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81409) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:28:27.949242 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 1184/2333] 10.222.0.27 () {58 vars in 958 bytes} [Thu Oct 13 04:28:27 2022] POST /identity/v3/auth/tokens => generated 3248 bytes in 175 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 13 04:28:27.950614 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cac9e463-99e4-4810-ab43-8d4dafebdb64 service cinder] Authenticating user token {{(pid=81408) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:28:27.964008 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cac9e463-99e4-4810-ab43-8d4dafebdb64 service cinder] Validating token access rules against request {{(pid=81408) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:28:28.078999 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d90f7a5f-9bec-4923-98eb-0ddf38533b01 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8ba823f313144da3b683bacef8fd62e0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81408) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:28:28.079838 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d90f7a5f-9bec-4923-98eb-0ddf38533b01 admin admin] REQUEST_METHOD: `POST` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:28:28.079941 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d90f7a5f-9bec-4923-98eb-0ddf38533b01 admin admin] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:28:28.080229 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d90f7a5f-9bec-4923-98eb-0ddf38533b01 admin admin] PATH_INFO: `/v3/projects` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:28:28.081147 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d90f7a5f-9bec-4923-98eb-0ddf38533b01 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:28:28.083274 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d90f7a5f-9bec-4923-98eb-0ddf38533b01 admin admin] RBAC: Authorization granted {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:28:28.141847 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 1150/2334] 10.222.0.27 () {60 vars in 1153 bytes} [Thu Oct 13 04:28:27 2022] POST /identity/v3/projects => generated 371 bytes in 192 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 13 04:28:28.146806 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dc1bfa21-15be-4a61-998a-f2e2c47218b7 None None] Authenticating user token {{(pid=81409) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:28:28.165103 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dc1bfa21-15be-4a61-998a-f2e2c47218b7 None None] Validating token access rules against request {{(pid=81409) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:28:28.252638 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eadd415c-9ef2-43f4-adcc-319837b0b329 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8ba823f313144da3b683bacef8fd62e0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81409) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:28:28.253501 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eadd415c-9ef2-43f4-adcc-319837b0b329 admin admin] REQUEST_METHOD: `POST` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:28:28.253875 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eadd415c-9ef2-43f4-adcc-319837b0b329 admin admin] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:28:28.254528 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eadd415c-9ef2-43f4-adcc-319837b0b329 admin admin] PATH_INFO: `/v3/users` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:28:28.255290 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eadd415c-9ef2-43f4-adcc-319837b0b329 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:28:28.257465 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eadd415c-9ef2-43f4-adcc-319837b0b329 admin admin] RBAC: Authorization granted {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:28:28.292006 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 1185/2335] 10.222.0.27 () {60 vars in 1144 bytes} [Thu Oct 13 04:28:28 2022] POST /identity/v3/users => generated 292 bytes in 144 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 13 04:28:28.292976 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d90f7a5f-9bec-4923-98eb-0ddf38533b01 admin admin] Authenticating user token {{(pid=81408) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:28:28.308407 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d90f7a5f-9bec-4923-98eb-0ddf38533b01 admin admin] Validating token access rules against request {{(pid=81408) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:28:28.313496 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-67883e17-5e07-4f6f-8a7d-82afe6ffa580 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8ba823f313144da3b683bacef8fd62e0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81408) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:28:28.314120 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67883e17-5e07-4f6f-8a7d-82afe6ffa580 admin admin] REQUEST_METHOD: `GET` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:28:28.314649 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67883e17-5e07-4f6f-8a7d-82afe6ffa580 admin admin] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:28:28.317582 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67883e17-5e07-4f6f-8a7d-82afe6ffa580 admin admin] PATH_INFO: `/v3/roles` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:28:28.317582 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-67883e17-5e07-4f6f-8a7d-82afe6ffa580 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:28:28.317582 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-67883e17-5e07-4f6f-8a7d-82afe6ffa580 admin admin] RBAC: Authorization granted {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:28:28.332322 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 1151/2336] 10.222.0.27 () {58 vars in 1122 bytes} [Thu Oct 13 04:28:28 2022] GET /identity/v3/roles => generated 1420 bytes in 40 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Oct 13 04:28:28.337590 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eadd415c-9ef2-43f4-adcc-319837b0b329 admin admin] Authenticating user token {{(pid=81409) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:28:28.349567 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eadd415c-9ef2-43f4-adcc-319837b0b329 admin admin] Validating token access rules against request {{(pid=81409) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:28:28.363380 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-af1376c2-6e28-4fc9-9944-00b4953057fc admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8ba823f313144da3b683bacef8fd62e0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81409) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:28:28.364068 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af1376c2-6e28-4fc9-9944-00b4953057fc admin admin] REQUEST_METHOD: `PUT` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:28:28.364391 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af1376c2-6e28-4fc9-9944-00b4953057fc admin admin] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:28:28.364667 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af1376c2-6e28-4fc9-9944-00b4953057fc admin admin] PATH_INFO: `/v3/projects/2b57f5dd803e436fbd1e4c597c0a5605/users/186fed5b8ca64f2cb53c2dde157dfae0/roles/ef55fbd94e1040be83695754075a7ed5` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:28:28.369741 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG dbcounter [-] [81409] Writing DB stats keystone:SELECT=93,keystone:INSERT=7 {{(pid=81409) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:28:28.392150 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-af1376c2-6e28-4fc9-9944-00b4953057fc admin admin] RBAC: Authorizing `identity:create_grant(project_id=2b57f5dd803e436fbd1e4c597c0a5605, user_id=186fed5b8ca64f2cb53c2dde157dfae0, role_id=ef55fbd94e1040be83695754075a7ed5)` {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:28:28.396085 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-af1376c2-6e28-4fc9-9944-00b4953057fc admin admin] RBAC: Authorization granted {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:28:28.425763 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 1186/2337] 10.222.0.27 () {60 vars in 1483 bytes} [Thu Oct 13 04:28:28 2022] PUT /identity/v3/projects/2b57f5dd803e436fbd1e4c597c0a5605/users/186fed5b8ca64f2cb53c2dde157dfae0/roles/ef55fbd94e1040be83695754075a7ed5 => generated 0 bytes in 89 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 13 04:28:28.436634 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-67883e17-5e07-4f6f-8a7d-82afe6ffa580 admin admin] Authenticating user token {{(pid=81408) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:28:28.448498 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-67883e17-5e07-4f6f-8a7d-82afe6ffa580 admin admin] Validating token access rules against request {{(pid=81408) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:28:28.522652 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c3c610b5-1f81-4889-9f8f-d266ece5d648 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8ba823f313144da3b683bacef8fd62e0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81408) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:28:28.523782 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c3c610b5-1f81-4889-9f8f-d266ece5d648 admin admin] REQUEST_METHOD: `GET` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:28:28.524192 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c3c610b5-1f81-4889-9f8f-d266ece5d648 admin admin] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:28:28.524468 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c3c610b5-1f81-4889-9f8f-d266ece5d648 admin admin] PATH_INFO: `/v3/roles` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:28:28.525113 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c3c610b5-1f81-4889-9f8f-d266ece5d648 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:28:28.526809 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c3c610b5-1f81-4889-9f8f-d266ece5d648 admin admin] RBAC: Authorization granted {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:28:28.538902 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 1152/2338] 10.222.0.27 () {58 vars in 1122 bytes} [Thu Oct 13 04:28:28 2022] GET /identity/v3/roles => generated 1420 bytes in 103 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 13 04:28:28.543710 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-af1376c2-6e28-4fc9-9944-00b4953057fc admin admin] Authenticating user token {{(pid=81409) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:28:28.554707 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-af1376c2-6e28-4fc9-9944-00b4953057fc admin admin] Validating token access rules against request {{(pid=81409) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:28:28.559965 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1fb3854d-be0f-4c79-97c6-8a718ef2ceeb admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8ba823f313144da3b683bacef8fd62e0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81409) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:28:28.560785 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1fb3854d-be0f-4c79-97c6-8a718ef2ceeb admin admin] REQUEST_METHOD: `PUT` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:28:28.561065 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1fb3854d-be0f-4c79-97c6-8a718ef2ceeb admin admin] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:28:28.561342 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1fb3854d-be0f-4c79-97c6-8a718ef2ceeb admin admin] PATH_INFO: `/v3/projects/2b57f5dd803e436fbd1e4c597c0a5605/users/186fed5b8ca64f2cb53c2dde157dfae0/roles/ef55fbd94e1040be83695754075a7ed5` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:28:28.572168 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1fb3854d-be0f-4c79-97c6-8a718ef2ceeb admin admin] RBAC: Authorizing `identity:create_grant(project_id=2b57f5dd803e436fbd1e4c597c0a5605, user_id=186fed5b8ca64f2cb53c2dde157dfae0, role_id=ef55fbd94e1040be83695754075a7ed5)` {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:28:28.572168 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1fb3854d-be0f-4c79-97c6-8a718ef2ceeb admin admin] RBAC: Authorization granted {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:28:28.586241 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 1187/2339] 10.222.0.27 () {60 vars in 1483 bytes} [Thu Oct 13 04:28:28 2022] PUT /identity/v3/projects/2b57f5dd803e436fbd1e4c597c0a5605/users/186fed5b8ca64f2cb53c2dde157dfae0/roles/ef55fbd94e1040be83695754075a7ed5 => generated 0 bytes in 39 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 13 04:28:28.592566 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d7580b2-69ed-4953-b52e-6f6f6d2348f6 None None] REQUEST_METHOD: `POST` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:28:28.592859 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d7580b2-69ed-4953-b52e-6f6f6d2348f6 None None] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:28:28.593135 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d7580b2-69ed-4953-b52e-6f6f6d2348f6 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:28:28.640613 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1fb3854d-be0f-4c79-97c6-8a718ef2ceeb admin admin] Authenticating user token {{(pid=81409) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:28:28.642473 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.fernet_utils [None req-1fb3854d-be0f-4c79-97c6-8a718ef2ceeb admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81409) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:28:28.652172 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.auth.core [None req-0d7580b2-69ed-4953-b52e-6f6f6d2348f6 None None] MFA Rules not processed for user `186fed5b8ca64f2cb53c2dde157dfae0`. Rule list: `[]` (Enabled: `True`). {{(pid=81408) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 13 04:28:28.653378 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.fernet_utils [None req-0d7580b2-69ed-4953-b52e-6f6f6d2348f6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81408) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:28:28.727773 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1fb3854d-be0f-4c79-97c6-8a718ef2ceeb admin admin] Validating token access rules against request {{(pid=81409) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:28:28.732327 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f37ce300-e6fb-4e08-a5a1-cb0ab57c414b service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6141b8a6c3d54e179d6879bb019f4d12', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dcec0431fb6849499d6aec35a234daa6', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81409) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:28:28.732927 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f37ce300-e6fb-4e08-a5a1-cb0ab57c414b service cinder] REQUEST_METHOD: `GET` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:28:28.733223 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f37ce300-e6fb-4e08-a5a1-cb0ab57c414b service cinder] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:28:28.733475 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f37ce300-e6fb-4e08-a5a1-cb0ab57c414b service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:28:28.735756 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.fernet_utils [None req-f37ce300-e6fb-4e08-a5a1-cb0ab57c414b service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81409) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:28:28.736217 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG dbcounter [-] [81408] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81408) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:28:28.758696 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 1153/2340] 10.222.0.27 () {58 vars in 958 bytes} [Thu Oct 13 04:28:28 2022] POST /identity/v3/auth/tokens => generated 3315 bytes in 167 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 13 04:28:28.770475 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0d7580b2-69ed-4953-b52e-6f6f6d2348f6 None None] Authenticating user token {{(pid=81408) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:28:28.784162 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0d7580b2-69ed-4953-b52e-6f6f6d2348f6 None None] Validating token access rules against request {{(pid=81408) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:28:28.788065 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0d2380e9-89d9-4cea-8a7b-e4c58e20988f service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6141b8a6c3d54e179d6879bb019f4d12', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dcec0431fb6849499d6aec35a234daa6', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81408) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:28:28.788894 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d2380e9-89d9-4cea-8a7b-e4c58e20988f service cinder] REQUEST_METHOD: `GET` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:28:28.789174 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d2380e9-89d9-4cea-8a7b-e4c58e20988f service cinder] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:28:28.789495 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d2380e9-89d9-4cea-8a7b-e4c58e20988f service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:28:28.799786 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0d2380e9-89d9-4cea-8a7b-e4c58e20988f service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:28:28.801253 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0d2380e9-89d9-4cea-8a7b-e4c58e20988f service cinder] RBAC: Authorization granted {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:28:28.812866 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 1154/2341] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Oct 13 04:28:28 2022] GET /identity/v3/auth/tokens => generated 3315 bytes in 43 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 13 04:28:28.814063 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f37ce300-e6fb-4e08-a5a1-cb0ab57c414b service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:28:28.816415 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f37ce300-e6fb-4e08-a5a1-cb0ab57c414b service cinder] RBAC: Authorization granted {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:28:28.827617 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 1188/2342] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Oct 13 04:28:28 2022] GET /identity/v3/auth/tokens => generated 3262 bytes in 188 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Oct 13 04:28:28.912805 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0d2380e9-89d9-4cea-8a7b-e4c58e20988f service cinder] Authenticating user token {{(pid=81408) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:28:28.922257 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0d2380e9-89d9-4cea-8a7b-e4c58e20988f service cinder] Validating token access rules against request {{(pid=81408) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:28:28.926503 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f604069c-554e-4715-b1b4-7ff5af649c10 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6141b8a6c3d54e179d6879bb019f4d12', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dcec0431fb6849499d6aec35a234daa6', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81408) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:28:28.927058 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f604069c-554e-4715-b1b4-7ff5af649c10 service cinder] REQUEST_METHOD: `GET` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:28:28.927345 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f604069c-554e-4715-b1b4-7ff5af649c10 service cinder] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:28:28.927691 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f604069c-554e-4715-b1b4-7ff5af649c10 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:28:28.998603 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f604069c-554e-4715-b1b4-7ff5af649c10 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:28:29.000215 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f604069c-554e-4715-b1b4-7ff5af649c10 service cinder] RBAC: Authorization granted {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:28:29.010519 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 1155/2343] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Oct 13 04:28:28 2022] GET /identity/v3/auth/tokens => generated 3248 bytes in 98 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 13 04:28:32.257309 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f37ce300-e6fb-4e08-a5a1-cb0ab57c414b service cinder] Authenticating user token {{(pid=81409) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:28:32.258780 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.fernet_utils [None req-f37ce300-e6fb-4e08-a5a1-cb0ab57c414b service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81409) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:28:32.315683 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f37ce300-e6fb-4e08-a5a1-cb0ab57c414b service cinder] Validating token access rules against request {{(pid=81409) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:28:32.319304 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-44ad939a-564c-4d56-9793-5f3054a16afd service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd8812c3974f84871bb2d761cf0016257', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dcec0431fb6849499d6aec35a234daa6', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81409) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:28:32.319906 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44ad939a-564c-4d56-9793-5f3054a16afd service neutron] REQUEST_METHOD: `GET` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:28:32.320183 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44ad939a-564c-4d56-9793-5f3054a16afd service neutron] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:28:32.320453 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44ad939a-564c-4d56-9793-5f3054a16afd service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:28:32.322169 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.fernet_utils [None req-44ad939a-564c-4d56-9793-5f3054a16afd service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81409) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:28:32.364721 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG dbcounter [-] [81409] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=81409) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:28:32.391515 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-44ad939a-564c-4d56-9793-5f3054a16afd service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:28:32.392932 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-44ad939a-564c-4d56-9793-5f3054a16afd service neutron] RBAC: Authorization granted {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:28:32.403211 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 1189/2344] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Oct 13 04:28:32 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 147 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 13 04:28:35.646318 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f604069c-554e-4715-b1b4-7ff5af649c10 service cinder] Authenticating user token {{(pid=81408) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:28:35.660239 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f604069c-554e-4715-b1b4-7ff5af649c10 service cinder] Validating token access rules against request {{(pid=81408) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:28:35.666628 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e50bafe6-13d2-4f23-961b-dc118b66f586 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6141b8a6c3d54e179d6879bb019f4d12', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dcec0431fb6849499d6aec35a234daa6', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81408) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:28:35.667842 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e50bafe6-13d2-4f23-961b-dc118b66f586 service cinder] REQUEST_METHOD: `GET` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:28:35.668221 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e50bafe6-13d2-4f23-961b-dc118b66f586 service cinder] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:28:35.670807 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e50bafe6-13d2-4f23-961b-dc118b66f586 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:28:35.671534 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.fernet_utils [None req-e50bafe6-13d2-4f23-961b-dc118b66f586 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81408) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:28:35.783928 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e50bafe6-13d2-4f23-961b-dc118b66f586 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:28:35.785554 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e50bafe6-13d2-4f23-961b-dc118b66f586 service cinder] RBAC: Authorization granted {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:28:35.796520 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 1156/2345] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Oct 13 04:28:35 2022] GET /identity/v3/auth/tokens => generated 3262 bytes in 151 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 13 04:28:38.197170 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-44ad939a-564c-4d56-9793-5f3054a16afd service neutron] Authenticating user token {{(pid=81409) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:28:38.199358 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.fernet_utils [None req-44ad939a-564c-4d56-9793-5f3054a16afd service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81409) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:28:38.211242 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-44ad939a-564c-4d56-9793-5f3054a16afd service neutron] Validating token access rules against request {{(pid=81409) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:28:38.216423 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7d2d194f-4e9c-4a5c-ae74-0d0f0fa01856 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8ba823f313144da3b683bacef8fd62e0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81409) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:28:38.216994 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d2d194f-4e9c-4a5c-ae74-0d0f0fa01856 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:28:38.217269 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d2d194f-4e9c-4a5c-ae74-0d0f0fa01856 admin admin] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:28:38.217538 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d2d194f-4e9c-4a5c-ae74-0d0f0fa01856 admin admin] PATH_INFO: `/v3/users/2a7c9f30017f446bb9178c04357905d6` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:28:38.218683 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7d2d194f-4e9c-4a5c-ae74-0d0f0fa01856 admin admin] RBAC: Authorizing `identity:delete_user(user_id=2a7c9f30017f446bb9178c04357905d6)` {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:28:38.220592 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7d2d194f-4e9c-4a5c-ae74-0d0f0fa01856 admin admin] RBAC: Authorization granted {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:28:38.291062 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.notifications [None req-7d2d194f-4e9c-4a5c-ae74-0d0f0fa01856 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '2a7c9f30017f446bb9178c04357905d6'} {{(pid=81409) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 13 04:28:38.292053 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.notifications [None req-7d2d194f-4e9c-4a5c-ae74-0d0f0fa01856 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '2a7c9f30017f446bb9178c04357905d6'} {{(pid=81409) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 13 04:28:38.306721 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.notifications [None req-7d2d194f-4e9c-4a5c-ae74-0d0f0fa01856 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '2a7c9f30017f446bb9178c04357905d6'} {{(pid=81409) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 13 04:28:38.307607 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.notifications [None req-7d2d194f-4e9c-4a5c-ae74-0d0f0fa01856 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '2a7c9f30017f446bb9178c04357905d6'} {{(pid=81409) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 13 04:28:38.328767 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.notifications [None req-7d2d194f-4e9c-4a5c-ae74-0d0f0fa01856 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '2a7c9f30017f446bb9178c04357905d6'} {{(pid=81409) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 13 04:28:38.359739 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 1190/2346] 10.222.0.27 () {58 vars in 1224 bytes} [Thu Oct 13 04:28:38 2022] DELETE /identity/v3/users/2a7c9f30017f446bb9178c04357905d6 => generated 0 bytes in 163 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 13 04:28:38.365412 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07ab0b56-661d-431f-85bc-ed8b6811a6e0 None None] REQUEST_METHOD: `POST` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:28:38.366335 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07ab0b56-661d-431f-85bc-ed8b6811a6e0 None None] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:28:38.366963 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07ab0b56-661d-431f-85bc-ed8b6811a6e0 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:28:38.414032 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.auth.core [None req-07ab0b56-661d-431f-85bc-ed8b6811a6e0 None None] MFA Rules not processed for user `0e2d979563d6421995451a0e86bf3022`. Rule list: `[]` (Enabled: `True`). {{(pid=81408) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 13 04:28:38.415308 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.fernet_utils [None req-07ab0b56-661d-431f-85bc-ed8b6811a6e0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81408) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:28:38.485395 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 1157/2347] 10.222.0.27 () {58 vars in 958 bytes} [Thu Oct 13 04:28:38 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 122 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 13 04:28:38.507311 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7d2d194f-4e9c-4a5c-ae74-0d0f0fa01856 admin admin] Authenticating user token {{(pid=81409) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:28:38.509451 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.fernet_utils [None req-7d2d194f-4e9c-4a5c-ae74-0d0f0fa01856 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81409) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:28:38.593380 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7d2d194f-4e9c-4a5c-ae74-0d0f0fa01856 admin admin] Validating token access rules against request {{(pid=81409) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:28:38.599198 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d50b8337-e550-43a6-815a-fa107bc781a4 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd8812c3974f84871bb2d761cf0016257', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dcec0431fb6849499d6aec35a234daa6', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81409) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:28:38.599830 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d50b8337-e550-43a6-815a-fa107bc781a4 service neutron] REQUEST_METHOD: `GET` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:28:38.600099 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d50b8337-e550-43a6-815a-fa107bc781a4 service neutron] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:28:38.600384 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d50b8337-e550-43a6-815a-fa107bc781a4 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:28:38.611272 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d50b8337-e550-43a6-815a-fa107bc781a4 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:28:38.612770 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d50b8337-e550-43a6-815a-fa107bc781a4 service neutron] RBAC: Authorization granted {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:28:38.630377 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 1191/2348] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Oct 13 04:28:38 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 123 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 13 04:28:39.060680 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-07ab0b56-661d-431f-85bc-ed8b6811a6e0 None None] Authenticating user token {{(pid=81408) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:28:39.062357 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.fernet_utils [None req-07ab0b56-661d-431f-85bc-ed8b6811a6e0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81408) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:28:39.102475 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-07ab0b56-661d-431f-85bc-ed8b6811a6e0 None None] Validating token access rules against request {{(pid=81408) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:28:39.133743 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-80bd0194-c258-4297-811d-28d5a8b56176 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8ba823f313144da3b683bacef8fd62e0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81408) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:28:39.135264 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80bd0194-c258-4297-811d-28d5a8b56176 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:28:39.135966 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80bd0194-c258-4297-811d-28d5a8b56176 admin admin] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:28:39.139048 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80bd0194-c258-4297-811d-28d5a8b56176 admin admin] PATH_INFO: `/v3/projects/36d9f45714674d75bdc9b7c155c1b666` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:28:39.140923 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-80bd0194-c258-4297-811d-28d5a8b56176 admin admin] RBAC: Authorizing `identity:delete_project(project_id=36d9f45714674d75bdc9b7c155c1b666)` {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:28:39.144630 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-80bd0194-c258-4297-811d-28d5a8b56176 admin admin] RBAC: Authorization granted {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:28:39.171386 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG dbcounter [-] [81408] Writing DB stats keystone:SELECT=100 {{(pid=81408) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:28:39.218425 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.notifications [None req-80bd0194-c258-4297-811d-28d5a8b56176 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '36d9f45714674d75bdc9b7c155c1b666', 'request_id': 'req-80bd0194-c258-4297-811d-28d5a8b56176'} {{(pid=81408) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 13 04:28:39.234348 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.notifications [None req-80bd0194-c258-4297-811d-28d5a8b56176 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '36d9f45714674d75bdc9b7c155c1b666', 'request_id': 'req-80bd0194-c258-4297-811d-28d5a8b56176'} {{(pid=81408) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 13 04:28:39.240490 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d50b8337-e550-43a6-815a-fa107bc781a4 service neutron] Authenticating user token {{(pid=81409) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:28:39.242604 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.fernet_utils [None req-d50b8337-e550-43a6-815a-fa107bc781a4 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81409) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:28:39.246035 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.notifications [None req-80bd0194-c258-4297-811d-28d5a8b56176 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '36d9f45714674d75bdc9b7c155c1b666', 'request_id': 'req-80bd0194-c258-4297-811d-28d5a8b56176'} {{(pid=81408) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 13 04:28:39.261757 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d50b8337-e550-43a6-815a-fa107bc781a4 service neutron] Validating token access rules against request {{(pid=81409) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:28:39.266705 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.notifications [None req-80bd0194-c258-4297-811d-28d5a8b56176 admin admin] The token cache is being invalidate because project 36d9f45714674d75bdc9b7c155c1b666 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81408) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 13 04:28:39.266998 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.notifications [None req-80bd0194-c258-4297-811d-28d5a8b56176 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81408) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 13 04:28:39.267646 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.notifications [None req-80bd0194-c258-4297-811d-28d5a8b56176 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81408) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 13 04:28:39.269130 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 1158/2349] 10.222.0.27 () {58 vars in 1233 bytes} [Thu Oct 13 04:28:39 2022] DELETE /identity/v3/projects/36d9f45714674d75bdc9b7c155c1b666 => generated 0 bytes in 209 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 13 04:28:39.273054 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-80bd0194-c258-4297-811d-28d5a8b56176 admin admin] Authenticating user token {{(pid=81408) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:28:39.274382 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.fernet_utils [None req-80bd0194-c258-4297-811d-28d5a8b56176 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81408) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:28:39.317585 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG dbcounter [-] [81409] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81409) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:28:39.334536 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bc09c075-6acc-4a5b-9f7b-fc0289d15891 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8ba823f313144da3b683bacef8fd62e0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81409) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:28:39.335339 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc09c075-6acc-4a5b-9f7b-fc0289d15891 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:28:39.335943 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc09c075-6acc-4a5b-9f7b-fc0289d15891 admin admin] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:28:39.336568 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc09c075-6acc-4a5b-9f7b-fc0289d15891 admin admin] PATH_INFO: `/v3/users/242490d937324b31bc95ae1982ea0b39` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:28:39.337841 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bc09c075-6acc-4a5b-9f7b-fc0289d15891 admin admin] RBAC: Authorizing `identity:delete_user(user_id=242490d937324b31bc95ae1982ea0b39)` {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:28:39.340878 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bc09c075-6acc-4a5b-9f7b-fc0289d15891 admin admin] RBAC: Authorization granted {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:28:39.346917 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-80bd0194-c258-4297-811d-28d5a8b56176 admin admin] Validating token access rules against request {{(pid=81408) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:28:39.351559 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e4503286-57b3-4360-80da-9fca7fe53b6f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8ba823f313144da3b683bacef8fd62e0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81408) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:28:39.352187 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4503286-57b3-4360-80da-9fca7fe53b6f admin admin] REQUEST_METHOD: `DELETE` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:28:39.352462 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4503286-57b3-4360-80da-9fca7fe53b6f admin admin] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:28:39.352736 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4503286-57b3-4360-80da-9fca7fe53b6f admin admin] PATH_INFO: `/v3/users/3c571c60a63c42a696b3ca12f9678618` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:28:39.353868 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e4503286-57b3-4360-80da-9fca7fe53b6f admin admin] RBAC: Authorizing `identity:delete_user(user_id=3c571c60a63c42a696b3ca12f9678618)` {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:28:39.356752 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e4503286-57b3-4360-80da-9fca7fe53b6f admin admin] RBAC: Authorization granted {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:28:39.397855 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.notifications [None req-bc09c075-6acc-4a5b-9f7b-fc0289d15891 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '242490d937324b31bc95ae1982ea0b39'} {{(pid=81409) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 13 04:28:39.398521 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.notifications [None req-bc09c075-6acc-4a5b-9f7b-fc0289d15891 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '242490d937324b31bc95ae1982ea0b39'} {{(pid=81409) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 13 04:28:39.410609 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.notifications [None req-bc09c075-6acc-4a5b-9f7b-fc0289d15891 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '242490d937324b31bc95ae1982ea0b39'} {{(pid=81409) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 13 04:28:39.411252 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.notifications [None req-e4503286-57b3-4360-80da-9fca7fe53b6f admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '3c571c60a63c42a696b3ca12f9678618'} {{(pid=81408) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 13 04:28:39.411766 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.notifications [None req-bc09c075-6acc-4a5b-9f7b-fc0289d15891 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '242490d937324b31bc95ae1982ea0b39'} {{(pid=81409) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 13 04:28:39.421065 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.notifications [None req-bc09c075-6acc-4a5b-9f7b-fc0289d15891 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '242490d937324b31bc95ae1982ea0b39'} {{(pid=81409) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 13 04:28:39.421963 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.notifications [None req-e4503286-57b3-4360-80da-9fca7fe53b6f admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '3c571c60a63c42a696b3ca12f9678618'} {{(pid=81408) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 13 04:28:39.422719 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.notifications [None req-e4503286-57b3-4360-80da-9fca7fe53b6f admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '3c571c60a63c42a696b3ca12f9678618'} {{(pid=81408) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 13 04:28:39.423511 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.notifications [None req-e4503286-57b3-4360-80da-9fca7fe53b6f admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '3c571c60a63c42a696b3ca12f9678618'} {{(pid=81408) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 13 04:28:39.432208 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.notifications [None req-e4503286-57b3-4360-80da-9fca7fe53b6f admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '3c571c60a63c42a696b3ca12f9678618'} {{(pid=81408) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 13 04:28:39.446240 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 1192/2350] 10.222.0.27 () {58 vars in 1224 bytes} [Thu Oct 13 04:28:39 2022] DELETE /identity/v3/users/242490d937324b31bc95ae1982ea0b39 => generated 0 bytes in 206 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 13 04:28:39.459252 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 1159/2351] 10.222.0.27 () {58 vars in 1224 bytes} [Thu Oct 13 04:28:39 2022] DELETE /identity/v3/users/3c571c60a63c42a696b3ca12f9678618 => generated 0 bytes in 186 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 13 04:28:39.658495 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bc09c075-6acc-4a5b-9f7b-fc0289d15891 admin admin] Authenticating user token {{(pid=81409) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:28:39.660352 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.fernet_utils [None req-bc09c075-6acc-4a5b-9f7b-fc0289d15891 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81409) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:28:39.738280 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bc09c075-6acc-4a5b-9f7b-fc0289d15891 admin admin] Validating token access rules against request {{(pid=81409) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:28:39.768575 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e4503286-57b3-4360-80da-9fca7fe53b6f admin admin] Authenticating user token {{(pid=81408) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:28:39.770255 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.fernet_utils [None req-e4503286-57b3-4360-80da-9fca7fe53b6f admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81408) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:28:39.802799 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG dbcounter [-] [81408] Writing DB stats keystone:SELECT=81,keystone:DELETE=17,keystone:INSERT=2 {{(pid=81408) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:28:39.805119 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-66c9a5a6-9ab2-4890-bd74-00903ffb30e0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8ba823f313144da3b683bacef8fd62e0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81409) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:28:39.805917 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66c9a5a6-9ab2-4890-bd74-00903ffb30e0 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:28:39.806092 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66c9a5a6-9ab2-4890-bd74-00903ffb30e0 admin admin] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:28:39.806402 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66c9a5a6-9ab2-4890-bd74-00903ffb30e0 admin admin] PATH_INFO: `/v3/projects/fabffb6796a34773987a4f7e414352d0` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:28:39.808382 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-66c9a5a6-9ab2-4890-bd74-00903ffb30e0 admin admin] RBAC: Authorizing `identity:delete_project(project_id=fabffb6796a34773987a4f7e414352d0)` {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:28:39.810521 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-66c9a5a6-9ab2-4890-bd74-00903ffb30e0 admin admin] RBAC: Authorization granted {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:28:39.832933 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG dbcounter [-] [81409] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81409) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:28:39.861988 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e4503286-57b3-4360-80da-9fca7fe53b6f admin admin] Validating token access rules against request {{(pid=81408) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:28:39.869300 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.notifications [None req-66c9a5a6-9ab2-4890-bd74-00903ffb30e0 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'fabffb6796a34773987a4f7e414352d0', 'request_id': 'req-66c9a5a6-9ab2-4890-bd74-00903ffb30e0'} {{(pid=81409) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 13 04:28:39.880508 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.notifications [None req-66c9a5a6-9ab2-4890-bd74-00903ffb30e0 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'fabffb6796a34773987a4f7e414352d0', 'request_id': 'req-66c9a5a6-9ab2-4890-bd74-00903ffb30e0'} {{(pid=81409) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 13 04:28:39.887940 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.notifications [None req-66c9a5a6-9ab2-4890-bd74-00903ffb30e0 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'fabffb6796a34773987a4f7e414352d0', 'request_id': 'req-66c9a5a6-9ab2-4890-bd74-00903ffb30e0'} {{(pid=81409) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 13 04:28:39.898851 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.notifications [None req-66c9a5a6-9ab2-4890-bd74-00903ffb30e0 admin admin] The token cache is being invalidate because project fabffb6796a34773987a4f7e414352d0 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81409) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 13 04:28:39.899145 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.notifications [None req-66c9a5a6-9ab2-4890-bd74-00903ffb30e0 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81409) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 13 04:28:39.899789 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.notifications [None req-66c9a5a6-9ab2-4890-bd74-00903ffb30e0 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81409) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 13 04:28:39.901272 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 1193/2352] 10.222.0.27 () {58 vars in 1233 bytes} [Thu Oct 13 04:28:39 2022] DELETE /identity/v3/projects/fabffb6796a34773987a4f7e414352d0 => generated 0 bytes in 243 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 13 04:28:39.905522 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-66c9a5a6-9ab2-4890-bd74-00903ffb30e0 admin admin] Authenticating user token {{(pid=81409) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:28:39.906863 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.fernet_utils [None req-66c9a5a6-9ab2-4890-bd74-00903ffb30e0 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81409) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:28:39.922409 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5737d32b-61d1-483c-8372-db3412fe4d0a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8ba823f313144da3b683bacef8fd62e0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81408) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:28:39.923053 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5737d32b-61d1-483c-8372-db3412fe4d0a admin admin] REQUEST_METHOD: `DELETE` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:28:39.923379 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5737d32b-61d1-483c-8372-db3412fe4d0a admin admin] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:28:39.923654 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5737d32b-61d1-483c-8372-db3412fe4d0a admin admin] PATH_INFO: `/v3/projects/797d67a0c1cc4d478291fef206129965` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:28:39.924873 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5737d32b-61d1-483c-8372-db3412fe4d0a admin admin] RBAC: Authorizing `identity:delete_project(project_id=797d67a0c1cc4d478291fef206129965)` {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:28:39.926988 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5737d32b-61d1-483c-8372-db3412fe4d0a admin admin] RBAC: Authorization granted {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:28:39.975176 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.notifications [None req-5737d32b-61d1-483c-8372-db3412fe4d0a admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '797d67a0c1cc4d478291fef206129965', 'request_id': 'req-5737d32b-61d1-483c-8372-db3412fe4d0a'} {{(pid=81408) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 13 04:28:39.977894 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-66c9a5a6-9ab2-4890-bd74-00903ffb30e0 admin admin] Validating token access rules against request {{(pid=81409) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:28:39.982866 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.notifications [None req-5737d32b-61d1-483c-8372-db3412fe4d0a admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '797d67a0c1cc4d478291fef206129965', 'request_id': 'req-5737d32b-61d1-483c-8372-db3412fe4d0a'} {{(pid=81408) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 13 04:28:39.989496 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.notifications [None req-5737d32b-61d1-483c-8372-db3412fe4d0a admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '797d67a0c1cc4d478291fef206129965', 'request_id': 'req-5737d32b-61d1-483c-8372-db3412fe4d0a'} {{(pid=81408) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 13 04:28:40.005599 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.notifications [None req-5737d32b-61d1-483c-8372-db3412fe4d0a admin admin] The token cache is being invalidate because project 797d67a0c1cc4d478291fef206129965 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81408) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 13 04:28:40.005885 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.notifications [None req-5737d32b-61d1-483c-8372-db3412fe4d0a admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81408) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 13 04:28:40.006560 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.notifications [None req-5737d32b-61d1-483c-8372-db3412fe4d0a admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81408) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 13 04:28:40.008144 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 1160/2353] 10.222.0.27 () {58 vars in 1233 bytes} [Thu Oct 13 04:28:39 2022] DELETE /identity/v3/projects/797d67a0c1cc4d478291fef206129965 => generated 0 bytes in 240 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 13 04:28:40.012258 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5737d32b-61d1-483c-8372-db3412fe4d0a admin admin] Authenticating user token {{(pid=81408) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:28:40.013523 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.fernet_utils [None req-5737d32b-61d1-483c-8372-db3412fe4d0a admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81408) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:28:40.039635 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-12007941-d288-4882-adce-b78e045dd540 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8ba823f313144da3b683bacef8fd62e0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81409) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:28:40.040650 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12007941-d288-4882-adce-b78e045dd540 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:28:40.041232 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12007941-d288-4882-adce-b78e045dd540 admin admin] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:28:40.041505 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12007941-d288-4882-adce-b78e045dd540 admin admin] PATH_INFO: `/v3/users/4617d83c9dac4f939091c1abfef5eb17` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:28:40.043184 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-12007941-d288-4882-adce-b78e045dd540 admin admin] RBAC: Authorizing `identity:delete_user(user_id=4617d83c9dac4f939091c1abfef5eb17)` {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:28:40.045300 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-12007941-d288-4882-adce-b78e045dd540 admin admin] RBAC: Authorization granted {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:28:40.077986 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG dbcounter [-] [81408] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=81408) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:28:40.083583 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5737d32b-61d1-483c-8372-db3412fe4d0a admin admin] Validating token access rules against request {{(pid=81408) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:28:40.089985 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e247cc3e-081f-49e1-aeeb-b31fd504e736 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8ba823f313144da3b683bacef8fd62e0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81408) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:28:40.090650 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e247cc3e-081f-49e1-aeeb-b31fd504e736 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:28:40.091144 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e247cc3e-081f-49e1-aeeb-b31fd504e736 admin admin] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:28:40.091744 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e247cc3e-081f-49e1-aeeb-b31fd504e736 admin admin] PATH_INFO: `/v3/users/d4aaf64993a44c309689022e70400951` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:28:40.093005 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e247cc3e-081f-49e1-aeeb-b31fd504e736 admin admin] RBAC: Authorizing `identity:delete_user(user_id=d4aaf64993a44c309689022e70400951)` {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:28:40.094978 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e247cc3e-081f-49e1-aeeb-b31fd504e736 admin admin] RBAC: Authorization granted {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:28:40.099590 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.notifications [None req-12007941-d288-4882-adce-b78e045dd540 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '4617d83c9dac4f939091c1abfef5eb17'} {{(pid=81409) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 13 04:28:40.100378 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.notifications [None req-12007941-d288-4882-adce-b78e045dd540 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '4617d83c9dac4f939091c1abfef5eb17'} {{(pid=81409) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 13 04:28:40.107847 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.notifications [None req-12007941-d288-4882-adce-b78e045dd540 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '4617d83c9dac4f939091c1abfef5eb17'} {{(pid=81409) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 13 04:28:40.108618 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.notifications [None req-12007941-d288-4882-adce-b78e045dd540 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '4617d83c9dac4f939091c1abfef5eb17'} {{(pid=81409) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 13 04:28:40.110296 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG dbcounter [-] [81409] Writing DB stats keystone:SELECT=85,keystone:DELETE=14,keystone:INSERT=1 {{(pid=81409) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:28:40.123846 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.notifications [None req-12007941-d288-4882-adce-b78e045dd540 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '4617d83c9dac4f939091c1abfef5eb17'} {{(pid=81409) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 13 04:28:40.146390 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 1194/2354] 10.222.0.27 () {58 vars in 1224 bytes} [Thu Oct 13 04:28:39 2022] DELETE /identity/v3/users/4617d83c9dac4f939091c1abfef5eb17 => generated 0 bytes in 241 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 13 04:28:40.151112 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.notifications [None req-e247cc3e-081f-49e1-aeeb-b31fd504e736 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'd4aaf64993a44c309689022e70400951'} {{(pid=81408) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 13 04:28:40.158819 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.notifications [None req-e247cc3e-081f-49e1-aeeb-b31fd504e736 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'd4aaf64993a44c309689022e70400951'} {{(pid=81408) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 13 04:28:40.160203 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.notifications [None req-e247cc3e-081f-49e1-aeeb-b31fd504e736 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'd4aaf64993a44c309689022e70400951'} {{(pid=81408) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 13 04:28:40.160775 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.notifications [None req-e247cc3e-081f-49e1-aeeb-b31fd504e736 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'd4aaf64993a44c309689022e70400951'} {{(pid=81408) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 13 04:28:40.174161 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.notifications [None req-e247cc3e-081f-49e1-aeeb-b31fd504e736 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'd4aaf64993a44c309689022e70400951'} {{(pid=81408) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 13 04:28:40.215101 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 1161/2355] 10.222.0.27 () {58 vars in 1224 bytes} [Thu Oct 13 04:28:40 2022] DELETE /identity/v3/users/d4aaf64993a44c309689022e70400951 => generated 0 bytes in 203 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 13 04:28:40.359576 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-12007941-d288-4882-adce-b78e045dd540 admin admin] Authenticating user token {{(pid=81409) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:28:40.361043 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.fernet_utils [None req-12007941-d288-4882-adce-b78e045dd540 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81409) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:28:40.485244 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-12007941-d288-4882-adce-b78e045dd540 admin admin] Validating token access rules against request {{(pid=81409) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:28:40.565682 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fe94d2aa-a0f8-4a43-bec0-a8f69406f8ca admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8ba823f313144da3b683bacef8fd62e0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81409) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:28:40.566411 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe94d2aa-a0f8-4a43-bec0-a8f69406f8ca admin admin] REQUEST_METHOD: `DELETE` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:28:40.566854 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe94d2aa-a0f8-4a43-bec0-a8f69406f8ca admin admin] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:28:40.567399 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe94d2aa-a0f8-4a43-bec0-a8f69406f8ca admin admin] PATH_INFO: `/v3/projects/dc5dcbaa1692443fb71e454c5d148bf8` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:28:40.568551 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fe94d2aa-a0f8-4a43-bec0-a8f69406f8ca admin admin] RBAC: Authorizing `identity:delete_project(project_id=dc5dcbaa1692443fb71e454c5d148bf8)` {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:28:40.569622 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e247cc3e-081f-49e1-aeeb-b31fd504e736 admin admin] Authenticating user token {{(pid=81408) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:28:40.570629 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fe94d2aa-a0f8-4a43-bec0-a8f69406f8ca admin admin] RBAC: Authorization granted {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:28:40.571870 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.fernet_utils [None req-e247cc3e-081f-49e1-aeeb-b31fd504e736 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81408) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:28:40.587817 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e247cc3e-081f-49e1-aeeb-b31fd504e736 admin admin] Validating token access rules against request {{(pid=81408) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:28:40.601448 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fd2b24b1-88be-4157-84c8-8b4b2768cd4a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8ba823f313144da3b683bacef8fd62e0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81408) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:28:40.602281 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd2b24b1-88be-4157-84c8-8b4b2768cd4a admin admin] REQUEST_METHOD: `DELETE` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:28:40.603122 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd2b24b1-88be-4157-84c8-8b4b2768cd4a admin admin] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:28:40.603412 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd2b24b1-88be-4157-84c8-8b4b2768cd4a admin admin] PATH_INFO: `/v3/projects/60d27631ff674889b323af891d908aae` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:28:40.605350 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fd2b24b1-88be-4157-84c8-8b4b2768cd4a admin admin] RBAC: Authorizing `identity:delete_project(project_id=60d27631ff674889b323af891d908aae)` {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:28:40.609482 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fd2b24b1-88be-4157-84c8-8b4b2768cd4a admin admin] RBAC: Authorization granted {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:28:40.631703 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.notifications [None req-fe94d2aa-a0f8-4a43-bec0-a8f69406f8ca admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'dc5dcbaa1692443fb71e454c5d148bf8', 'request_id': 'req-fe94d2aa-a0f8-4a43-bec0-a8f69406f8ca'} {{(pid=81409) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 13 04:28:40.643823 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.notifications [None req-fe94d2aa-a0f8-4a43-bec0-a8f69406f8ca admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'dc5dcbaa1692443fb71e454c5d148bf8', 'request_id': 'req-fe94d2aa-a0f8-4a43-bec0-a8f69406f8ca'} {{(pid=81409) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 13 04:28:40.654471 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.notifications [None req-fe94d2aa-a0f8-4a43-bec0-a8f69406f8ca admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'dc5dcbaa1692443fb71e454c5d148bf8', 'request_id': 'req-fe94d2aa-a0f8-4a43-bec0-a8f69406f8ca'} {{(pid=81409) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 13 04:28:40.665916 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.notifications [None req-fe94d2aa-a0f8-4a43-bec0-a8f69406f8ca admin admin] The token cache is being invalidate because project dc5dcbaa1692443fb71e454c5d148bf8 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81409) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 13 04:28:40.666456 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.notifications [None req-fe94d2aa-a0f8-4a43-bec0-a8f69406f8ca admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81409) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 13 04:28:40.667134 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.notifications [None req-fe94d2aa-a0f8-4a43-bec0-a8f69406f8ca admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81409) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 13 04:28:40.668903 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 1195/2356] 10.222.0.27 () {58 vars in 1233 bytes} [Thu Oct 13 04:28:40 2022] DELETE /identity/v3/projects/dc5dcbaa1692443fb71e454c5d148bf8 => generated 0 bytes in 310 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 13 04:28:40.669177 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.notifications [None req-fd2b24b1-88be-4157-84c8-8b4b2768cd4a admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '60d27631ff674889b323af891d908aae', 'request_id': 'req-fd2b24b1-88be-4157-84c8-8b4b2768cd4a'} {{(pid=81408) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 13 04:28:40.677274 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.notifications [None req-fd2b24b1-88be-4157-84c8-8b4b2768cd4a admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '60d27631ff674889b323af891d908aae', 'request_id': 'req-fd2b24b1-88be-4157-84c8-8b4b2768cd4a'} {{(pid=81408) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 13 04:28:40.685516 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.notifications [None req-fd2b24b1-88be-4157-84c8-8b4b2768cd4a admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '60d27631ff674889b323af891d908aae', 'request_id': 'req-fd2b24b1-88be-4157-84c8-8b4b2768cd4a'} {{(pid=81408) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 13 04:28:40.697877 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.notifications [None req-fd2b24b1-88be-4157-84c8-8b4b2768cd4a admin admin] The token cache is being invalidate because project 60d27631ff674889b323af891d908aae was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81408) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 13 04:28:40.698161 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.notifications [None req-fd2b24b1-88be-4157-84c8-8b4b2768cd4a admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81408) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 13 04:28:40.699939 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.notifications [None req-fd2b24b1-88be-4157-84c8-8b4b2768cd4a admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81408) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 13 04:28:40.701783 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 1162/2357] 10.222.0.27 () {58 vars in 1233 bytes} [Thu Oct 13 04:28:40 2022] DELETE /identity/v3/projects/60d27631ff674889b323af891d908aae => generated 0 bytes in 133 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 13 04:28:40.708112 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5998a8ee-d1c8-490a-b826-5551ef16d54a None None] REQUEST_METHOD: `POST` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:28:40.708462 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5998a8ee-d1c8-490a-b826-5551ef16d54a None None] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:28:40.708825 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5998a8ee-d1c8-490a-b826-5551ef16d54a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:28:40.734760 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.auth.core [None req-5998a8ee-d1c8-490a-b826-5551ef16d54a None None] MFA Rules not processed for user `0e2d979563d6421995451a0e86bf3022`. Rule list: `[]` (Enabled: `True`). {{(pid=81409) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 13 04:28:40.736122 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.fernet_utils [None req-5998a8ee-d1c8-490a-b826-5551ef16d54a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81409) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:28:40.742360 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG dbcounter [-] [81409] Writing DB stats keystone:INSERT=2,keystone:DELETE=10,keystone:SELECT=88 {{(pid=81409) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:28:40.810592 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 1196/2358] 10.222.0.27 () {58 vars in 958 bytes} [Thu Oct 13 04:28:40 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 104 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 13 04:28:40.821289 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4214a95c-4d05-416d-b42a-db6063fb5cf0 None None] REQUEST_METHOD: `POST` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:28:40.821644 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4214a95c-4d05-416d-b42a-db6063fb5cf0 None None] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:28:40.821912 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4214a95c-4d05-416d-b42a-db6063fb5cf0 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:28:40.844788 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.auth.core [None req-4214a95c-4d05-416d-b42a-db6063fb5cf0 None None] MFA Rules not processed for user `0e2d979563d6421995451a0e86bf3022`. Rule list: `[]` (Enabled: `True`). {{(pid=81408) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 13 04:28:40.846045 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.fernet_utils [None req-4214a95c-4d05-416d-b42a-db6063fb5cf0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81408) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:28:40.855574 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 1163/2359] 10.222.0.27 () {58 vars in 958 bytes} [Thu Oct 13 04:28:40 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 35 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 13 04:28:40.860594 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5998a8ee-d1c8-490a-b826-5551ef16d54a None None] Authenticating user token {{(pid=81409) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:28:40.872517 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5998a8ee-d1c8-490a-b826-5551ef16d54a None None] Validating token access rules against request {{(pid=81409) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:28:40.877490 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bc4983dc-2eef-4bdf-995c-ea99c8bdc452 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8ba823f313144da3b683bacef8fd62e0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81409) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:28:40.878198 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc4983dc-2eef-4bdf-995c-ea99c8bdc452 admin admin] REQUEST_METHOD: `GET` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:28:40.878574 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc4983dc-2eef-4bdf-995c-ea99c8bdc452 admin admin] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:28:40.878920 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc4983dc-2eef-4bdf-995c-ea99c8bdc452 admin admin] PATH_INFO: `/v3/domains` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:28:40.879636 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bc4983dc-2eef-4bdf-995c-ea99c8bdc452 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81409) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 13 04:28:40.880032 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bc4983dc-2eef-4bdf-995c-ea99c8bdc452 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:28:40.887051 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bc4983dc-2eef-4bdf-995c-ea99c8bdc452 admin admin] RBAC: Authorization granted {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:28:40.900609 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 1197/2360] 10.222.0.27 () {58 vars in 1153 bytes} [Thu Oct 13 04:28:40 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 41 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 13 04:28:40.905660 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4214a95c-4d05-416d-b42a-db6063fb5cf0 None None] Authenticating user token {{(pid=81408) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:28:40.915649 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4214a95c-4d05-416d-b42a-db6063fb5cf0 None None] Validating token access rules against request {{(pid=81408) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:28:40.921550 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6b17471f-5537-4954-912c-26b1636d8040 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8ba823f313144da3b683bacef8fd62e0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81408) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:28:40.922171 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b17471f-5537-4954-912c-26b1636d8040 admin admin] REQUEST_METHOD: `POST` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:28:40.922495 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b17471f-5537-4954-912c-26b1636d8040 admin admin] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:28:40.922784 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b17471f-5537-4954-912c-26b1636d8040 admin admin] PATH_INFO: `/v3/projects` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:28:40.923422 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6b17471f-5537-4954-912c-26b1636d8040 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:28:40.925406 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6b17471f-5537-4954-912c-26b1636d8040 admin admin] RBAC: Authorization granted {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:28:40.959757 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 1164/2361] 10.222.0.27 () {60 vars in 1153 bytes} [Thu Oct 13 04:28:40 2022] POST /identity/v3/projects => generated 381 bytes in 55 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 13 04:28:40.964689 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bc4983dc-2eef-4bdf-995c-ea99c8bdc452 admin admin] Authenticating user token {{(pid=81409) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:28:40.974854 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bc4983dc-2eef-4bdf-995c-ea99c8bdc452 admin admin] Validating token access rules against request {{(pid=81409) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:28:41.034999 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-658ce647-a659-4e73-b145-b68dc8144076 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8ba823f313144da3b683bacef8fd62e0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81409) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:28:41.035601 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-658ce647-a659-4e73-b145-b68dc8144076 admin admin] REQUEST_METHOD: `POST` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:28:41.035876 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-658ce647-a659-4e73-b145-b68dc8144076 admin admin] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:28:41.036144 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-658ce647-a659-4e73-b145-b68dc8144076 admin admin] PATH_INFO: `/v3/users` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:28:41.036792 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-658ce647-a659-4e73-b145-b68dc8144076 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:28:41.038798 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-658ce647-a659-4e73-b145-b68dc8144076 admin admin] RBAC: Authorization granted {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:28:41.059286 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 1198/2362] 10.222.0.27 () {60 vars in 1144 bytes} [Thu Oct 13 04:28:40 2022] POST /identity/v3/users => generated 291 bytes in 96 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 13 04:28:41.064602 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6b17471f-5537-4954-912c-26b1636d8040 admin admin] Authenticating user token {{(pid=81408) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:28:41.078827 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6b17471f-5537-4954-912c-26b1636d8040 admin admin] Validating token access rules against request {{(pid=81408) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:28:41.083501 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ef963ecd-fb6f-4277-9d01-3cfddef98859 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8ba823f313144da3b683bacef8fd62e0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81408) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:28:41.084100 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef963ecd-fb6f-4277-9d01-3cfddef98859 admin admin] REQUEST_METHOD: `GET` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:28:41.084375 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef963ecd-fb6f-4277-9d01-3cfddef98859 admin admin] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:28:41.084644 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef963ecd-fb6f-4277-9d01-3cfddef98859 admin admin] PATH_INFO: `/v3/roles` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:28:41.085282 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ef963ecd-fb6f-4277-9d01-3cfddef98859 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:28:41.086916 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ef963ecd-fb6f-4277-9d01-3cfddef98859 admin admin] RBAC: Authorization granted {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:28:41.097776 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 1165/2363] 10.222.0.27 () {58 vars in 1122 bytes} [Thu Oct 13 04:28:41 2022] GET /identity/v3/roles => generated 1420 bytes in 35 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 13 04:28:41.101897 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-658ce647-a659-4e73-b145-b68dc8144076 admin admin] Authenticating user token {{(pid=81409) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:28:41.112587 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-658ce647-a659-4e73-b145-b68dc8144076 admin admin] Validating token access rules against request {{(pid=81409) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:28:41.117203 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9047f19c-01dd-4cda-b564-c5577f73e614 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8ba823f313144da3b683bacef8fd62e0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81409) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:28:41.117822 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9047f19c-01dd-4cda-b564-c5577f73e614 admin admin] REQUEST_METHOD: `GET` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:28:41.118098 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9047f19c-01dd-4cda-b564-c5577f73e614 admin admin] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:28:41.118397 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9047f19c-01dd-4cda-b564-c5577f73e614 admin admin] PATH_INFO: `/v3/roles` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:28:41.119034 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9047f19c-01dd-4cda-b564-c5577f73e614 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:28:41.120626 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9047f19c-01dd-4cda-b564-c5577f73e614 admin admin] RBAC: Authorization granted {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:28:41.136450 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 1199/2364] 10.222.0.27 () {58 vars in 1122 bytes} [Thu Oct 13 04:28:41 2022] GET /identity/v3/roles => generated 1420 bytes in 34 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 13 04:28:41.139629 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ef963ecd-fb6f-4277-9d01-3cfddef98859 admin admin] Authenticating user token {{(pid=81408) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:28:41.150255 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ef963ecd-fb6f-4277-9d01-3cfddef98859 admin admin] Validating token access rules against request {{(pid=81408) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:28:41.155715 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-463405d6-09f5-4bad-af19-97c7d3d010fc admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8ba823f313144da3b683bacef8fd62e0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81408) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:28:41.156361 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-463405d6-09f5-4bad-af19-97c7d3d010fc admin admin] REQUEST_METHOD: `PUT` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:28:41.156638 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-463405d6-09f5-4bad-af19-97c7d3d010fc admin admin] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:28:41.156912 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-463405d6-09f5-4bad-af19-97c7d3d010fc admin admin] PATH_INFO: `/v3/projects/904fd54724a54e0c8fcbe19090ed6229/users/32df27ba55e44daf94082073cbd7ddc4/roles/cc56c4ae13a7424ea771c15da4b153c1` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:28:41.171575 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-463405d6-09f5-4bad-af19-97c7d3d010fc admin admin] RBAC: Authorizing `identity:create_grant(project_id=904fd54724a54e0c8fcbe19090ed6229, user_id=32df27ba55e44daf94082073cbd7ddc4, role_id=cc56c4ae13a7424ea771c15da4b153c1)` {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:28:41.174276 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-463405d6-09f5-4bad-af19-97c7d3d010fc admin admin] RBAC: Authorization granted {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:28:41.176755 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG dbcounter [-] [81408] Writing DB stats keystone:SELECT=80,keystone:DELETE=17,keystone:INSERT=3 {{(pid=81408) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:28:41.189533 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 1166/2365] 10.222.0.27 () {60 vars in 1483 bytes} [Thu Oct 13 04:28:41 2022] PUT /identity/v3/projects/904fd54724a54e0c8fcbe19090ed6229/users/32df27ba55e44daf94082073cbd7ddc4/roles/cc56c4ae13a7424ea771c15da4b153c1 => generated 0 bytes in 51 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 13 04:28:41.202268 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e74ed448-4266-4029-a5c5-a89f66d9e07a None None] REQUEST_METHOD: `POST` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:28:41.202560 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e74ed448-4266-4029-a5c5-a89f66d9e07a None None] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:28:41.202830 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e74ed448-4266-4029-a5c5-a89f66d9e07a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:28:41.231291 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.auth.core [None req-e74ed448-4266-4029-a5c5-a89f66d9e07a None None] MFA Rules not processed for user `32df27ba55e44daf94082073cbd7ddc4`. Rule list: `[]` (Enabled: `True`). {{(pid=81409) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 13 04:28:41.232520 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.fernet_utils [None req-e74ed448-4266-4029-a5c5-a89f66d9e07a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81409) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:28:41.300563 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 1200/2366] 10.222.0.27 () {58 vars in 958 bytes} [Thu Oct 13 04:28:41 2022] POST /identity/v3/auth/tokens => generated 3258 bytes in 99 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 13 04:28:41.305279 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-463405d6-09f5-4bad-af19-97c7d3d010fc admin admin] Authenticating user token {{(pid=81408) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:28:41.375563 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-463405d6-09f5-4bad-af19-97c7d3d010fc admin admin] Validating token access rules against request {{(pid=81408) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:28:41.380255 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-36c52a0d-5d31-46b7-bf26-8eb8711addbd admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8ba823f313144da3b683bacef8fd62e0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81408) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:28:41.380857 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36c52a0d-5d31-46b7-bf26-8eb8711addbd admin admin] REQUEST_METHOD: `POST` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:28:41.381136 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36c52a0d-5d31-46b7-bf26-8eb8711addbd admin admin] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:28:41.381409 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36c52a0d-5d31-46b7-bf26-8eb8711addbd admin admin] PATH_INFO: `/v3/projects` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:28:41.382052 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-36c52a0d-5d31-46b7-bf26-8eb8711addbd admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:28:41.384061 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-36c52a0d-5d31-46b7-bf26-8eb8711addbd admin admin] RBAC: Authorization granted {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:28:41.413027 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 1167/2367] 10.222.0.27 () {60 vars in 1153 bytes} [Thu Oct 13 04:28:41 2022] POST /identity/v3/projects => generated 381 bytes in 108 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 13 04:28:41.417028 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e74ed448-4266-4029-a5c5-a89f66d9e07a None None] Authenticating user token {{(pid=81409) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:28:41.428067 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e74ed448-4266-4029-a5c5-a89f66d9e07a None None] Validating token access rules against request {{(pid=81409) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:28:41.431649 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG dbcounter [-] [81409] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81409) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:28:41.489201 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a8fa8618-1e02-4986-83bd-8c4ab3e57503 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8ba823f313144da3b683bacef8fd62e0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81409) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:28:41.489804 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8fa8618-1e02-4986-83bd-8c4ab3e57503 admin admin] REQUEST_METHOD: `POST` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:28:41.490076 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8fa8618-1e02-4986-83bd-8c4ab3e57503 admin admin] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:28:41.490382 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8fa8618-1e02-4986-83bd-8c4ab3e57503 admin admin] PATH_INFO: `/v3/users` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:28:41.491018 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a8fa8618-1e02-4986-83bd-8c4ab3e57503 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:28:41.492918 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a8fa8618-1e02-4986-83bd-8c4ab3e57503 admin admin] RBAC: Authorization granted {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:28:41.515514 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 1201/2368] 10.222.0.27 () {60 vars in 1144 bytes} [Thu Oct 13 04:28:41 2022] POST /identity/v3/users => generated 297 bytes in 98 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 13 04:28:41.519527 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-36c52a0d-5d31-46b7-bf26-8eb8711addbd admin admin] Authenticating user token {{(pid=81408) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:28:41.530320 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-36c52a0d-5d31-46b7-bf26-8eb8711addbd admin admin] Validating token access rules against request {{(pid=81408) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:28:41.534946 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-33517b95-2be1-4c06-83e4-f74b2bf7eea2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8ba823f313144da3b683bacef8fd62e0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81408) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:28:41.535545 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33517b95-2be1-4c06-83e4-f74b2bf7eea2 admin admin] REQUEST_METHOD: `GET` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:28:41.535822 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33517b95-2be1-4c06-83e4-f74b2bf7eea2 admin admin] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:28:41.536092 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33517b95-2be1-4c06-83e4-f74b2bf7eea2 admin admin] PATH_INFO: `/v3/roles` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:28:41.536735 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-33517b95-2be1-4c06-83e4-f74b2bf7eea2 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:28:41.538369 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-33517b95-2be1-4c06-83e4-f74b2bf7eea2 admin admin] RBAC: Authorization granted {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:28:41.548860 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 1168/2369] 10.222.0.27 () {58 vars in 1122 bytes} [Thu Oct 13 04:28:41 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 13 04:28:41.553060 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a8fa8618-1e02-4986-83bd-8c4ab3e57503 admin admin] Authenticating user token {{(pid=81409) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:28:41.564019 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a8fa8618-1e02-4986-83bd-8c4ab3e57503 admin admin] Validating token access rules against request {{(pid=81409) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:28:41.568621 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f07a5a5f-3bce-455c-914b-55901f6a62e5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8ba823f313144da3b683bacef8fd62e0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81409) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:28:41.569279 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f07a5a5f-3bce-455c-914b-55901f6a62e5 admin admin] REQUEST_METHOD: `PUT` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:28:41.569554 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f07a5a5f-3bce-455c-914b-55901f6a62e5 admin admin] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:28:41.569823 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f07a5a5f-3bce-455c-914b-55901f6a62e5 admin admin] PATH_INFO: `/v3/projects/1a4f73e6458c4f618ebf32049815d9f3/users/070c2441e04a4f2b9d493a9fa58e9537/roles/ef55fbd94e1040be83695754075a7ed5` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:28:41.587170 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f07a5a5f-3bce-455c-914b-55901f6a62e5 admin admin] RBAC: Authorizing `identity:create_grant(project_id=1a4f73e6458c4f618ebf32049815d9f3, user_id=070c2441e04a4f2b9d493a9fa58e9537, role_id=ef55fbd94e1040be83695754075a7ed5)` {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:28:41.589887 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f07a5a5f-3bce-455c-914b-55901f6a62e5 admin admin] RBAC: Authorization granted {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:28:41.607376 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 1202/2370] 10.222.0.27 () {60 vars in 1483 bytes} [Thu Oct 13 04:28:41 2022] PUT /identity/v3/projects/1a4f73e6458c4f618ebf32049815d9f3/users/070c2441e04a4f2b9d493a9fa58e9537/roles/ef55fbd94e1040be83695754075a7ed5 => generated 0 bytes in 55 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 13 04:28:41.611361 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-33517b95-2be1-4c06-83e4-f74b2bf7eea2 admin admin] Authenticating user token {{(pid=81408) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:28:41.622617 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-33517b95-2be1-4c06-83e4-f74b2bf7eea2 admin admin] Validating token access rules against request {{(pid=81408) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:28:41.697389 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-495ef73c-d5a1-4217-9fb7-12eaa01930c3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8ba823f313144da3b683bacef8fd62e0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81408) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:28:41.698174 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-495ef73c-d5a1-4217-9fb7-12eaa01930c3 admin admin] REQUEST_METHOD: `GET` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:28:41.699279 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-495ef73c-d5a1-4217-9fb7-12eaa01930c3 admin admin] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:28:41.699679 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-495ef73c-d5a1-4217-9fb7-12eaa01930c3 admin admin] PATH_INFO: `/v3/roles` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:28:41.700368 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-495ef73c-d5a1-4217-9fb7-12eaa01930c3 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:28:41.702403 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-495ef73c-d5a1-4217-9fb7-12eaa01930c3 admin admin] RBAC: Authorization granted {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:28:41.715899 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 1169/2371] 10.222.0.27 () {58 vars in 1122 bytes} [Thu Oct 13 04:28:41 2022] GET /identity/v3/roles => generated 1420 bytes in 105 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 13 04:28:41.720437 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f07a5a5f-3bce-455c-914b-55901f6a62e5 admin admin] Authenticating user token {{(pid=81409) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:28:41.737225 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f07a5a5f-3bce-455c-914b-55901f6a62e5 admin admin] Validating token access rules against request {{(pid=81409) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:28:41.742122 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-39b96792-e6ff-428e-af1a-84903aa3cb8b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8ba823f313144da3b683bacef8fd62e0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81409) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:28:41.742900 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39b96792-e6ff-428e-af1a-84903aa3cb8b admin admin] REQUEST_METHOD: `PUT` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:28:41.743247 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39b96792-e6ff-428e-af1a-84903aa3cb8b admin admin] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:28:41.743604 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39b96792-e6ff-428e-af1a-84903aa3cb8b admin admin] PATH_INFO: `/v3/projects/1a4f73e6458c4f618ebf32049815d9f3/users/070c2441e04a4f2b9d493a9fa58e9537/roles/ef55fbd94e1040be83695754075a7ed5` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:28:41.746148 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-39b96792-e6ff-428e-af1a-84903aa3cb8b admin admin] RBAC: Authorizing `identity:create_grant(project_id=1a4f73e6458c4f618ebf32049815d9f3, user_id=070c2441e04a4f2b9d493a9fa58e9537, role_id=ef55fbd94e1040be83695754075a7ed5)` {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:28:41.749052 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-39b96792-e6ff-428e-af1a-84903aa3cb8b admin admin] RBAC: Authorization granted {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:28:41.759664 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 1203/2372] 10.222.0.27 () {60 vars in 1483 bytes} [Thu Oct 13 04:28:41 2022] PUT /identity/v3/projects/1a4f73e6458c4f618ebf32049815d9f3/users/070c2441e04a4f2b9d493a9fa58e9537/roles/ef55fbd94e1040be83695754075a7ed5 => generated 0 bytes in 40 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 13 04:28:41.771443 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b78890c0-b9b8-48a0-8c0c-a73e000ade17 None None] REQUEST_METHOD: `POST` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:28:41.771703 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b78890c0-b9b8-48a0-8c0c-a73e000ade17 None None] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:28:41.772050 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b78890c0-b9b8-48a0-8c0c-a73e000ade17 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:28:41.804224 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.auth.core [None req-b78890c0-b9b8-48a0-8c0c-a73e000ade17 None None] MFA Rules not processed for user `070c2441e04a4f2b9d493a9fa58e9537`. Rule list: `[]` (Enabled: `True`). {{(pid=81408) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 13 04:28:41.805564 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.fernet_utils [None req-b78890c0-b9b8-48a0-8c0c-a73e000ade17 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81408) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:28:41.880059 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG dbcounter [-] [81408] Writing DB stats keystone:INSERT=2,keystone:SELECT=98 {{(pid=81408) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:28:41.900321 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 1170/2373] 10.222.0.27 () {58 vars in 958 bytes} [Thu Oct 13 04:28:41 2022] POST /identity/v3/auth/tokens => generated 3325 bytes in 130 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 13 04:28:41.912851 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-39b96792-e6ff-428e-af1a-84903aa3cb8b admin admin] Authenticating user token {{(pid=81409) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:28:41.914598 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.fernet_utils [None req-39b96792-e6ff-428e-af1a-84903aa3cb8b admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81409) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:28:41.973803 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-39b96792-e6ff-428e-af1a-84903aa3cb8b admin admin] Validating token access rules against request {{(pid=81409) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:28:41.977487 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-62fd4e6b-d12f-4944-8628-add85e22a5e1 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6141b8a6c3d54e179d6879bb019f4d12', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dcec0431fb6849499d6aec35a234daa6', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81409) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:28:41.978102 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62fd4e6b-d12f-4944-8628-add85e22a5e1 service cinder] REQUEST_METHOD: `GET` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:28:41.978424 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62fd4e6b-d12f-4944-8628-add85e22a5e1 service cinder] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:28:41.978695 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62fd4e6b-d12f-4944-8628-add85e22a5e1 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:28:41.988661 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-62fd4e6b-d12f-4944-8628-add85e22a5e1 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:28:41.990097 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-62fd4e6b-d12f-4944-8628-add85e22a5e1 service cinder] RBAC: Authorization granted {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:28:42.000604 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 1204/2374] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Oct 13 04:28:41 2022] GET /identity/v3/auth/tokens => generated 3325 bytes in 88 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 13 04:28:42.074509 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b78890c0-b9b8-48a0-8c0c-a73e000ade17 None None] Authenticating user token {{(pid=81408) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:28:42.085087 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b78890c0-b9b8-48a0-8c0c-a73e000ade17 None None] Validating token access rules against request {{(pid=81408) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:28:42.088818 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6d7b6361-7f73-4e5f-ab4a-2f89a0ad5d8a service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6141b8a6c3d54e179d6879bb019f4d12', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dcec0431fb6849499d6aec35a234daa6', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81408) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:28:42.089540 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d7b6361-7f73-4e5f-ab4a-2f89a0ad5d8a service cinder] REQUEST_METHOD: `GET` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:28:42.089841 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d7b6361-7f73-4e5f-ab4a-2f89a0ad5d8a service cinder] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:28:42.090113 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d7b6361-7f73-4e5f-ab4a-2f89a0ad5d8a service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:28:42.151077 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6d7b6361-7f73-4e5f-ab4a-2f89a0ad5d8a service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:28:42.152480 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6d7b6361-7f73-4e5f-ab4a-2f89a0ad5d8a service cinder] RBAC: Authorization granted {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:28:42.163898 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 1171/2375] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Oct 13 04:28:42 2022] GET /identity/v3/auth/tokens => generated 3258 bytes in 90 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 13 04:28:42.442339 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-62fd4e6b-d12f-4944-8628-add85e22a5e1 service cinder] Authenticating user token {{(pid=81409) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:28:42.506050 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-62fd4e6b-d12f-4944-8628-add85e22a5e1 service cinder] Validating token access rules against request {{(pid=81409) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:28:42.510722 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fa1e1dba-1dfd-4b40-b0b6-86e9e85c58fc admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8ba823f313144da3b683bacef8fd62e0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81409) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:28:42.511892 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa1e1dba-1dfd-4b40-b0b6-86e9e85c58fc admin admin] REQUEST_METHOD: `DELETE` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:28:42.512167 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa1e1dba-1dfd-4b40-b0b6-86e9e85c58fc admin admin] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:28:42.512434 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa1e1dba-1dfd-4b40-b0b6-86e9e85c58fc admin admin] PATH_INFO: `/v3/users/32df27ba55e44daf94082073cbd7ddc4` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:28:42.513554 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fa1e1dba-1dfd-4b40-b0b6-86e9e85c58fc admin admin] RBAC: Authorizing `identity:delete_user(user_id=32df27ba55e44daf94082073cbd7ddc4)` {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:28:42.515493 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fa1e1dba-1dfd-4b40-b0b6-86e9e85c58fc admin admin] RBAC: Authorization granted {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:28:42.517848 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG dbcounter [-] [81409] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=81409) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:28:42.561858 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.notifications [None req-fa1e1dba-1dfd-4b40-b0b6-86e9e85c58fc admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '32df27ba55e44daf94082073cbd7ddc4'} {{(pid=81409) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 13 04:28:42.562520 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.notifications [None req-fa1e1dba-1dfd-4b40-b0b6-86e9e85c58fc admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '32df27ba55e44daf94082073cbd7ddc4'} {{(pid=81409) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 13 04:28:42.569261 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.notifications [None req-fa1e1dba-1dfd-4b40-b0b6-86e9e85c58fc admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '32df27ba55e44daf94082073cbd7ddc4'} {{(pid=81409) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 13 04:28:42.569874 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.notifications [None req-fa1e1dba-1dfd-4b40-b0b6-86e9e85c58fc admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '32df27ba55e44daf94082073cbd7ddc4'} {{(pid=81409) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 13 04:28:42.577753 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.notifications [None req-fa1e1dba-1dfd-4b40-b0b6-86e9e85c58fc admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '32df27ba55e44daf94082073cbd7ddc4'} {{(pid=81409) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 13 04:28:42.594968 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 1205/2376] 10.222.0.27 () {58 vars in 1224 bytes} [Thu Oct 13 04:28:42 2022] DELETE /identity/v3/users/32df27ba55e44daf94082073cbd7ddc4 => generated 0 bytes in 153 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 13 04:28:42.599459 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2d3234c-ce34-485f-a705-a47e0a66f87a None None] REQUEST_METHOD: `POST` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:28:42.599741 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2d3234c-ce34-485f-a705-a47e0a66f87a None None] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:28:42.600012 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2d3234c-ce34-485f-a705-a47e0a66f87a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:28:42.620804 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.auth.core [None req-c2d3234c-ce34-485f-a705-a47e0a66f87a None None] MFA Rules not processed for user `0e2d979563d6421995451a0e86bf3022`. Rule list: `[]` (Enabled: `True`). {{(pid=81408) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 13 04:28:42.622160 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.fernet_utils [None req-c2d3234c-ce34-485f-a705-a47e0a66f87a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81408) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:28:42.685242 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 1172/2377] 10.222.0.27 () {58 vars in 958 bytes} [Thu Oct 13 04:28:42 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 87 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 13 04:28:42.694932 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fa1e1dba-1dfd-4b40-b0b6-86e9e85c58fc admin admin] Authenticating user token {{(pid=81409) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:28:42.696336 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.fernet_utils [None req-fa1e1dba-1dfd-4b40-b0b6-86e9e85c58fc admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81409) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:28:42.755249 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fa1e1dba-1dfd-4b40-b0b6-86e9e85c58fc admin admin] Validating token access rules against request {{(pid=81409) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:28:42.759059 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0bbf46eb-1818-4e44-92af-2fa7ffb3e47d service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd8812c3974f84871bb2d761cf0016257', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dcec0431fb6849499d6aec35a234daa6', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81409) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:28:42.759677 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0bbf46eb-1818-4e44-92af-2fa7ffb3e47d service neutron] REQUEST_METHOD: `GET` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:28:42.759952 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0bbf46eb-1818-4e44-92af-2fa7ffb3e47d service neutron] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:28:42.760221 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0bbf46eb-1818-4e44-92af-2fa7ffb3e47d service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:28:42.769385 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0bbf46eb-1818-4e44-92af-2fa7ffb3e47d service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:28:42.770846 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0bbf46eb-1818-4e44-92af-2fa7ffb3e47d service neutron] RBAC: Authorization granted {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:28:42.781574 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 1206/2378] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Oct 13 04:28:42 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 87 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 13 04:28:43.062258 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c2d3234c-ce34-485f-a705-a47e0a66f87a None None] Authenticating user token {{(pid=81408) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:28:43.063804 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.fernet_utils [None req-c2d3234c-ce34-485f-a705-a47e0a66f87a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81408) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:28:43.078728 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c2d3234c-ce34-485f-a705-a47e0a66f87a None None] Validating token access rules against request {{(pid=81408) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:28:43.084510 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-887880c2-5be5-4608-b6dd-1a671866555c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8ba823f313144da3b683bacef8fd62e0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81408) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:28:43.085356 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-887880c2-5be5-4608-b6dd-1a671866555c admin admin] REQUEST_METHOD: `DELETE` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:28:43.085705 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-887880c2-5be5-4608-b6dd-1a671866555c admin admin] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:28:43.086089 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-887880c2-5be5-4608-b6dd-1a671866555c admin admin] PATH_INFO: `/v3/projects/904fd54724a54e0c8fcbe19090ed6229` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:28:43.087436 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-887880c2-5be5-4608-b6dd-1a671866555c admin admin] RBAC: Authorizing `identity:delete_project(project_id=904fd54724a54e0c8fcbe19090ed6229)` {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:28:43.090117 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-887880c2-5be5-4608-b6dd-1a671866555c admin admin] RBAC: Authorization granted {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:28:43.144748 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.notifications [None req-887880c2-5be5-4608-b6dd-1a671866555c admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '904fd54724a54e0c8fcbe19090ed6229', 'request_id': 'req-887880c2-5be5-4608-b6dd-1a671866555c'} {{(pid=81408) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 13 04:28:43.154142 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.notifications [None req-887880c2-5be5-4608-b6dd-1a671866555c admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '904fd54724a54e0c8fcbe19090ed6229', 'request_id': 'req-887880c2-5be5-4608-b6dd-1a671866555c'} {{(pid=81408) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 13 04:28:43.163012 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.notifications [None req-887880c2-5be5-4608-b6dd-1a671866555c admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '904fd54724a54e0c8fcbe19090ed6229', 'request_id': 'req-887880c2-5be5-4608-b6dd-1a671866555c'} {{(pid=81408) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 13 04:28:43.174370 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.notifications [None req-887880c2-5be5-4608-b6dd-1a671866555c admin admin] The token cache is being invalidate because project 904fd54724a54e0c8fcbe19090ed6229 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81408) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 13 04:28:43.174664 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.notifications [None req-887880c2-5be5-4608-b6dd-1a671866555c admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81408) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 13 04:28:43.175325 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.notifications [None req-887880c2-5be5-4608-b6dd-1a671866555c admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81408) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 13 04:28:43.177097 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 1173/2379] 10.222.0.27 () {58 vars in 1233 bytes} [Thu Oct 13 04:28:43 2022] DELETE /identity/v3/projects/904fd54724a54e0c8fcbe19090ed6229 => generated 0 bytes in 115 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 13 04:28:43.180795 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0bbf46eb-1818-4e44-92af-2fa7ffb3e47d service neutron] Authenticating user token {{(pid=81409) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:28:43.191834 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0bbf46eb-1818-4e44-92af-2fa7ffb3e47d service neutron] Validating token access rules against request {{(pid=81409) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:28:43.259240 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-53f36bae-ceb0-407b-88a5-70cbf346a7ad admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8ba823f313144da3b683bacef8fd62e0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81409) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:28:43.259876 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53f36bae-ceb0-407b-88a5-70cbf346a7ad admin admin] REQUEST_METHOD: `DELETE` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:28:43.260136 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53f36bae-ceb0-407b-88a5-70cbf346a7ad admin admin] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:28:43.260437 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53f36bae-ceb0-407b-88a5-70cbf346a7ad admin admin] PATH_INFO: `/v3/users/070c2441e04a4f2b9d493a9fa58e9537` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:28:43.261720 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-53f36bae-ceb0-407b-88a5-70cbf346a7ad admin admin] RBAC: Authorizing `identity:delete_user(user_id=070c2441e04a4f2b9d493a9fa58e9537)` {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:28:43.263720 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-53f36bae-ceb0-407b-88a5-70cbf346a7ad admin admin] RBAC: Authorization granted {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:28:43.295204 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG dbcounter [-] [81409] Writing DB stats keystone:SELECT=85,keystone:DELETE=14,keystone:INSERT=1 {{(pid=81409) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:28:43.323315 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.notifications [None req-53f36bae-ceb0-407b-88a5-70cbf346a7ad admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '070c2441e04a4f2b9d493a9fa58e9537'} {{(pid=81409) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 13 04:28:43.325262 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.notifications [None req-53f36bae-ceb0-407b-88a5-70cbf346a7ad admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '070c2441e04a4f2b9d493a9fa58e9537'} {{(pid=81409) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 13 04:28:43.332222 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.notifications [None req-53f36bae-ceb0-407b-88a5-70cbf346a7ad admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '070c2441e04a4f2b9d493a9fa58e9537'} {{(pid=81409) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 13 04:28:43.332858 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.notifications [None req-53f36bae-ceb0-407b-88a5-70cbf346a7ad admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '070c2441e04a4f2b9d493a9fa58e9537'} {{(pid=81409) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 13 04:28:43.341096 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.notifications [None req-53f36bae-ceb0-407b-88a5-70cbf346a7ad admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '070c2441e04a4f2b9d493a9fa58e9537'} {{(pid=81409) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 13 04:28:43.359944 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 1207/2380] 10.222.0.27 () {58 vars in 1224 bytes} [Thu Oct 13 04:28:43 2022] DELETE /identity/v3/users/070c2441e04a4f2b9d493a9fa58e9537 => generated 0 bytes in 180 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 13 04:28:43.637264 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-887880c2-5be5-4608-b6dd-1a671866555c admin admin] Authenticating user token {{(pid=81408) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:28:43.639050 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.fernet_utils [None req-887880c2-5be5-4608-b6dd-1a671866555c admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81408) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:28:43.657571 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-887880c2-5be5-4608-b6dd-1a671866555c admin admin] Validating token access rules against request {{(pid=81408) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:28:43.667456 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG dbcounter [-] [81408] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=81408) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:28:43.718021 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6b4d5295-58d2-4e84-88cf-e752135075c7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8ba823f313144da3b683bacef8fd62e0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81408) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:28:43.718593 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b4d5295-58d2-4e84-88cf-e752135075c7 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:28:43.718872 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b4d5295-58d2-4e84-88cf-e752135075c7 admin admin] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:28:43.719143 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b4d5295-58d2-4e84-88cf-e752135075c7 admin admin] PATH_INFO: `/v3/projects/1a4f73e6458c4f618ebf32049815d9f3` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:28:43.720256 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6b4d5295-58d2-4e84-88cf-e752135075c7 admin admin] RBAC: Authorizing `identity:delete_project(project_id=1a4f73e6458c4f618ebf32049815d9f3)` {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:28:43.722159 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6b4d5295-58d2-4e84-88cf-e752135075c7 admin admin] RBAC: Authorization granted {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:28:43.767342 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.notifications [None req-6b4d5295-58d2-4e84-88cf-e752135075c7 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '1a4f73e6458c4f618ebf32049815d9f3', 'request_id': 'req-6b4d5295-58d2-4e84-88cf-e752135075c7'} {{(pid=81408) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 13 04:28:43.774363 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.notifications [None req-6b4d5295-58d2-4e84-88cf-e752135075c7 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '1a4f73e6458c4f618ebf32049815d9f3', 'request_id': 'req-6b4d5295-58d2-4e84-88cf-e752135075c7'} {{(pid=81408) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 13 04:28:43.781140 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.notifications [None req-6b4d5295-58d2-4e84-88cf-e752135075c7 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '1a4f73e6458c4f618ebf32049815d9f3', 'request_id': 'req-6b4d5295-58d2-4e84-88cf-e752135075c7'} {{(pid=81408) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 13 04:28:43.791779 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.notifications [None req-6b4d5295-58d2-4e84-88cf-e752135075c7 admin admin] The token cache is being invalidate because project 1a4f73e6458c4f618ebf32049815d9f3 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81408) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 13 04:28:43.792064 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.notifications [None req-6b4d5295-58d2-4e84-88cf-e752135075c7 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81408) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 13 04:28:43.793646 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.notifications [None req-6b4d5295-58d2-4e84-88cf-e752135075c7 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81408) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 13 04:28:43.794993 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 1174/2381] 10.222.0.27 () {58 vars in 1233 bytes} [Thu Oct 13 04:28:43 2022] DELETE /identity/v3/projects/1a4f73e6458c4f618ebf32049815d9f3 => generated 0 bytes in 159 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 13 04:28:43.800139 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6649d760-1e9c-4279-abaf-114f29eed524 None None] REQUEST_METHOD: `POST` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:28:43.800621 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6649d760-1e9c-4279-abaf-114f29eed524 None None] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:28:43.800987 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6649d760-1e9c-4279-abaf-114f29eed524 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:28:43.824922 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.auth.core [None req-6649d760-1e9c-4279-abaf-114f29eed524 None None] MFA Rules not processed for user `0e2d979563d6421995451a0e86bf3022`. Rule list: `[]` (Enabled: `True`). {{(pid=81409) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 13 04:28:43.826225 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.fernet_utils [None req-6649d760-1e9c-4279-abaf-114f29eed524 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81409) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:28:43.884990 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 1208/2382] 10.222.0.27 () {58 vars in 958 bytes} [Thu Oct 13 04:28:43 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 86 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 13 04:28:43.894507 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef3f92e3-8faf-4db0-a897-4d0f723459f3 None None] REQUEST_METHOD: `POST` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:28:43.894797 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef3f92e3-8faf-4db0-a897-4d0f723459f3 None None] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:28:43.895092 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef3f92e3-8faf-4db0-a897-4d0f723459f3 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:28:43.915769 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.auth.core [None req-ef3f92e3-8faf-4db0-a897-4d0f723459f3 None None] MFA Rules not processed for user `0e2d979563d6421995451a0e86bf3022`. Rule list: `[]` (Enabled: `True`). {{(pid=81408) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 13 04:28:43.916983 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.fernet_utils [None req-ef3f92e3-8faf-4db0-a897-4d0f723459f3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81408) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:28:43.925030 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 1175/2383] 10.222.0.27 () {58 vars in 958 bytes} [Thu Oct 13 04:28:43 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 32 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 13 04:28:43.929253 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6649d760-1e9c-4279-abaf-114f29eed524 None None] Authenticating user token {{(pid=81409) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:28:43.939532 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6649d760-1e9c-4279-abaf-114f29eed524 None None] Validating token access rules against request {{(pid=81409) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:28:43.943990 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ca0c34c1-fe93-418f-bc77-7cbcfaa13e8e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8ba823f313144da3b683bacef8fd62e0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81409) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:28:43.944577 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca0c34c1-fe93-418f-bc77-7cbcfaa13e8e admin admin] REQUEST_METHOD: `GET` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:28:43.944849 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca0c34c1-fe93-418f-bc77-7cbcfaa13e8e admin admin] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:28:43.945113 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca0c34c1-fe93-418f-bc77-7cbcfaa13e8e admin admin] PATH_INFO: `/v3/domains` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:28:43.945744 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ca0c34c1-fe93-418f-bc77-7cbcfaa13e8e admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81409) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 13 04:28:43.946039 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ca0c34c1-fe93-418f-bc77-7cbcfaa13e8e admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:28:43.947652 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ca0c34c1-fe93-418f-bc77-7cbcfaa13e8e admin admin] RBAC: Authorization granted {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:28:43.959762 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 1209/2384] 10.222.0.27 () {58 vars in 1153 bytes} [Thu Oct 13 04:28:43 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 13 04:28:43.963867 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ef3f92e3-8faf-4db0-a897-4d0f723459f3 None None] Authenticating user token {{(pid=81408) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:28:43.973429 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ef3f92e3-8faf-4db0-a897-4d0f723459f3 None None] Validating token access rules against request {{(pid=81408) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:28:43.978127 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-736e69f6-f023-4ca7-ae7c-f8eb3b956f5f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8ba823f313144da3b683bacef8fd62e0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81408) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:28:43.978750 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-736e69f6-f023-4ca7-ae7c-f8eb3b956f5f admin admin] REQUEST_METHOD: `POST` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:28:43.979024 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-736e69f6-f023-4ca7-ae7c-f8eb3b956f5f admin admin] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:28:43.979293 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-736e69f6-f023-4ca7-ae7c-f8eb3b956f5f admin admin] PATH_INFO: `/v3/projects` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:28:43.980011 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-736e69f6-f023-4ca7-ae7c-f8eb3b956f5f admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:28:43.982359 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-736e69f6-f023-4ca7-ae7c-f8eb3b956f5f admin admin] RBAC: Authorization granted {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:28:44.009438 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 1176/2385] 10.222.0.27 () {60 vars in 1153 bytes} [Thu Oct 13 04:28:43 2022] POST /identity/v3/projects => generated 375 bytes in 46 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 13 04:28:44.014181 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ca0c34c1-fe93-418f-bc77-7cbcfaa13e8e admin admin] Authenticating user token {{(pid=81409) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:28:44.024060 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ca0c34c1-fe93-418f-bc77-7cbcfaa13e8e admin admin] Validating token access rules against request {{(pid=81409) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:28:44.079637 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ce8a23b1-4fad-4a07-8005-02e6eb0d0816 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8ba823f313144da3b683bacef8fd62e0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81409) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:28:44.080243 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce8a23b1-4fad-4a07-8005-02e6eb0d0816 admin admin] REQUEST_METHOD: `POST` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:28:44.080522 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce8a23b1-4fad-4a07-8005-02e6eb0d0816 admin admin] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:28:44.080797 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce8a23b1-4fad-4a07-8005-02e6eb0d0816 admin admin] PATH_INFO: `/v3/users` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:28:44.081424 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ce8a23b1-4fad-4a07-8005-02e6eb0d0816 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:28:44.083393 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ce8a23b1-4fad-4a07-8005-02e6eb0d0816 admin admin] RBAC: Authorization granted {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:28:44.105291 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 1210/2386] 10.222.0.27 () {60 vars in 1144 bytes} [Thu Oct 13 04:28:44 2022] POST /identity/v3/users => generated 288 bytes in 92 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 13 04:28:44.110172 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-736e69f6-f023-4ca7-ae7c-f8eb3b956f5f admin admin] Authenticating user token {{(pid=81408) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:28:44.121198 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-736e69f6-f023-4ca7-ae7c-f8eb3b956f5f admin admin] Validating token access rules against request {{(pid=81408) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:28:44.126363 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ca93aef1-b9fc-4196-8913-32a01de8ade2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8ba823f313144da3b683bacef8fd62e0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81408) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:28:44.126952 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca93aef1-b9fc-4196-8913-32a01de8ade2 admin admin] REQUEST_METHOD: `GET` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:28:44.127224 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca93aef1-b9fc-4196-8913-32a01de8ade2 admin admin] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:28:44.127491 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca93aef1-b9fc-4196-8913-32a01de8ade2 admin admin] PATH_INFO: `/v3/roles` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:28:44.128117 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ca93aef1-b9fc-4196-8913-32a01de8ade2 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:28:44.129900 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ca93aef1-b9fc-4196-8913-32a01de8ade2 admin admin] RBAC: Authorization granted {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:28:44.141719 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 1177/2387] 10.222.0.27 () {58 vars in 1122 bytes} [Thu Oct 13 04:28:44 2022] GET /identity/v3/roles => generated 1420 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 13 04:28:44.145688 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ce8a23b1-4fad-4a07-8005-02e6eb0d0816 admin admin] Authenticating user token {{(pid=81409) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:28:44.155789 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ce8a23b1-4fad-4a07-8005-02e6eb0d0816 admin admin] Validating token access rules against request {{(pid=81409) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:28:44.160691 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-48023ccc-3319-42c6-9bb1-7f09c2246f7d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8ba823f313144da3b683bacef8fd62e0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81409) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:28:44.161286 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48023ccc-3319-42c6-9bb1-7f09c2246f7d admin admin] REQUEST_METHOD: `GET` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:28:44.161556 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48023ccc-3319-42c6-9bb1-7f09c2246f7d admin admin] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:28:44.161820 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48023ccc-3319-42c6-9bb1-7f09c2246f7d admin admin] PATH_INFO: `/v3/roles` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:28:44.162491 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-48023ccc-3319-42c6-9bb1-7f09c2246f7d admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:28:44.164092 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-48023ccc-3319-42c6-9bb1-7f09c2246f7d admin admin] RBAC: Authorization granted {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:28:44.175037 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 1211/2388] 10.222.0.27 () {58 vars in 1122 bytes} [Thu Oct 13 04:28:44 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 13 04:28:44.179135 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ca93aef1-b9fc-4196-8913-32a01de8ade2 admin admin] Authenticating user token {{(pid=81408) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:28:44.189529 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ca93aef1-b9fc-4196-8913-32a01de8ade2 admin admin] Validating token access rules against request {{(pid=81408) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:28:44.194064 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-03e80ca9-4e22-4914-aefd-82416ec00e39 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8ba823f313144da3b683bacef8fd62e0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81408) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:28:44.194739 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03e80ca9-4e22-4914-aefd-82416ec00e39 admin admin] REQUEST_METHOD: `PUT` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:28:44.195012 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03e80ca9-4e22-4914-aefd-82416ec00e39 admin admin] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:28:44.195279 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03e80ca9-4e22-4914-aefd-82416ec00e39 admin admin] PATH_INFO: `/v3/projects/24069f5c5d4e4fd7a5e68e6f44443119/users/d7242e39b7ca4c29b1505f282d3350f8/roles/cc56c4ae13a7424ea771c15da4b153c1` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:28:44.209493 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-03e80ca9-4e22-4914-aefd-82416ec00e39 admin admin] RBAC: Authorizing `identity:create_grant(project_id=24069f5c5d4e4fd7a5e68e6f44443119, user_id=d7242e39b7ca4c29b1505f282d3350f8, role_id=cc56c4ae13a7424ea771c15da4b153c1)` {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:28:44.212165 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-03e80ca9-4e22-4914-aefd-82416ec00e39 admin admin] RBAC: Authorization granted {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:28:44.225410 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 1178/2389] 10.222.0.27 () {60 vars in 1483 bytes} [Thu Oct 13 04:28:44 2022] PUT /identity/v3/projects/24069f5c5d4e4fd7a5e68e6f44443119/users/d7242e39b7ca4c29b1505f282d3350f8/roles/cc56c4ae13a7424ea771c15da4b153c1 => generated 0 bytes in 47 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 13 04:28:44.237132 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5911aa71-f6ed-4cb9-b438-77dbd054addf None None] REQUEST_METHOD: `POST` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:28:44.237442 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5911aa71-f6ed-4cb9-b438-77dbd054addf None None] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:28:44.237743 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5911aa71-f6ed-4cb9-b438-77dbd054addf None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:28:44.256845 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG dbcounter [-] [81409] Writing DB stats keystone:DELETE=6,keystone:SELECT=90,keystone:INSERT=4 {{(pid=81409) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:28:44.267493 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.auth.core [None req-5911aa71-f6ed-4cb9-b438-77dbd054addf None None] MFA Rules not processed for user `d7242e39b7ca4c29b1505f282d3350f8`. Rule list: `[]` (Enabled: `True`). {{(pid=81409) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 13 04:28:44.268993 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.fernet_utils [None req-5911aa71-f6ed-4cb9-b438-77dbd054addf None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81409) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:28:44.333903 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 1212/2390] 10.222.0.27 () {58 vars in 958 bytes} [Thu Oct 13 04:28:44 2022] POST /identity/v3/auth/tokens => generated 3252 bytes in 98 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 13 04:28:44.345700 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-03e80ca9-4e22-4914-aefd-82416ec00e39 admin admin] Authenticating user token {{(pid=81408) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:28:44.347484 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.fernet_utils [None req-03e80ca9-4e22-4914-aefd-82416ec00e39 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81408) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:28:44.395618 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG dbcounter [-] [81408] Writing DB stats keystone:SELECT=90,keystone:DELETE=7,keystone:INSERT=3 {{(pid=81408) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:28:44.414722 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-03e80ca9-4e22-4914-aefd-82416ec00e39 admin admin] Validating token access rules against request {{(pid=81408) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:28:44.419331 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-155e4bc8-f03b-43cc-bbe5-e09b5ab7a593 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6141b8a6c3d54e179d6879bb019f4d12', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dcec0431fb6849499d6aec35a234daa6', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81408) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:28:44.420044 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-155e4bc8-f03b-43cc-bbe5-e09b5ab7a593 service cinder] REQUEST_METHOD: `GET` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:28:44.420216 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-155e4bc8-f03b-43cc-bbe5-e09b5ab7a593 service cinder] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:28:44.420489 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-155e4bc8-f03b-43cc-bbe5-e09b5ab7a593 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:28:44.430235 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-155e4bc8-f03b-43cc-bbe5-e09b5ab7a593 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:28:44.431664 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-155e4bc8-f03b-43cc-bbe5-e09b5ab7a593 service cinder] RBAC: Authorization granted {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:28:44.441163 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 1179/2391] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Oct 13 04:28:44 2022] GET /identity/v3/auth/tokens => generated 3252 bytes in 96 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 13 04:28:54.330768 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG dbcounter [-] [81409] Writing DB stats keystone:SELECT=28 {{(pid=81409) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:28:54.436988 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG dbcounter [-] [81408] Writing DB stats keystone:SELECT=9 {{(pid=81408) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:28:56.840925 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5911aa71-f6ed-4cb9-b438-77dbd054addf None None] Authenticating user token {{(pid=81409) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:28:56.842418 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.fernet_utils [None req-5911aa71-f6ed-4cb9-b438-77dbd054addf None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81409) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:28:56.912578 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5911aa71-f6ed-4cb9-b438-77dbd054addf None None] Validating token access rules against request {{(pid=81409) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:28:56.917018 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0a2775c0-7e02-4d1f-b68c-c5cb54d88c15 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8ba823f313144da3b683bacef8fd62e0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81409) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:28:56.917638 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a2775c0-7e02-4d1f-b68c-c5cb54d88c15 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:28:56.917929 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a2775c0-7e02-4d1f-b68c-c5cb54d88c15 admin admin] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:28:56.918859 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a2775c0-7e02-4d1f-b68c-c5cb54d88c15 admin admin] PATH_INFO: `/v3/users/6382f287013d4cd4b8201ec3a9103d22` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:28:56.920002 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0a2775c0-7e02-4d1f-b68c-c5cb54d88c15 admin admin] RBAC: Authorizing `identity:delete_user(user_id=6382f287013d4cd4b8201ec3a9103d22)` {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:28:56.921922 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0a2775c0-7e02-4d1f-b68c-c5cb54d88c15 admin admin] RBAC: Authorization granted {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:28:56.966804 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.notifications [None req-0a2775c0-7e02-4d1f-b68c-c5cb54d88c15 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '6382f287013d4cd4b8201ec3a9103d22'} {{(pid=81409) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 13 04:28:56.967426 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.notifications [None req-0a2775c0-7e02-4d1f-b68c-c5cb54d88c15 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '6382f287013d4cd4b8201ec3a9103d22'} {{(pid=81409) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 13 04:28:56.974250 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.notifications [None req-0a2775c0-7e02-4d1f-b68c-c5cb54d88c15 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '6382f287013d4cd4b8201ec3a9103d22'} {{(pid=81409) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 13 04:28:56.974941 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.notifications [None req-0a2775c0-7e02-4d1f-b68c-c5cb54d88c15 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '6382f287013d4cd4b8201ec3a9103d22'} {{(pid=81409) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 13 04:28:56.982749 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.notifications [None req-0a2775c0-7e02-4d1f-b68c-c5cb54d88c15 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '6382f287013d4cd4b8201ec3a9103d22'} {{(pid=81409) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 13 04:28:57.000017 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 1213/2392] 10.222.0.27 () {58 vars in 1224 bytes} [Thu Oct 13 04:28:56 2022] DELETE /identity/v3/users/6382f287013d4cd4b8201ec3a9103d22 => generated 0 bytes in 160 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 13 04:28:57.004528 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4def3fb-fbda-4e14-9d97-91f3519713d1 None None] REQUEST_METHOD: `POST` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:28:57.005014 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4def3fb-fbda-4e14-9d97-91f3519713d1 None None] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:28:57.005116 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4def3fb-fbda-4e14-9d97-91f3519713d1 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:28:57.024365 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.auth.core [None req-c4def3fb-fbda-4e14-9d97-91f3519713d1 None None] MFA Rules not processed for user `0e2d979563d6421995451a0e86bf3022`. Rule list: `[]` (Enabled: `True`). {{(pid=81408) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 13 04:28:57.025579 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.fernet_utils [None req-c4def3fb-fbda-4e14-9d97-91f3519713d1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81408) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:28:57.083174 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 1180/2393] 10.222.0.27 () {58 vars in 958 bytes} [Thu Oct 13 04:28:57 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 80 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 13 04:28:57.093517 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0a2775c0-7e02-4d1f-b68c-c5cb54d88c15 admin admin] Authenticating user token {{(pid=81409) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:28:57.094975 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.fernet_utils [None req-0a2775c0-7e02-4d1f-b68c-c5cb54d88c15 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81409) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:28:57.153252 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0a2775c0-7e02-4d1f-b68c-c5cb54d88c15 admin admin] Validating token access rules against request {{(pid=81409) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:28:57.156981 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5c5cbcdc-dccb-4761-8c03-388750e7c4de service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd8812c3974f84871bb2d761cf0016257', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dcec0431fb6849499d6aec35a234daa6', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81409) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:28:57.157571 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c5cbcdc-dccb-4761-8c03-388750e7c4de service neutron] REQUEST_METHOD: `GET` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:28:57.157843 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c5cbcdc-dccb-4761-8c03-388750e7c4de service neutron] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:28:57.158109 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c5cbcdc-dccb-4761-8c03-388750e7c4de service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:28:57.167147 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5c5cbcdc-dccb-4761-8c03-388750e7c4de service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:28:57.168825 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5c5cbcdc-dccb-4761-8c03-388750e7c4de service neutron] RBAC: Authorization granted {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:28:57.178339 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 1214/2394] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Oct 13 04:28:57 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 85 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 13 04:28:57.447057 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c4def3fb-fbda-4e14-9d97-91f3519713d1 None None] Authenticating user token {{(pid=81408) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:28:57.448617 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.fernet_utils [None req-c4def3fb-fbda-4e14-9d97-91f3519713d1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81408) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:28:57.461284 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c4def3fb-fbda-4e14-9d97-91f3519713d1 None None] Validating token access rules against request {{(pid=81408) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:28:57.466676 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b25cb537-5f60-4862-8773-2b9ee19c099d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8ba823f313144da3b683bacef8fd62e0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81408) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:28:57.467376 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b25cb537-5f60-4862-8773-2b9ee19c099d admin admin] REQUEST_METHOD: `DELETE` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:28:57.467719 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b25cb537-5f60-4862-8773-2b9ee19c099d admin admin] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:28:57.467998 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b25cb537-5f60-4862-8773-2b9ee19c099d admin admin] PATH_INFO: `/v3/projects/4df9b6305d2d47bd98de154e629c596f` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:28:57.469126 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b25cb537-5f60-4862-8773-2b9ee19c099d admin admin] RBAC: Authorizing `identity:delete_project(project_id=4df9b6305d2d47bd98de154e629c596f)` {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:28:57.471098 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b25cb537-5f60-4862-8773-2b9ee19c099d admin admin] RBAC: Authorization granted {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:28:57.517622 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.notifications [None req-b25cb537-5f60-4862-8773-2b9ee19c099d admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '4df9b6305d2d47bd98de154e629c596f', 'request_id': 'req-b25cb537-5f60-4862-8773-2b9ee19c099d'} {{(pid=81408) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 13 04:28:57.524532 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.notifications [None req-b25cb537-5f60-4862-8773-2b9ee19c099d admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '4df9b6305d2d47bd98de154e629c596f', 'request_id': 'req-b25cb537-5f60-4862-8773-2b9ee19c099d'} {{(pid=81408) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 13 04:28:57.531494 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.notifications [None req-b25cb537-5f60-4862-8773-2b9ee19c099d admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '4df9b6305d2d47bd98de154e629c596f', 'request_id': 'req-b25cb537-5f60-4862-8773-2b9ee19c099d'} {{(pid=81408) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 13 04:28:57.541563 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.notifications [None req-b25cb537-5f60-4862-8773-2b9ee19c099d admin admin] The token cache is being invalidate because project 4df9b6305d2d47bd98de154e629c596f was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81408) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 13 04:28:57.541848 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.notifications [None req-b25cb537-5f60-4862-8773-2b9ee19c099d admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81408) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 13 04:28:57.542506 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.notifications [None req-b25cb537-5f60-4862-8773-2b9ee19c099d admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81408) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 13 04:28:57.544892 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 1181/2395] 10.222.0.27 () {58 vars in 1233 bytes} [Thu Oct 13 04:28:57 2022] DELETE /identity/v3/projects/4df9b6305d2d47bd98de154e629c596f => generated 0 bytes in 98 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 13 04:28:57.550860 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c828ebf-ad65-4a6e-992e-2041cc600f4d None None] REQUEST_METHOD: `POST` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:28:57.551236 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c828ebf-ad65-4a6e-992e-2041cc600f4d None None] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:28:57.551509 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c828ebf-ad65-4a6e-992e-2041cc600f4d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:28:57.570547 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.auth.core [None req-3c828ebf-ad65-4a6e-992e-2041cc600f4d None None] MFA Rules not processed for user `0e2d979563d6421995451a0e86bf3022`. Rule list: `[]` (Enabled: `True`). {{(pid=81409) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 13 04:28:57.571743 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.fernet_utils [None req-3c828ebf-ad65-4a6e-992e-2041cc600f4d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81409) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:28:57.588114 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG dbcounter [-] [81409] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81409) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:28:57.632210 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 1215/2396] 10.222.0.27 () {58 vars in 958 bytes} [Thu Oct 13 04:28:57 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 82 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 13 04:28:57.641486 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63eb25b9-0b28-4042-a7bd-5cc8f9dd8da4 None None] REQUEST_METHOD: `POST` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:28:57.641775 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63eb25b9-0b28-4042-a7bd-5cc8f9dd8da4 None None] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:28:57.642052 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63eb25b9-0b28-4042-a7bd-5cc8f9dd8da4 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:28:57.662617 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.auth.core [None req-63eb25b9-0b28-4042-a7bd-5cc8f9dd8da4 None None] MFA Rules not processed for user `0e2d979563d6421995451a0e86bf3022`. Rule list: `[]` (Enabled: `True`). {{(pid=81408) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 13 04:28:57.663832 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.fernet_utils [None req-63eb25b9-0b28-4042-a7bd-5cc8f9dd8da4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81408) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:28:57.672351 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 1182/2397] 10.222.0.27 () {58 vars in 958 bytes} [Thu Oct 13 04:28:57 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 32 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 13 04:28:57.676807 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3c828ebf-ad65-4a6e-992e-2041cc600f4d None None] Authenticating user token {{(pid=81409) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:28:57.687452 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3c828ebf-ad65-4a6e-992e-2041cc600f4d None None] Validating token access rules against request {{(pid=81409) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:28:57.691948 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4f6be4d3-3abe-42e2-abdf-834eea142c85 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8ba823f313144da3b683bacef8fd62e0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81409) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:28:57.692547 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f6be4d3-3abe-42e2-abdf-834eea142c85 admin admin] REQUEST_METHOD: `GET` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:28:57.692822 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f6be4d3-3abe-42e2-abdf-834eea142c85 admin admin] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:28:57.693091 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f6be4d3-3abe-42e2-abdf-834eea142c85 admin admin] PATH_INFO: `/v3/domains` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:28:57.693730 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4f6be4d3-3abe-42e2-abdf-834eea142c85 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81409) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 13 04:28:57.694033 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4f6be4d3-3abe-42e2-abdf-834eea142c85 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:28:57.695646 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4f6be4d3-3abe-42e2-abdf-834eea142c85 admin admin] RBAC: Authorization granted {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:28:57.707767 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 1216/2398] 10.222.0.27 () {58 vars in 1153 bytes} [Thu Oct 13 04:28:57 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 13 04:28:57.712096 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-63eb25b9-0b28-4042-a7bd-5cc8f9dd8da4 None None] Authenticating user token {{(pid=81408) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:28:57.721289 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-63eb25b9-0b28-4042-a7bd-5cc8f9dd8da4 None None] Validating token access rules against request {{(pid=81408) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:28:57.726014 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4eadee16-4c4e-4680-8e4e-d7142837a5f6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8ba823f313144da3b683bacef8fd62e0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81408) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:28:57.726650 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4eadee16-4c4e-4680-8e4e-d7142837a5f6 admin admin] REQUEST_METHOD: `POST` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:28:57.726927 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4eadee16-4c4e-4680-8e4e-d7142837a5f6 admin admin] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:28:57.727198 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4eadee16-4c4e-4680-8e4e-d7142837a5f6 admin admin] PATH_INFO: `/v3/projects` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:28:57.727831 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4eadee16-4c4e-4680-8e4e-d7142837a5f6 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:28:57.729746 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4eadee16-4c4e-4680-8e4e-d7142837a5f6 admin admin] RBAC: Authorization granted {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:28:57.757014 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 1183/2399] 10.222.0.27 () {60 vars in 1153 bytes} [Thu Oct 13 04:28:57 2022] POST /identity/v3/projects => generated 383 bytes in 45 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 13 04:28:57.761100 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4f6be4d3-3abe-42e2-abdf-834eea142c85 admin admin] Authenticating user token {{(pid=81409) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:28:57.771930 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4f6be4d3-3abe-42e2-abdf-834eea142c85 admin admin] Validating token access rules against request {{(pid=81409) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:28:57.827819 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-40cf9155-eb56-4a61-b4cb-98fc02389533 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8ba823f313144da3b683bacef8fd62e0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81409) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:28:57.828425 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40cf9155-eb56-4a61-b4cb-98fc02389533 admin admin] REQUEST_METHOD: `POST` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:28:57.828702 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40cf9155-eb56-4a61-b4cb-98fc02389533 admin admin] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:28:57.828974 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40cf9155-eb56-4a61-b4cb-98fc02389533 admin admin] PATH_INFO: `/v3/users` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:28:57.829611 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-40cf9155-eb56-4a61-b4cb-98fc02389533 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:28:57.831564 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-40cf9155-eb56-4a61-b4cb-98fc02389533 admin admin] RBAC: Authorization granted {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:28:57.852053 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 1217/2400] 10.222.0.27 () {60 vars in 1144 bytes} [Thu Oct 13 04:28:57 2022] POST /identity/v3/users => generated 292 bytes in 91 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 13 04:28:57.859699 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4eadee16-4c4e-4680-8e4e-d7142837a5f6 admin admin] Authenticating user token {{(pid=81408) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:28:57.869898 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4eadee16-4c4e-4680-8e4e-d7142837a5f6 admin admin] Validating token access rules against request {{(pid=81408) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:28:57.875826 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5a313733-5ebe-4806-8596-5033ddbc7013 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8ba823f313144da3b683bacef8fd62e0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81408) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:28:57.876424 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a313733-5ebe-4806-8596-5033ddbc7013 admin admin] REQUEST_METHOD: `GET` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:28:57.876727 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a313733-5ebe-4806-8596-5033ddbc7013 admin admin] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:28:57.877251 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a313733-5ebe-4806-8596-5033ddbc7013 admin admin] PATH_INFO: `/v3/roles` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:28:57.877617 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5a313733-5ebe-4806-8596-5033ddbc7013 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:28:57.879230 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5a313733-5ebe-4806-8596-5033ddbc7013 admin admin] RBAC: Authorization granted {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:28:57.890185 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 1184/2401] 10.222.0.27 () {58 vars in 1122 bytes} [Thu Oct 13 04:28:57 2022] GET /identity/v3/roles => generated 1420 bytes in 34 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 13 04:28:57.894103 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-40cf9155-eb56-4a61-b4cb-98fc02389533 admin admin] Authenticating user token {{(pid=81409) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:28:57.904051 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-40cf9155-eb56-4a61-b4cb-98fc02389533 admin admin] Validating token access rules against request {{(pid=81409) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:28:57.908572 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cba4418f-58ac-46e2-ac05-714137ca50cc admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8ba823f313144da3b683bacef8fd62e0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81409) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:28:57.909161 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cba4418f-58ac-46e2-ac05-714137ca50cc admin admin] REQUEST_METHOD: `GET` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:28:57.909436 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cba4418f-58ac-46e2-ac05-714137ca50cc admin admin] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:28:57.909707 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cba4418f-58ac-46e2-ac05-714137ca50cc admin admin] PATH_INFO: `/v3/roles` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:28:57.910373 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cba4418f-58ac-46e2-ac05-714137ca50cc admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:28:57.911960 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cba4418f-58ac-46e2-ac05-714137ca50cc admin admin] RBAC: Authorization granted {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:28:57.924173 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 1218/2402] 10.222.0.27 () {58 vars in 1122 bytes} [Thu Oct 13 04:28:57 2022] GET /identity/v3/roles => generated 1420 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 13 04:28:57.928450 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5a313733-5ebe-4806-8596-5033ddbc7013 admin admin] Authenticating user token {{(pid=81408) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:28:57.939441 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5a313733-5ebe-4806-8596-5033ddbc7013 admin admin] Validating token access rules against request {{(pid=81408) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:28:57.944298 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7bef3713-a027-4673-8ba2-0e88c85a1a06 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8ba823f313144da3b683bacef8fd62e0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81408) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:28:57.945128 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7bef3713-a027-4673-8ba2-0e88c85a1a06 admin admin] REQUEST_METHOD: `PUT` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:28:57.945410 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7bef3713-a027-4673-8ba2-0e88c85a1a06 admin admin] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:28:57.945689 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7bef3713-a027-4673-8ba2-0e88c85a1a06 admin admin] PATH_INFO: `/v3/projects/7f78c358e5c0423d9ba9520c90f163b5/users/4efdcf15e0e448e2a8d651555f4de29a/roles/cc56c4ae13a7424ea771c15da4b153c1` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:28:57.960674 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7bef3713-a027-4673-8ba2-0e88c85a1a06 admin admin] RBAC: Authorizing `identity:create_grant(project_id=7f78c358e5c0423d9ba9520c90f163b5, user_id=4efdcf15e0e448e2a8d651555f4de29a, role_id=cc56c4ae13a7424ea771c15da4b153c1)` {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:28:57.963366 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7bef3713-a027-4673-8ba2-0e88c85a1a06 admin admin] RBAC: Authorization granted {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:28:57.976103 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 1185/2403] 10.222.0.27 () {60 vars in 1483 bytes} [Thu Oct 13 04:28:57 2022] PUT /identity/v3/projects/7f78c358e5c0423d9ba9520c90f163b5/users/4efdcf15e0e448e2a8d651555f4de29a/roles/cc56c4ae13a7424ea771c15da4b153c1 => generated 0 bytes in 48 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 13 04:28:57.981548 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3841d878-a80b-4f0c-b07a-8585203e6b88 None None] REQUEST_METHOD: `POST` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:28:57.981832 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3841d878-a80b-4f0c-b07a-8585203e6b88 None None] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:28:57.982106 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3841d878-a80b-4f0c-b07a-8585203e6b88 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:28:58.000693 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.auth.core [None req-3841d878-a80b-4f0c-b07a-8585203e6b88 None None] MFA Rules not processed for user `0e2d979563d6421995451a0e86bf3022`. Rule list: `[]` (Enabled: `True`). {{(pid=81409) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 13 04:28:58.001893 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.fernet_utils [None req-3841d878-a80b-4f0c-b07a-8585203e6b88 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81409) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:28:58.060127 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 1219/2404] 10.222.0.27 () {58 vars in 958 bytes} [Thu Oct 13 04:28:57 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 80 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 13 04:28:58.069787 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7bef3713-a027-4673-8ba2-0e88c85a1a06 admin admin] Authenticating user token {{(pid=81408) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:28:58.071490 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.fernet_utils [None req-7bef3713-a027-4673-8ba2-0e88c85a1a06 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81408) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:28:58.095072 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG dbcounter [-] [81408] Writing DB stats keystone:SELECT=90,keystone:DELETE=7,keystone:INSERT=3 {{(pid=81408) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:28:58.130929 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7bef3713-a027-4673-8ba2-0e88c85a1a06 admin admin] Validating token access rules against request {{(pid=81408) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:28:58.134906 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1b0fedcf-b65c-4838-8ce2-5f3dd8f0c459 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd8812c3974f84871bb2d761cf0016257', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dcec0431fb6849499d6aec35a234daa6', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81408) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:28:58.135510 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b0fedcf-b65c-4838-8ce2-5f3dd8f0c459 service neutron] REQUEST_METHOD: `GET` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:28:58.135788 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b0fedcf-b65c-4838-8ce2-5f3dd8f0c459 service neutron] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:28:58.136059 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b0fedcf-b65c-4838-8ce2-5f3dd8f0c459 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:28:58.145828 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1b0fedcf-b65c-4838-8ce2-5f3dd8f0c459 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:28:58.147227 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1b0fedcf-b65c-4838-8ce2-5f3dd8f0c459 service neutron] RBAC: Authorization granted {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:28:58.157392 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 1186/2405] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Oct 13 04:28:58 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 88 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 13 04:28:58.823927 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-449ded7f-a988-4c49-bf33-f6ff2443a9a4 None None] REQUEST_METHOD: `POST` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:28:58.824988 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-449ded7f-a988-4c49-bf33-f6ff2443a9a4 None None] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:28:58.825100 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-449ded7f-a988-4c49-bf33-f6ff2443a9a4 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:28:58.864394 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.auth.core [None req-449ded7f-a988-4c49-bf33-f6ff2443a9a4 None None] MFA Rules not processed for user `4efdcf15e0e448e2a8d651555f4de29a`. Rule list: `[]` (Enabled: `True`). {{(pid=81409) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 13 04:28:58.865635 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.fernet_utils [None req-449ded7f-a988-4c49-bf33-f6ff2443a9a4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81409) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:28:58.875152 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG dbcounter [-] [81409] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81409) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:28:58.965848 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 1220/2406] 10.222.0.27 () {58 vars in 958 bytes} [Thu Oct 13 04:28:58 2022] POST /identity/v3/auth/tokens => generated 3260 bytes in 143 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 13 04:28:58.976430 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1b0fedcf-b65c-4838-8ce2-5f3dd8f0c459 service neutron] Authenticating user token {{(pid=81408) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:28:58.977804 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.fernet_utils [None req-1b0fedcf-b65c-4838-8ce2-5f3dd8f0c459 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81408) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:28:59.037399 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1b0fedcf-b65c-4838-8ce2-5f3dd8f0c459 service neutron] Validating token access rules against request {{(pid=81408) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:28:59.041121 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-857441ce-d7bb-453d-8890-d52b08bf0c74 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6141b8a6c3d54e179d6879bb019f4d12', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dcec0431fb6849499d6aec35a234daa6', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81408) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:28:59.041721 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-857441ce-d7bb-453d-8890-d52b08bf0c74 service cinder] REQUEST_METHOD: `GET` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:28:59.042000 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-857441ce-d7bb-453d-8890-d52b08bf0c74 service cinder] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:28:59.042302 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-857441ce-d7bb-453d-8890-d52b08bf0c74 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:28:59.051606 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-857441ce-d7bb-453d-8890-d52b08bf0c74 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:28:59.053038 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-857441ce-d7bb-453d-8890-d52b08bf0c74 service cinder] RBAC: Authorization granted {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:28:59.061833 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 1187/2407] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Oct 13 04:28:58 2022] GET /identity/v3/auth/tokens => generated 3260 bytes in 86 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 13 04:29:00.421126 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c3a3e7a1-79bb-4596-a7bd-272a5c7c055a None None] REQUEST_METHOD: `POST` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:29:00.421563 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c3a3e7a1-79bb-4596-a7bd-272a5c7c055a None None] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:29:00.421710 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c3a3e7a1-79bb-4596-a7bd-272a5c7c055a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:29:00.441547 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.auth.core [None req-c3a3e7a1-79bb-4596-a7bd-272a5c7c055a None None] MFA Rules not processed for user `4efdcf15e0e448e2a8d651555f4de29a`. Rule list: `[]` (Enabled: `True`). {{(pid=81409) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 13 04:29:00.442784 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.fernet_utils [None req-c3a3e7a1-79bb-4596-a7bd-272a5c7c055a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81409) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:29:00.450524 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 1221/2408] 10.222.0.27 () {58 vars in 958 bytes} [Thu Oct 13 04:29:00 2022] POST /identity/v3/auth/tokens => generated 3260 bytes in 30 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 13 04:29:09.057789 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG dbcounter [-] [81408] Writing DB stats keystone:SELECT=45 {{(pid=81408) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:29:10.436797 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG dbcounter [-] [81409] Writing DB stats keystone:SELECT=34 {{(pid=81409) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:29:11.737742 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-857441ce-d7bb-453d-8890-d52b08bf0c74 service cinder] Authenticating user token {{(pid=81408) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:29:11.739530 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.fernet_utils [None req-857441ce-d7bb-453d-8890-d52b08bf0c74 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81408) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:29:11.798329 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-857441ce-d7bb-453d-8890-d52b08bf0c74 service cinder] Validating token access rules against request {{(pid=81408) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:29:11.803093 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fe8b6457-aa56-46e9-b69b-b90c39b72a28 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8ba823f313144da3b683bacef8fd62e0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81408) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:29:11.803729 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe8b6457-aa56-46e9-b69b-b90c39b72a28 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:29:11.804009 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe8b6457-aa56-46e9-b69b-b90c39b72a28 admin admin] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:29:11.804334 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe8b6457-aa56-46e9-b69b-b90c39b72a28 admin admin] PATH_INFO: `/v3/users/d7242e39b7ca4c29b1505f282d3350f8` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:29:11.805429 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fe8b6457-aa56-46e9-b69b-b90c39b72a28 admin admin] RBAC: Authorizing `identity:delete_user(user_id=d7242e39b7ca4c29b1505f282d3350f8)` {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:29:11.807414 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fe8b6457-aa56-46e9-b69b-b90c39b72a28 admin admin] RBAC: Authorization granted {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:29:11.858678 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.notifications [None req-fe8b6457-aa56-46e9-b69b-b90c39b72a28 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'd7242e39b7ca4c29b1505f282d3350f8'} {{(pid=81408) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 13 04:29:11.866607 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.notifications [None req-fe8b6457-aa56-46e9-b69b-b90c39b72a28 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'd7242e39b7ca4c29b1505f282d3350f8'} {{(pid=81408) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 13 04:29:11.867525 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.notifications [None req-fe8b6457-aa56-46e9-b69b-b90c39b72a28 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'd7242e39b7ca4c29b1505f282d3350f8'} {{(pid=81408) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 13 04:29:11.868132 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.notifications [None req-fe8b6457-aa56-46e9-b69b-b90c39b72a28 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'd7242e39b7ca4c29b1505f282d3350f8'} {{(pid=81408) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 13 04:29:11.877570 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.notifications [None req-fe8b6457-aa56-46e9-b69b-b90c39b72a28 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'd7242e39b7ca4c29b1505f282d3350f8'} {{(pid=81408) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 13 04:29:11.895952 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 1188/2409] 10.222.0.27 () {58 vars in 1224 bytes} [Thu Oct 13 04:29:11 2022] DELETE /identity/v3/users/d7242e39b7ca4c29b1505f282d3350f8 => generated 0 bytes in 159 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 13 04:29:11.900120 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31097e3c-25de-4ed8-bce7-e61cdc678a10 None None] REQUEST_METHOD: `POST` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:29:11.900649 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31097e3c-25de-4ed8-bce7-e61cdc678a10 None None] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:29:11.900649 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31097e3c-25de-4ed8-bce7-e61cdc678a10 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:29:11.931889 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.auth.core [None req-31097e3c-25de-4ed8-bce7-e61cdc678a10 None None] MFA Rules not processed for user `0e2d979563d6421995451a0e86bf3022`. Rule list: `[]` (Enabled: `True`). {{(pid=81409) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 13 04:29:11.933371 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.fernet_utils [None req-31097e3c-25de-4ed8-bce7-e61cdc678a10 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81409) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:29:12.008160 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 1222/2410] 10.222.0.27 () {58 vars in 958 bytes} [Thu Oct 13 04:29:11 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 109 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 13 04:29:12.017947 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fe8b6457-aa56-46e9-b69b-b90c39b72a28 admin admin] Authenticating user token {{(pid=81408) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:29:12.019459 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.fernet_utils [None req-fe8b6457-aa56-46e9-b69b-b90c39b72a28 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81408) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:29:12.095106 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fe8b6457-aa56-46e9-b69b-b90c39b72a28 admin admin] Validating token access rules against request {{(pid=81408) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:29:12.098744 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-efe38d77-0ab2-4246-a89d-e913fd37d92a service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd8812c3974f84871bb2d761cf0016257', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dcec0431fb6849499d6aec35a234daa6', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81408) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:29:12.099344 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-efe38d77-0ab2-4246-a89d-e913fd37d92a service neutron] REQUEST_METHOD: `GET` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:29:12.099622 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-efe38d77-0ab2-4246-a89d-e913fd37d92a service neutron] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:29:12.099893 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-efe38d77-0ab2-4246-a89d-e913fd37d92a service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:29:12.109018 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-efe38d77-0ab2-4246-a89d-e913fd37d92a service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:29:12.110517 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-efe38d77-0ab2-4246-a89d-e913fd37d92a service neutron] RBAC: Authorization granted {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:29:12.119797 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 1189/2411] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Oct 13 04:29:12 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 102 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 13 04:29:12.387622 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-31097e3c-25de-4ed8-bce7-e61cdc678a10 None None] Authenticating user token {{(pid=81409) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:29:12.389061 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.fernet_utils [None req-31097e3c-25de-4ed8-bce7-e61cdc678a10 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81409) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:29:12.400837 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-31097e3c-25de-4ed8-bce7-e61cdc678a10 None None] Validating token access rules against request {{(pid=81409) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:29:12.406160 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e3396bdf-28bb-4421-9962-59ecee53b6d5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8ba823f313144da3b683bacef8fd62e0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81409) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:29:12.406821 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3396bdf-28bb-4421-9962-59ecee53b6d5 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:29:12.407102 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3396bdf-28bb-4421-9962-59ecee53b6d5 admin admin] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:29:12.407372 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3396bdf-28bb-4421-9962-59ecee53b6d5 admin admin] PATH_INFO: `/v3/projects/24069f5c5d4e4fd7a5e68e6f44443119` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:29:12.417432 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e3396bdf-28bb-4421-9962-59ecee53b6d5 admin admin] RBAC: Authorizing `identity:delete_project(project_id=24069f5c5d4e4fd7a5e68e6f44443119)` {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:29:12.419562 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e3396bdf-28bb-4421-9962-59ecee53b6d5 admin admin] RBAC: Authorization granted {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:29:12.468698 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.notifications [None req-e3396bdf-28bb-4421-9962-59ecee53b6d5 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '24069f5c5d4e4fd7a5e68e6f44443119', 'request_id': 'req-e3396bdf-28bb-4421-9962-59ecee53b6d5'} {{(pid=81409) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 13 04:29:12.477586 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.notifications [None req-e3396bdf-28bb-4421-9962-59ecee53b6d5 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '24069f5c5d4e4fd7a5e68e6f44443119', 'request_id': 'req-e3396bdf-28bb-4421-9962-59ecee53b6d5'} {{(pid=81409) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 13 04:29:12.485195 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.notifications [None req-e3396bdf-28bb-4421-9962-59ecee53b6d5 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '24069f5c5d4e4fd7a5e68e6f44443119', 'request_id': 'req-e3396bdf-28bb-4421-9962-59ecee53b6d5'} {{(pid=81409) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 13 04:29:12.496385 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.notifications [None req-e3396bdf-28bb-4421-9962-59ecee53b6d5 admin admin] The token cache is being invalidate because project 24069f5c5d4e4fd7a5e68e6f44443119 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81409) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 13 04:29:12.496791 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.notifications [None req-e3396bdf-28bb-4421-9962-59ecee53b6d5 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81409) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 13 04:29:12.498012 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.notifications [None req-e3396bdf-28bb-4421-9962-59ecee53b6d5 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81409) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 13 04:29:12.499548 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 1223/2412] 10.222.0.27 () {58 vars in 1233 bytes} [Thu Oct 13 04:29:12 2022] DELETE /identity/v3/projects/24069f5c5d4e4fd7a5e68e6f44443119 => generated 0 bytes in 113 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 13 04:29:12.505508 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e3e42bb-1fb9-4401-a07d-038b7b8baf55 None None] REQUEST_METHOD: `POST` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:29:12.505915 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e3e42bb-1fb9-4401-a07d-038b7b8baf55 None None] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:29:12.506076 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e3e42bb-1fb9-4401-a07d-038b7b8baf55 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:29:12.546580 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG dbcounter [-] [81408] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81408) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:29:12.570730 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.auth.core [None req-0e3e42bb-1fb9-4401-a07d-038b7b8baf55 None None] MFA Rules not processed for user `0e2d979563d6421995451a0e86bf3022`. Rule list: `[]` (Enabled: `True`). {{(pid=81408) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 13 04:29:12.571939 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.fernet_utils [None req-0e3e42bb-1fb9-4401-a07d-038b7b8baf55 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81408) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:29:12.642321 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 1190/2413] 10.222.0.27 () {58 vars in 958 bytes} [Thu Oct 13 04:29:12 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 131 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 13 04:29:12.644861 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e1a3951-727d-48a6-a1bb-12f130114acc None None] REQUEST_METHOD: `POST` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:29:12.645145 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e1a3951-727d-48a6-a1bb-12f130114acc None None] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:29:12.645419 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e1a3951-727d-48a6-a1bb-12f130114acc None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:29:12.668373 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.auth.core [None req-2e1a3951-727d-48a6-a1bb-12f130114acc None None] MFA Rules not processed for user `0e2d979563d6421995451a0e86bf3022`. Rule list: `[]` (Enabled: `True`). {{(pid=81409) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 13 04:29:12.669705 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.fernet_utils [None req-2e1a3951-727d-48a6-a1bb-12f130114acc None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81409) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:29:12.677745 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 1224/2414] 10.222.0.27 () {58 vars in 958 bytes} [Thu Oct 13 04:29:12 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 34 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 13 04:29:12.682153 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0e3e42bb-1fb9-4401-a07d-038b7b8baf55 None None] Authenticating user token {{(pid=81408) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:29:12.692362 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0e3e42bb-1fb9-4401-a07d-038b7b8baf55 None None] Validating token access rules against request {{(pid=81408) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:29:12.697217 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7c368603-9527-40fc-8626-62aeb84b1887 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8ba823f313144da3b683bacef8fd62e0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81408) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:29:12.697823 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c368603-9527-40fc-8626-62aeb84b1887 admin admin] REQUEST_METHOD: `GET` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:29:12.698100 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c368603-9527-40fc-8626-62aeb84b1887 admin admin] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:29:12.698403 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c368603-9527-40fc-8626-62aeb84b1887 admin admin] PATH_INFO: `/v3/domains` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:29:12.699084 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7c368603-9527-40fc-8626-62aeb84b1887 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81408) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 13 04:29:12.699386 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7c368603-9527-40fc-8626-62aeb84b1887 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:29:12.700988 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7c368603-9527-40fc-8626-62aeb84b1887 admin admin] RBAC: Authorization granted {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:29:12.714657 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 1191/2415] 10.222.0.27 () {58 vars in 1153 bytes} [Thu Oct 13 04:29:12 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 13 04:29:12.718048 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2e1a3951-727d-48a6-a1bb-12f130114acc None None] Authenticating user token {{(pid=81409) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:29:12.727643 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2e1a3951-727d-48a6-a1bb-12f130114acc None None] Validating token access rules against request {{(pid=81409) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:29:12.732922 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e1ac7e2f-83a8-4c31-9a48-97dd8b36ed0f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8ba823f313144da3b683bacef8fd62e0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81409) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:29:12.733522 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1ac7e2f-83a8-4c31-9a48-97dd8b36ed0f admin admin] REQUEST_METHOD: `POST` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:29:12.733794 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1ac7e2f-83a8-4c31-9a48-97dd8b36ed0f admin admin] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:29:12.734060 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1ac7e2f-83a8-4c31-9a48-97dd8b36ed0f admin admin] PATH_INFO: `/v3/projects` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:29:12.734732 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e1ac7e2f-83a8-4c31-9a48-97dd8b36ed0f admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:29:12.736620 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e1ac7e2f-83a8-4c31-9a48-97dd8b36ed0f admin admin] RBAC: Authorization granted {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:29:12.767802 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 1225/2416] 10.222.0.27 () {60 vars in 1153 bytes} [Thu Oct 13 04:29:12 2022] POST /identity/v3/projects => generated 369 bytes in 48 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 13 04:29:12.769690 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7c368603-9527-40fc-8626-62aeb84b1887 admin admin] Authenticating user token {{(pid=81408) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:29:12.783841 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7c368603-9527-40fc-8626-62aeb84b1887 admin admin] Validating token access rules against request {{(pid=81408) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:29:12.841816 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ac236f1f-d378-4dad-9758-577776b8a51e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8ba823f313144da3b683bacef8fd62e0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81408) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:29:12.842456 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac236f1f-d378-4dad-9758-577776b8a51e admin admin] REQUEST_METHOD: `POST` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:29:12.842733 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac236f1f-d378-4dad-9758-577776b8a51e admin admin] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:29:12.843010 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac236f1f-d378-4dad-9758-577776b8a51e admin admin] PATH_INFO: `/v3/users` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:29:12.843637 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ac236f1f-d378-4dad-9758-577776b8a51e admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:29:12.845640 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ac236f1f-d378-4dad-9758-577776b8a51e admin admin] RBAC: Authorization granted {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:29:12.874646 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 1192/2417] 10.222.0.27 () {60 vars in 1144 bytes} [Thu Oct 13 04:29:12 2022] POST /identity/v3/users => generated 285 bytes in 103 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 13 04:29:12.876364 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e1ac7e2f-83a8-4c31-9a48-97dd8b36ed0f admin admin] Authenticating user token {{(pid=81409) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:29:12.876828 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ac236f1f-d378-4dad-9758-577776b8a51e admin admin] Authenticating user token {{(pid=81408) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:29:12.878478 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.fernet_utils [None req-ac236f1f-d378-4dad-9758-577776b8a51e admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81408) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:29:12.887057 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e1ac7e2f-83a8-4c31-9a48-97dd8b36ed0f admin admin] Validating token access rules against request {{(pid=81409) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:29:12.890172 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ac236f1f-d378-4dad-9758-577776b8a51e admin admin] Validating token access rules against request {{(pid=81408) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:29:12.891850 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f4b2a8e1-8e33-497d-a048-f78af34265ac admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8ba823f313144da3b683bacef8fd62e0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81409) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:29:12.892521 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f4b2a8e1-8e33-497d-a048-f78af34265ac admin admin] REQUEST_METHOD: `GET` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:29:12.892800 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f4b2a8e1-8e33-497d-a048-f78af34265ac admin admin] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:29:12.893069 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f4b2a8e1-8e33-497d-a048-f78af34265ac admin admin] PATH_INFO: `/v3/roles` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:29:12.893709 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f4b2a8e1-8e33-497d-a048-f78af34265ac admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:29:12.894880 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6f365cc4-352b-4ee2-8c82-d59d91da6e5e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8ba823f313144da3b683bacef8fd62e0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81408) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:29:12.895373 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f4b2a8e1-8e33-497d-a048-f78af34265ac admin admin] RBAC: Authorization granted {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:29:12.895503 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f365cc4-352b-4ee2-8c82-d59d91da6e5e admin admin] REQUEST_METHOD: `DELETE` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:29:12.895757 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f365cc4-352b-4ee2-8c82-d59d91da6e5e admin admin] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:29:12.896077 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f365cc4-352b-4ee2-8c82-d59d91da6e5e admin admin] PATH_INFO: `/v3/users/778eb55039f94988853ee1f24b64336c` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:29:12.897199 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6f365cc4-352b-4ee2-8c82-d59d91da6e5e admin admin] RBAC: Authorizing `identity:delete_user(user_id=778eb55039f94988853ee1f24b64336c)` {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:29:12.899220 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6f365cc4-352b-4ee2-8c82-d59d91da6e5e admin admin] RBAC: Authorization granted {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:29:12.905600 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 1226/2418] 10.222.0.27 () {58 vars in 1122 bytes} [Thu Oct 13 04:29:12 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 13 04:29:12.909454 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f4b2a8e1-8e33-497d-a048-f78af34265ac admin admin] Authenticating user token {{(pid=81409) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:29:12.920446 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f4b2a8e1-8e33-497d-a048-f78af34265ac admin admin] Validating token access rules against request {{(pid=81409) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:29:12.925065 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b4548d60-f2b4-454b-a090-598abb21f765 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8ba823f313144da3b683bacef8fd62e0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81409) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:29:12.925843 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4548d60-f2b4-454b-a090-598abb21f765 admin admin] REQUEST_METHOD: `GET` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:29:12.926106 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4548d60-f2b4-454b-a090-598abb21f765 admin admin] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:29:12.926442 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4548d60-f2b4-454b-a090-598abb21f765 admin admin] PATH_INFO: `/v3/roles` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:29:12.927234 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b4548d60-f2b4-454b-a090-598abb21f765 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:29:12.928935 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b4548d60-f2b4-454b-a090-598abb21f765 admin admin] RBAC: Authorization granted {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:29:12.940969 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 1227/2419] 10.222.0.27 () {58 vars in 1122 bytes} [Thu Oct 13 04:29:12 2022] GET /identity/v3/roles => generated 1420 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Oct 13 04:29:12.944594 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b4548d60-f2b4-454b-a090-598abb21f765 admin admin] Authenticating user token {{(pid=81409) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:29:12.948820 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.notifications [None req-6f365cc4-352b-4ee2-8c82-d59d91da6e5e admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '778eb55039f94988853ee1f24b64336c'} {{(pid=81408) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 13 04:29:12.954640 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b4548d60-f2b4-454b-a090-598abb21f765 admin admin] Validating token access rules against request {{(pid=81409) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:29:12.955738 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.notifications [None req-6f365cc4-352b-4ee2-8c82-d59d91da6e5e admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '778eb55039f94988853ee1f24b64336c'} {{(pid=81408) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 13 04:29:12.956404 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.notifications [None req-6f365cc4-352b-4ee2-8c82-d59d91da6e5e admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '778eb55039f94988853ee1f24b64336c'} {{(pid=81408) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 13 04:29:12.957007 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.notifications [None req-6f365cc4-352b-4ee2-8c82-d59d91da6e5e admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '778eb55039f94988853ee1f24b64336c'} {{(pid=81408) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 13 04:29:12.959245 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-66205c87-1780-403b-97a2-736e89ae860d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8ba823f313144da3b683bacef8fd62e0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81409) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:29:12.959904 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66205c87-1780-403b-97a2-736e89ae860d admin admin] REQUEST_METHOD: `PUT` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:29:12.960179 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66205c87-1780-403b-97a2-736e89ae860d admin admin] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:29:12.960454 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66205c87-1780-403b-97a2-736e89ae860d admin admin] PATH_INFO: `/v3/projects/8df7ef685ef049afa8de293738c98406/users/d8c58849da3a41d7b1c068b0bb9a512c/roles/cc56c4ae13a7424ea771c15da4b153c1` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:29:12.964923 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.notifications [None req-6f365cc4-352b-4ee2-8c82-d59d91da6e5e admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '778eb55039f94988853ee1f24b64336c'} {{(pid=81408) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 13 04:29:12.972730 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG dbcounter [-] [81408] Writing DB stats keystone:SELECT=87,keystone:INSERT=4,keystone:DELETE=9 {{(pid=81408) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:29:12.977504 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-66205c87-1780-403b-97a2-736e89ae860d admin admin] RBAC: Authorizing `identity:create_grant(project_id=8df7ef685ef049afa8de293738c98406, user_id=d8c58849da3a41d7b1c068b0bb9a512c, role_id=cc56c4ae13a7424ea771c15da4b153c1)` {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:29:12.981295 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-66205c87-1780-403b-97a2-736e89ae860d admin admin] RBAC: Authorization granted {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:29:12.987970 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 1193/2420] 10.222.0.27 () {58 vars in 1224 bytes} [Thu Oct 13 04:29:12 2022] DELETE /identity/v3/users/778eb55039f94988853ee1f24b64336c => generated 0 bytes in 111 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 13 04:29:12.992681 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4861dcc-1b25-4a65-8167-bb0b3fc0a7ac None None] REQUEST_METHOD: `POST` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:29:12.992961 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4861dcc-1b25-4a65-8167-bb0b3fc0a7ac None None] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:29:12.993228 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4861dcc-1b25-4a65-8167-bb0b3fc0a7ac None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:29:12.994970 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 1228/2421] 10.222.0.27 () {60 vars in 1483 bytes} [Thu Oct 13 04:29:12 2022] PUT /identity/v3/projects/8df7ef685ef049afa8de293738c98406/users/d8c58849da3a41d7b1c068b0bb9a512c/roles/cc56c4ae13a7424ea771c15da4b153c1 => generated 0 bytes in 51 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 13 04:29:13.005791 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4b3b7bc-9153-4cf4-a5ce-8752d6358484 None None] REQUEST_METHOD: `POST` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:29:13.006131 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4b3b7bc-9153-4cf4-a5ce-8752d6358484 None None] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:29:13.006463 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4b3b7bc-9153-4cf4-a5ce-8752d6358484 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:29:13.016001 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG dbcounter [-] [81409] Writing DB stats keystone:SELECT=90,keystone:DELETE=7,keystone:INSERT=3 {{(pid=81409) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:29:13.022501 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.auth.core [None req-d4861dcc-1b25-4a65-8167-bb0b3fc0a7ac None None] MFA Rules not processed for user `0e2d979563d6421995451a0e86bf3022`. Rule list: `[]` (Enabled: `True`). {{(pid=81408) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 13 04:29:13.023710 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.fernet_utils [None req-d4861dcc-1b25-4a65-8167-bb0b3fc0a7ac None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81408) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:29:13.044040 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.auth.core [None req-e4b3b7bc-9153-4cf4-a5ce-8752d6358484 None None] MFA Rules not processed for user `d8c58849da3a41d7b1c068b0bb9a512c`. Rule list: `[]` (Enabled: `True`). {{(pid=81409) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 13 04:29:13.045388 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.fernet_utils [None req-e4b3b7bc-9153-4cf4-a5ce-8752d6358484 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81409) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:29:13.090978 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 1194/2422] 10.222.0.27 () {58 vars in 958 bytes} [Thu Oct 13 04:29:12 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 99 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 13 04:29:13.098130 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d4861dcc-1b25-4a65-8167-bb0b3fc0a7ac None None] Authenticating user token {{(pid=81408) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:29:13.100326 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.fernet_utils [None req-d4861dcc-1b25-4a65-8167-bb0b3fc0a7ac None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81408) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:29:13.124719 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 1229/2423] 10.222.0.27 () {58 vars in 958 bytes} [Thu Oct 13 04:29:13 2022] POST /identity/v3/auth/tokens => generated 3246 bytes in 116 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 13 04:29:13.133422 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e4b3b7bc-9153-4cf4-a5ce-8752d6358484 None None] Authenticating user token {{(pid=81409) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:29:13.135668 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.fernet_utils [None req-e4b3b7bc-9153-4cf4-a5ce-8752d6358484 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81409) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:29:13.168834 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d4861dcc-1b25-4a65-8167-bb0b3fc0a7ac None None] Validating token access rules against request {{(pid=81408) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:29:13.172850 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f74fd0c2-53d7-461a-8d17-12ff625094fb service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd8812c3974f84871bb2d761cf0016257', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dcec0431fb6849499d6aec35a234daa6', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81408) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:29:13.173534 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f74fd0c2-53d7-461a-8d17-12ff625094fb service neutron] REQUEST_METHOD: `GET` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:29:13.173892 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f74fd0c2-53d7-461a-8d17-12ff625094fb service neutron] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:29:13.174242 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f74fd0c2-53d7-461a-8d17-12ff625094fb service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:29:13.191319 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f74fd0c2-53d7-461a-8d17-12ff625094fb service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:29:13.192766 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f74fd0c2-53d7-461a-8d17-12ff625094fb service neutron] RBAC: Authorization granted {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:29:13.203222 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e4b3b7bc-9153-4cf4-a5ce-8752d6358484 None None] Validating token access rules against request {{(pid=81409) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:29:13.203409 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 1195/2424] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Oct 13 04:29:13 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 106 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 13 04:29:13.207134 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-88afcd64-c826-4dc6-bf0b-19bcd972d3f7 service glance] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a6700f49b86543f6a5b92cad8faaba42', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dcec0431fb6849499d6aec35a234daa6', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81409) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:29:13.207734 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88afcd64-c826-4dc6-bf0b-19bcd972d3f7 service glance] REQUEST_METHOD: `GET` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:29:13.208005 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88afcd64-c826-4dc6-bf0b-19bcd972d3f7 service glance] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:29:13.208269 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88afcd64-c826-4dc6-bf0b-19bcd972d3f7 service glance] PATH_INFO: `/v3/auth/tokens` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:29:13.219562 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-88afcd64-c826-4dc6-bf0b-19bcd972d3f7 service glance] RBAC: Authorizing `identity:validate_token()` {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:29:13.221604 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-88afcd64-c826-4dc6-bf0b-19bcd972d3f7 service glance] RBAC: Authorization granted {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:29:13.233867 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 1230/2425] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Oct 13 04:29:13 2022] GET /identity/v3/auth/tokens => generated 3246 bytes in 102 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 13 04:29:13.550640 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f74fd0c2-53d7-461a-8d17-12ff625094fb service neutron] Authenticating user token {{(pid=81408) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:29:13.555518 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.fernet_utils [None req-f74fd0c2-53d7-461a-8d17-12ff625094fb service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81408) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:29:13.575031 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f74fd0c2-53d7-461a-8d17-12ff625094fb service neutron] Validating token access rules against request {{(pid=81408) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:29:13.584665 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b06db5a7-2a6c-4299-a240-199c43149ed7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8ba823f313144da3b683bacef8fd62e0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81408) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:29:13.585658 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b06db5a7-2a6c-4299-a240-199c43149ed7 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:29:13.585960 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b06db5a7-2a6c-4299-a240-199c43149ed7 admin admin] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:29:13.586327 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b06db5a7-2a6c-4299-a240-199c43149ed7 admin admin] PATH_INFO: `/v3/projects/728402e7d8a6419a8ebce7ccdbb965ef` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:29:13.587667 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b06db5a7-2a6c-4299-a240-199c43149ed7 admin admin] RBAC: Authorizing `identity:delete_project(project_id=728402e7d8a6419a8ebce7ccdbb965ef)` {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:29:13.591918 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b06db5a7-2a6c-4299-a240-199c43149ed7 admin admin] RBAC: Authorization granted {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:29:13.654988 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.notifications [None req-b06db5a7-2a6c-4299-a240-199c43149ed7 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '728402e7d8a6419a8ebce7ccdbb965ef', 'request_id': 'req-b06db5a7-2a6c-4299-a240-199c43149ed7'} {{(pid=81408) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 13 04:29:13.663582 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.notifications [None req-b06db5a7-2a6c-4299-a240-199c43149ed7 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '728402e7d8a6419a8ebce7ccdbb965ef', 'request_id': 'req-b06db5a7-2a6c-4299-a240-199c43149ed7'} {{(pid=81408) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 13 04:29:13.672326 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.notifications [None req-b06db5a7-2a6c-4299-a240-199c43149ed7 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '728402e7d8a6419a8ebce7ccdbb965ef', 'request_id': 'req-b06db5a7-2a6c-4299-a240-199c43149ed7'} {{(pid=81408) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 13 04:29:13.688503 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.notifications [None req-b06db5a7-2a6c-4299-a240-199c43149ed7 admin admin] The token cache is being invalidate because project 728402e7d8a6419a8ebce7ccdbb965ef was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81408) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 13 04:29:13.688977 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.notifications [None req-b06db5a7-2a6c-4299-a240-199c43149ed7 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81408) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 13 04:29:13.690008 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.notifications [None req-b06db5a7-2a6c-4299-a240-199c43149ed7 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81408) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 13 04:29:13.692220 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 1196/2426] 10.222.0.27 () {58 vars in 1233 bytes} [Thu Oct 13 04:29:13 2022] DELETE /identity/v3/projects/728402e7d8a6419a8ebce7ccdbb965ef => generated 0 bytes in 142 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 13 04:29:13.696655 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-88afcd64-c826-4dc6-bf0b-19bcd972d3f7 service glance] Authenticating user token {{(pid=81409) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:29:13.708929 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-88afcd64-c826-4dc6-bf0b-19bcd972d3f7 service glance] Validating token access rules against request {{(pid=81409) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:29:13.801733 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4284644c-ea71-4fcd-bc24-6374d625b53b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8ba823f313144da3b683bacef8fd62e0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81409) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:29:13.802452 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4284644c-ea71-4fcd-bc24-6374d625b53b admin admin] REQUEST_METHOD: `DELETE` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:29:13.802724 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4284644c-ea71-4fcd-bc24-6374d625b53b admin admin] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:29:13.802997 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4284644c-ea71-4fcd-bc24-6374d625b53b admin admin] PATH_INFO: `/v3/users/186fed5b8ca64f2cb53c2dde157dfae0` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:29:13.804140 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4284644c-ea71-4fcd-bc24-6374d625b53b admin admin] RBAC: Authorizing `identity:delete_user(user_id=186fed5b8ca64f2cb53c2dde157dfae0)` {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:29:13.806174 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4284644c-ea71-4fcd-bc24-6374d625b53b admin admin] RBAC: Authorization granted {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:29:13.861221 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG dbcounter [-] [81409] Writing DB stats keystone:SELECT=95,keystone:DELETE=5 {{(pid=81409) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:29:13.876268 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.notifications [None req-4284644c-ea71-4fcd-bc24-6374d625b53b admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '186fed5b8ca64f2cb53c2dde157dfae0'} {{(pid=81409) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 13 04:29:13.877365 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.notifications [None req-4284644c-ea71-4fcd-bc24-6374d625b53b admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '186fed5b8ca64f2cb53c2dde157dfae0'} {{(pid=81409) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 13 04:29:13.887497 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.notifications [None req-4284644c-ea71-4fcd-bc24-6374d625b53b admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '186fed5b8ca64f2cb53c2dde157dfae0'} {{(pid=81409) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 13 04:29:13.888379 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.notifications [None req-4284644c-ea71-4fcd-bc24-6374d625b53b admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '186fed5b8ca64f2cb53c2dde157dfae0'} {{(pid=81409) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 13 04:29:13.897411 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.notifications [None req-4284644c-ea71-4fcd-bc24-6374d625b53b admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '186fed5b8ca64f2cb53c2dde157dfae0'} {{(pid=81409) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 13 04:29:13.915526 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 1231/2427] 10.222.0.27 () {58 vars in 1224 bytes} [Thu Oct 13 04:29:13 2022] DELETE /identity/v3/users/186fed5b8ca64f2cb53c2dde157dfae0 => generated 0 bytes in 219 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 13 04:29:14.231164 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b06db5a7-2a6c-4299-a240-199c43149ed7 admin admin] Authenticating user token {{(pid=81408) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:29:14.232850 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.fernet_utils [None req-b06db5a7-2a6c-4299-a240-199c43149ed7 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81408) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:29:14.259471 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b06db5a7-2a6c-4299-a240-199c43149ed7 admin admin] Validating token access rules against request {{(pid=81408) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:29:14.279358 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG dbcounter [-] [81408] Writing DB stats keystone:SELECT=91,keystone:DELETE=8,keystone:INSERT=1 {{(pid=81408) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:29:14.336211 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e69f8390-11bb-41bd-a403-76b3954fbef1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8ba823f313144da3b683bacef8fd62e0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81408) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:29:14.336933 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e69f8390-11bb-41bd-a403-76b3954fbef1 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:29:14.337311 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e69f8390-11bb-41bd-a403-76b3954fbef1 admin admin] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:29:14.337807 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e69f8390-11bb-41bd-a403-76b3954fbef1 admin admin] PATH_INFO: `/v3/projects/2b57f5dd803e436fbd1e4c597c0a5605` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:29:14.339672 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e69f8390-11bb-41bd-a403-76b3954fbef1 admin admin] RBAC: Authorizing `identity:delete_project(project_id=2b57f5dd803e436fbd1e4c597c0a5605)` {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:29:14.342039 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e69f8390-11bb-41bd-a403-76b3954fbef1 admin admin] RBAC: Authorization granted {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:29:14.405139 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.notifications [None req-e69f8390-11bb-41bd-a403-76b3954fbef1 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '2b57f5dd803e436fbd1e4c597c0a5605', 'request_id': 'req-e69f8390-11bb-41bd-a403-76b3954fbef1'} {{(pid=81408) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 13 04:29:14.424049 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.notifications [None req-e69f8390-11bb-41bd-a403-76b3954fbef1 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '2b57f5dd803e436fbd1e4c597c0a5605', 'request_id': 'req-e69f8390-11bb-41bd-a403-76b3954fbef1'} {{(pid=81408) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 13 04:29:14.433669 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.notifications [None req-e69f8390-11bb-41bd-a403-76b3954fbef1 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '2b57f5dd803e436fbd1e4c597c0a5605', 'request_id': 'req-e69f8390-11bb-41bd-a403-76b3954fbef1'} {{(pid=81408) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 13 04:29:14.446752 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.notifications [None req-e69f8390-11bb-41bd-a403-76b3954fbef1 admin admin] The token cache is being invalidate because project 2b57f5dd803e436fbd1e4c597c0a5605 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81408) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 13 04:29:14.447043 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.notifications [None req-e69f8390-11bb-41bd-a403-76b3954fbef1 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81408) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 13 04:29:14.447702 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.notifications [None req-e69f8390-11bb-41bd-a403-76b3954fbef1 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81408) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 13 04:29:14.449959 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 1197/2428] 10.222.0.27 () {58 vars in 1233 bytes} [Thu Oct 13 04:29:14 2022] DELETE /identity/v3/projects/2b57f5dd803e436fbd1e4c597c0a5605 => generated 0 bytes in 222 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 13 04:29:14.455014 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2e132a1-d821-4908-be4d-33355fecd30e None None] REQUEST_METHOD: `POST` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:29:14.455302 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2e132a1-d821-4908-be4d-33355fecd30e None None] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:29:14.455575 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2e132a1-d821-4908-be4d-33355fecd30e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:29:14.477971 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.auth.core [None req-f2e132a1-d821-4908-be4d-33355fecd30e None None] MFA Rules not processed for user `0e2d979563d6421995451a0e86bf3022`. Rule list: `[]` (Enabled: `True`). {{(pid=81409) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 13 04:29:14.479233 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.fernet_utils [None req-f2e132a1-d821-4908-be4d-33355fecd30e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81409) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:29:14.545179 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 1232/2429] 10.222.0.27 () {58 vars in 958 bytes} [Thu Oct 13 04:29:14 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 91 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 13 04:29:14.553455 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1cca48c6-78d6-48ae-ae1f-700c2142f5ca None None] REQUEST_METHOD: `POST` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:29:14.553978 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1cca48c6-78d6-48ae-ae1f-700c2142f5ca None None] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:29:14.554559 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1cca48c6-78d6-48ae-ae1f-700c2142f5ca None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:29:14.576808 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.auth.core [None req-1cca48c6-78d6-48ae-ae1f-700c2142f5ca None None] MFA Rules not processed for user `0e2d979563d6421995451a0e86bf3022`. Rule list: `[]` (Enabled: `True`). {{(pid=81408) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 13 04:29:14.578052 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.fernet_utils [None req-1cca48c6-78d6-48ae-ae1f-700c2142f5ca None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81408) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:29:14.587382 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 1198/2430] 10.222.0.27 () {58 vars in 958 bytes} [Thu Oct 13 04:29:14 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 35 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 13 04:29:14.592276 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f2e132a1-d821-4908-be4d-33355fecd30e None None] Authenticating user token {{(pid=81409) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:29:14.602398 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f2e132a1-d821-4908-be4d-33355fecd30e None None] Validating token access rules against request {{(pid=81409) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:29:14.607355 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-af8666dc-05ae-43d1-b9d5-fb68caa7e5b3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8ba823f313144da3b683bacef8fd62e0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81409) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:29:14.607968 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af8666dc-05ae-43d1-b9d5-fb68caa7e5b3 admin admin] REQUEST_METHOD: `GET` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:29:14.608243 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af8666dc-05ae-43d1-b9d5-fb68caa7e5b3 admin admin] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:29:14.608512 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af8666dc-05ae-43d1-b9d5-fb68caa7e5b3 admin admin] PATH_INFO: `/v3/domains` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:29:14.609190 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-af8666dc-05ae-43d1-b9d5-fb68caa7e5b3 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81409) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 13 04:29:14.609489 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-af8666dc-05ae-43d1-b9d5-fb68caa7e5b3 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:29:14.611120 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-af8666dc-05ae-43d1-b9d5-fb68caa7e5b3 admin admin] RBAC: Authorization granted {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:29:14.624075 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 1233/2431] 10.222.0.27 () {58 vars in 1153 bytes} [Thu Oct 13 04:29:14 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 13 04:29:14.628760 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1cca48c6-78d6-48ae-ae1f-700c2142f5ca None None] Authenticating user token {{(pid=81408) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:29:14.640328 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1cca48c6-78d6-48ae-ae1f-700c2142f5ca None None] Validating token access rules against request {{(pid=81408) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:29:14.645050 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f168f0b-6781-493a-a6e9-b163eb9d5448 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8ba823f313144da3b683bacef8fd62e0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81408) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:29:14.645649 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f168f0b-6781-493a-a6e9-b163eb9d5448 admin admin] REQUEST_METHOD: `POST` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:29:14.645923 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f168f0b-6781-493a-a6e9-b163eb9d5448 admin admin] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:29:14.646192 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f168f0b-6781-493a-a6e9-b163eb9d5448 admin admin] PATH_INFO: `/v3/projects` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:29:14.646857 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5f168f0b-6781-493a-a6e9-b163eb9d5448 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:29:14.649348 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5f168f0b-6781-493a-a6e9-b163eb9d5448 admin admin] RBAC: Authorization granted {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:29:14.679368 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 1199/2432] 10.222.0.27 () {60 vars in 1153 bytes} [Thu Oct 13 04:29:14 2022] POST /identity/v3/projects => generated 369 bytes in 51 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 13 04:29:14.684697 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-af8666dc-05ae-43d1-b9d5-fb68caa7e5b3 admin admin] Authenticating user token {{(pid=81409) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:29:14.695900 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-af8666dc-05ae-43d1-b9d5-fb68caa7e5b3 admin admin] Validating token access rules against request {{(pid=81409) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:29:14.758362 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0f6fb6fb-65b4-4983-85d9-926a3f574a45 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8ba823f313144da3b683bacef8fd62e0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81409) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:29:14.758971 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f6fb6fb-65b4-4983-85d9-926a3f574a45 admin admin] REQUEST_METHOD: `POST` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:29:14.759272 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f6fb6fb-65b4-4983-85d9-926a3f574a45 admin admin] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:29:14.759539 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f6fb6fb-65b4-4983-85d9-926a3f574a45 admin admin] PATH_INFO: `/v3/users` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:29:14.760178 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0f6fb6fb-65b4-4983-85d9-926a3f574a45 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:29:14.762253 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0f6fb6fb-65b4-4983-85d9-926a3f574a45 admin admin] RBAC: Authorization granted {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:29:14.785724 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 1234/2433] 10.222.0.27 () {60 vars in 1144 bytes} [Thu Oct 13 04:29:14 2022] POST /identity/v3/users => generated 285 bytes in 102 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 13 04:29:14.790112 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f168f0b-6781-493a-a6e9-b163eb9d5448 admin admin] Authenticating user token {{(pid=81408) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:29:14.801549 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f168f0b-6781-493a-a6e9-b163eb9d5448 admin admin] Validating token access rules against request {{(pid=81408) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:29:14.806488 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1a4a1b14-aabd-4c02-8d45-b9cce2e95819 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8ba823f313144da3b683bacef8fd62e0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81408) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:29:14.807094 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a4a1b14-aabd-4c02-8d45-b9cce2e95819 admin admin] REQUEST_METHOD: `GET` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:29:14.807389 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a4a1b14-aabd-4c02-8d45-b9cce2e95819 admin admin] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:29:14.807682 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a4a1b14-aabd-4c02-8d45-b9cce2e95819 admin admin] PATH_INFO: `/v3/roles` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:29:14.808598 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1a4a1b14-aabd-4c02-8d45-b9cce2e95819 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:29:14.810309 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1a4a1b14-aabd-4c02-8d45-b9cce2e95819 admin admin] RBAC: Authorization granted {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:29:14.820996 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 1200/2434] 10.222.0.27 () {58 vars in 1122 bytes} [Thu Oct 13 04:29:14 2022] GET /identity/v3/roles => generated 1420 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 13 04:29:14.825358 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0f6fb6fb-65b4-4983-85d9-926a3f574a45 admin admin] Authenticating user token {{(pid=81409) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:29:14.836094 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0f6fb6fb-65b4-4983-85d9-926a3f574a45 admin admin] Validating token access rules against request {{(pid=81409) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:29:14.843005 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a108cbe3-8ece-4212-ab42-d4b34f3002ed admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8ba823f313144da3b683bacef8fd62e0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81409) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:29:14.843621 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a108cbe3-8ece-4212-ab42-d4b34f3002ed admin admin] REQUEST_METHOD: `GET` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:29:14.843895 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a108cbe3-8ece-4212-ab42-d4b34f3002ed admin admin] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:29:14.844162 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a108cbe3-8ece-4212-ab42-d4b34f3002ed admin admin] PATH_INFO: `/v3/roles` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:29:14.844907 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a108cbe3-8ece-4212-ab42-d4b34f3002ed admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:29:14.846704 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a108cbe3-8ece-4212-ab42-d4b34f3002ed admin admin] RBAC: Authorization granted {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:29:14.864531 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1a4a1b14-aabd-4c02-8d45-b9cce2e95819 admin admin] Authenticating user token {{(pid=81408) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:29:14.865564 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 1235/2435] 10.222.0.27 () {58 vars in 1122 bytes} [Thu Oct 13 04:29:14 2022] GET /identity/v3/roles => generated 1420 bytes in 40 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 13 04:29:14.876824 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1a4a1b14-aabd-4c02-8d45-b9cce2e95819 admin admin] Validating token access rules against request {{(pid=81408) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:29:14.891861 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b55c225b-22e1-492d-9cd5-d1574ca5a25e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8ba823f313144da3b683bacef8fd62e0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81408) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:29:14.893605 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b55c225b-22e1-492d-9cd5-d1574ca5a25e admin admin] REQUEST_METHOD: `PUT` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:29:14.894153 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b55c225b-22e1-492d-9cd5-d1574ca5a25e admin admin] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:29:14.894452 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b55c225b-22e1-492d-9cd5-d1574ca5a25e admin admin] PATH_INFO: `/v3/projects/f8d6f417874147a7ba641ddc193eb891/users/c355f70f51e64a18ae5f9a1e47a9e65c/roles/cc56c4ae13a7424ea771c15da4b153c1` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:29:14.912037 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b55c225b-22e1-492d-9cd5-d1574ca5a25e admin admin] RBAC: Authorizing `identity:create_grant(project_id=f8d6f417874147a7ba641ddc193eb891, user_id=c355f70f51e64a18ae5f9a1e47a9e65c, role_id=cc56c4ae13a7424ea771c15da4b153c1)` {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:29:14.914836 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b55c225b-22e1-492d-9cd5-d1574ca5a25e admin admin] RBAC: Authorization granted {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:29:14.928967 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 1201/2436] 10.222.0.27 () {60 vars in 1483 bytes} [Thu Oct 13 04:29:14 2022] PUT /identity/v3/projects/f8d6f417874147a7ba641ddc193eb891/users/c355f70f51e64a18ae5f9a1e47a9e65c/roles/cc56c4ae13a7424ea771c15da4b153c1 => generated 0 bytes in 65 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 13 04:29:14.940162 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82b95894-a768-4326-80d2-d3c980e2fee4 None None] REQUEST_METHOD: `POST` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:29:14.940668 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82b95894-a768-4326-80d2-d3c980e2fee4 None None] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:29:14.940769 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82b95894-a768-4326-80d2-d3c980e2fee4 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:29:14.971083 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.auth.core [None req-82b95894-a768-4326-80d2-d3c980e2fee4 None None] MFA Rules not processed for user `c355f70f51e64a18ae5f9a1e47a9e65c`. Rule list: `[]` (Enabled: `True`). {{(pid=81409) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 13 04:29:14.972592 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.fernet_utils [None req-82b95894-a768-4326-80d2-d3c980e2fee4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81409) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:29:14.982439 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG dbcounter [-] [81409] Writing DB stats keystone:SELECT=91,keystone:DELETE=5,keystone:INSERT=4 {{(pid=81409) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:29:15.043352 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 1236/2437] 10.222.0.27 () {58 vars in 958 bytes} [Thu Oct 13 04:29:14 2022] POST /identity/v3/auth/tokens => generated 3246 bytes in 104 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 13 04:29:15.047943 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b55c225b-22e1-492d-9cd5-d1574ca5a25e admin admin] Authenticating user token {{(pid=81408) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:29:15.094481 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG dbcounter [-] [81408] Writing DB stats keystone:SELECT=90,keystone:DELETE=7,keystone:INSERT=3 {{(pid=81408) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:29:15.114541 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b55c225b-22e1-492d-9cd5-d1574ca5a25e admin admin] Validating token access rules against request {{(pid=81408) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:29:15.119205 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-89f92d08-bd0e-47f4-afe3-7c0096845652 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8ba823f313144da3b683bacef8fd62e0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81408) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:29:15.119807 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89f92d08-bd0e-47f4-afe3-7c0096845652 admin admin] REQUEST_METHOD: `POST` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:29:15.120082 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89f92d08-bd0e-47f4-afe3-7c0096845652 admin admin] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:29:15.120352 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89f92d08-bd0e-47f4-afe3-7c0096845652 admin admin] PATH_INFO: `/v3/projects` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:29:15.121136 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-89f92d08-bd0e-47f4-afe3-7c0096845652 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:29:15.123578 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-89f92d08-bd0e-47f4-afe3-7c0096845652 admin admin] RBAC: Authorization granted {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:29:15.152573 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 1202/2438] 10.222.0.27 () {60 vars in 1153 bytes} [Thu Oct 13 04:29:15 2022] POST /identity/v3/projects => generated 369 bytes in 105 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 13 04:29:15.156805 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-82b95894-a768-4326-80d2-d3c980e2fee4 None None] Authenticating user token {{(pid=81409) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:29:15.169095 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-82b95894-a768-4326-80d2-d3c980e2fee4 None None] Validating token access rules against request {{(pid=81409) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:29:15.228480 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c46887ef-ef5a-45c8-8980-d1eaa1b157f2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8ba823f313144da3b683bacef8fd62e0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81409) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:29:15.229088 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c46887ef-ef5a-45c8-8980-d1eaa1b157f2 admin admin] REQUEST_METHOD: `POST` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:29:15.229360 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c46887ef-ef5a-45c8-8980-d1eaa1b157f2 admin admin] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:29:15.229627 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c46887ef-ef5a-45c8-8980-d1eaa1b157f2 admin admin] PATH_INFO: `/v3/users` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:29:15.230723 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c46887ef-ef5a-45c8-8980-d1eaa1b157f2 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:29:15.232896 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c46887ef-ef5a-45c8-8980-d1eaa1b157f2 admin admin] RBAC: Authorization granted {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:29:15.254770 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 1237/2439] 10.222.0.27 () {60 vars in 1144 bytes} [Thu Oct 13 04:29:15 2022] POST /identity/v3/users => generated 291 bytes in 98 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 13 04:29:15.258964 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-89f92d08-bd0e-47f4-afe3-7c0096845652 admin admin] Authenticating user token {{(pid=81408) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:29:15.270844 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-89f92d08-bd0e-47f4-afe3-7c0096845652 admin admin] Validating token access rules against request {{(pid=81408) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:29:15.279756 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-16532b5f-3fc9-4ba3-aba2-60d48dceacb7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8ba823f313144da3b683bacef8fd62e0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81408) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:29:15.280380 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16532b5f-3fc9-4ba3-aba2-60d48dceacb7 admin admin] REQUEST_METHOD: `GET` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:29:15.280656 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16532b5f-3fc9-4ba3-aba2-60d48dceacb7 admin admin] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:29:15.280949 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16532b5f-3fc9-4ba3-aba2-60d48dceacb7 admin admin] PATH_INFO: `/v3/roles` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:29:15.281631 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-16532b5f-3fc9-4ba3-aba2-60d48dceacb7 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:29:15.283271 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-16532b5f-3fc9-4ba3-aba2-60d48dceacb7 admin admin] RBAC: Authorization granted {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:29:15.296610 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 1203/2440] 10.222.0.27 () {58 vars in 1122 bytes} [Thu Oct 13 04:29:15 2022] GET /identity/v3/roles => generated 1420 bytes in 38 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 13 04:29:15.301010 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c46887ef-ef5a-45c8-8980-d1eaa1b157f2 admin admin] Authenticating user token {{(pid=81409) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:29:15.312013 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c46887ef-ef5a-45c8-8980-d1eaa1b157f2 admin admin] Validating token access rules against request {{(pid=81409) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:29:15.316797 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-80b23805-8e76-40d3-bcd2-4c5f183f8846 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8ba823f313144da3b683bacef8fd62e0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81409) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:29:15.317633 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80b23805-8e76-40d3-bcd2-4c5f183f8846 admin admin] REQUEST_METHOD: `PUT` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:29:15.317909 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80b23805-8e76-40d3-bcd2-4c5f183f8846 admin admin] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:29:15.318177 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80b23805-8e76-40d3-bcd2-4c5f183f8846 admin admin] PATH_INFO: `/v3/projects/9ff74d24f31643ce9f136ee4b099ead6/users/a25ba64e178740c5a9f10a791553f1dc/roles/ef55fbd94e1040be83695754075a7ed5` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:29:15.335284 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-80b23805-8e76-40d3-bcd2-4c5f183f8846 admin admin] RBAC: Authorizing `identity:create_grant(project_id=9ff74d24f31643ce9f136ee4b099ead6, user_id=a25ba64e178740c5a9f10a791553f1dc, role_id=ef55fbd94e1040be83695754075a7ed5)` {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:29:15.337931 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-80b23805-8e76-40d3-bcd2-4c5f183f8846 admin admin] RBAC: Authorization granted {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:29:15.353105 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 1238/2441] 10.222.0.27 () {60 vars in 1483 bytes} [Thu Oct 13 04:29:15 2022] PUT /identity/v3/projects/9ff74d24f31643ce9f136ee4b099ead6/users/a25ba64e178740c5a9f10a791553f1dc/roles/ef55fbd94e1040be83695754075a7ed5 => generated 0 bytes in 53 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 13 04:29:15.358829 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-16532b5f-3fc9-4ba3-aba2-60d48dceacb7 admin admin] Authenticating user token {{(pid=81408) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:29:15.372479 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-16532b5f-3fc9-4ba3-aba2-60d48dceacb7 admin admin] Validating token access rules against request {{(pid=81408) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:29:15.452357 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-840647ad-2b5a-4e1f-820c-8dca7a6e2411 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8ba823f313144da3b683bacef8fd62e0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81408) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:29:15.453057 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-840647ad-2b5a-4e1f-820c-8dca7a6e2411 admin admin] REQUEST_METHOD: `GET` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:29:15.453332 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-840647ad-2b5a-4e1f-820c-8dca7a6e2411 admin admin] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:29:15.453597 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-840647ad-2b5a-4e1f-820c-8dca7a6e2411 admin admin] PATH_INFO: `/v3/roles` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:29:15.454363 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-840647ad-2b5a-4e1f-820c-8dca7a6e2411 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:29:15.456160 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-840647ad-2b5a-4e1f-820c-8dca7a6e2411 admin admin] RBAC: Authorization granted {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:29:15.474260 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 1204/2442] 10.222.0.27 () {58 vars in 1122 bytes} [Thu Oct 13 04:29:15 2022] GET /identity/v3/roles => generated 1420 bytes in 117 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 13 04:29:15.477539 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-80b23805-8e76-40d3-bcd2-4c5f183f8846 admin admin] Authenticating user token {{(pid=81409) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:29:15.490404 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-80b23805-8e76-40d3-bcd2-4c5f183f8846 admin admin] Validating token access rules against request {{(pid=81409) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:29:15.497200 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8b892b35-7655-4b38-a588-87f97a8278cc admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8ba823f313144da3b683bacef8fd62e0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81409) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:29:15.497871 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b892b35-7655-4b38-a588-87f97a8278cc admin admin] REQUEST_METHOD: `PUT` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:29:15.498140 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b892b35-7655-4b38-a588-87f97a8278cc admin admin] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:29:15.498450 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b892b35-7655-4b38-a588-87f97a8278cc admin admin] PATH_INFO: `/v3/projects/9ff74d24f31643ce9f136ee4b099ead6/users/a25ba64e178740c5a9f10a791553f1dc/roles/ef55fbd94e1040be83695754075a7ed5` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:29:15.500504 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8b892b35-7655-4b38-a588-87f97a8278cc admin admin] RBAC: Authorizing `identity:create_grant(project_id=9ff74d24f31643ce9f136ee4b099ead6, user_id=a25ba64e178740c5a9f10a791553f1dc, role_id=ef55fbd94e1040be83695754075a7ed5)` {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:29:15.503091 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8b892b35-7655-4b38-a588-87f97a8278cc admin admin] RBAC: Authorization granted {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:29:15.515964 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 1239/2443] 10.222.0.27 () {60 vars in 1483 bytes} [Thu Oct 13 04:29:15 2022] PUT /identity/v3/projects/9ff74d24f31643ce9f136ee4b099ead6/users/a25ba64e178740c5a9f10a791553f1dc/roles/ef55fbd94e1040be83695754075a7ed5 => generated 0 bytes in 39 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 13 04:29:15.528526 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96273785-ae9b-4a67-9543-4122a845d922 None None] REQUEST_METHOD: `POST` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:29:15.528806 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96273785-ae9b-4a67-9543-4122a845d922 None None] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:29:15.529076 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96273785-ae9b-4a67-9543-4122a845d922 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:29:15.560576 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.auth.core [None req-96273785-ae9b-4a67-9543-4122a845d922 None None] MFA Rules not processed for user `a25ba64e178740c5a9f10a791553f1dc`. Rule list: `[]` (Enabled: `True`). {{(pid=81408) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 13 04:29:15.561801 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.fernet_utils [None req-96273785-ae9b-4a67-9543-4122a845d922 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81408) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:29:15.636384 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 1205/2444] 10.222.0.27 () {58 vars in 958 bytes} [Thu Oct 13 04:29:15 2022] POST /identity/v3/auth/tokens => generated 3313 bytes in 110 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 13 04:29:15.648643 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8b892b35-7655-4b38-a588-87f97a8278cc admin admin] Authenticating user token {{(pid=81409) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:29:15.650403 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.fernet_utils [None req-8b892b35-7655-4b38-a588-87f97a8278cc admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81409) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:29:15.719120 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8b892b35-7655-4b38-a588-87f97a8278cc admin admin] Validating token access rules against request {{(pid=81409) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:29:15.723063 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-19d3ae4d-d536-4c9e-81c9-b4c57adaccd8 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6141b8a6c3d54e179d6879bb019f4d12', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dcec0431fb6849499d6aec35a234daa6', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81409) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:29:15.723673 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19d3ae4d-d536-4c9e-81c9-b4c57adaccd8 service cinder] REQUEST_METHOD: `GET` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:29:15.723947 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19d3ae4d-d536-4c9e-81c9-b4c57adaccd8 service cinder] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:29:15.724217 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19d3ae4d-d536-4c9e-81c9-b4c57adaccd8 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:29:15.735401 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-19d3ae4d-d536-4c9e-81c9-b4c57adaccd8 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:29:15.736829 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-19d3ae4d-d536-4c9e-81c9-b4c57adaccd8 service cinder] RBAC: Authorization granted {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:29:15.747082 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 1240/2445] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Oct 13 04:29:15 2022] GET /identity/v3/auth/tokens => generated 3313 bytes in 99 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 13 04:29:15.796077 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-96273785-ae9b-4a67-9543-4122a845d922 None None] Authenticating user token {{(pid=81408) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:29:15.806029 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-96273785-ae9b-4a67-9543-4122a845d922 None None] Validating token access rules against request {{(pid=81408) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:29:15.809945 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bc7a0919-93ce-4812-bd8f-391a1315b7d6 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6141b8a6c3d54e179d6879bb019f4d12', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dcec0431fb6849499d6aec35a234daa6', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81408) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:29:15.810574 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc7a0919-93ce-4812-bd8f-391a1315b7d6 service cinder] REQUEST_METHOD: `GET` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:29:15.810846 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc7a0919-93ce-4812-bd8f-391a1315b7d6 service cinder] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:29:15.811114 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc7a0919-93ce-4812-bd8f-391a1315b7d6 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:29:15.846577 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG dbcounter [-] [81408] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81408) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:29:15.878752 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bc7a0919-93ce-4812-bd8f-391a1315b7d6 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:29:15.880260 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bc7a0919-93ce-4812-bd8f-391a1315b7d6 service cinder] RBAC: Authorization granted {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:29:15.890836 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 1206/2446] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Oct 13 04:29:15 2022] GET /identity/v3/auth/tokens => generated 3246 bytes in 95 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 13 04:29:19.327048 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-19d3ae4d-d536-4c9e-81c9-b4c57adaccd8 service cinder] Authenticating user token {{(pid=81409) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:29:19.332743 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG dbcounter [-] [81409] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=81409) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:29:19.394897 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-19d3ae4d-d536-4c9e-81c9-b4c57adaccd8 service cinder] Validating token access rules against request {{(pid=81409) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:29:19.399934 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-26aff5dd-1560-4818-b2e2-7bb7441a1a24 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8ba823f313144da3b683bacef8fd62e0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81409) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:29:19.400557 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26aff5dd-1560-4818-b2e2-7bb7441a1a24 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:29:19.400831 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26aff5dd-1560-4818-b2e2-7bb7441a1a24 admin admin] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:29:19.401103 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26aff5dd-1560-4818-b2e2-7bb7441a1a24 admin admin] PATH_INFO: `/v3/users/c355f70f51e64a18ae5f9a1e47a9e65c` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:29:19.402764 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-26aff5dd-1560-4818-b2e2-7bb7441a1a24 admin admin] RBAC: Authorizing `identity:delete_user(user_id=c355f70f51e64a18ae5f9a1e47a9e65c)` {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:29:19.404668 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-26aff5dd-1560-4818-b2e2-7bb7441a1a24 admin admin] RBAC: Authorization granted {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:29:19.455204 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.notifications [None req-26aff5dd-1560-4818-b2e2-7bb7441a1a24 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'c355f70f51e64a18ae5f9a1e47a9e65c'} {{(pid=81409) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 13 04:29:19.455846 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.notifications [None req-26aff5dd-1560-4818-b2e2-7bb7441a1a24 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'c355f70f51e64a18ae5f9a1e47a9e65c'} {{(pid=81409) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 13 04:29:19.463670 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.notifications [None req-26aff5dd-1560-4818-b2e2-7bb7441a1a24 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'c355f70f51e64a18ae5f9a1e47a9e65c'} {{(pid=81409) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 13 04:29:19.465142 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.notifications [None req-26aff5dd-1560-4818-b2e2-7bb7441a1a24 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'c355f70f51e64a18ae5f9a1e47a9e65c'} {{(pid=81409) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 13 04:29:19.472216 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.notifications [None req-26aff5dd-1560-4818-b2e2-7bb7441a1a24 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'c355f70f51e64a18ae5f9a1e47a9e65c'} {{(pid=81409) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 13 04:29:19.491561 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 1241/2447] 10.222.0.27 () {58 vars in 1224 bytes} [Thu Oct 13 04:29:19 2022] DELETE /identity/v3/users/c355f70f51e64a18ae5f9a1e47a9e65c => generated 0 bytes in 165 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 13 04:29:19.496277 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a73ba944-8724-48ea-8a56-4dd869246938 None None] REQUEST_METHOD: `POST` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:29:19.496817 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a73ba944-8724-48ea-8a56-4dd869246938 None None] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:29:19.496817 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a73ba944-8724-48ea-8a56-4dd869246938 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:29:19.519334 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.auth.core [None req-a73ba944-8724-48ea-8a56-4dd869246938 None None] MFA Rules not processed for user `0e2d979563d6421995451a0e86bf3022`. Rule list: `[]` (Enabled: `True`). {{(pid=81408) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 13 04:29:19.520453 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.fernet_utils [None req-a73ba944-8724-48ea-8a56-4dd869246938 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81408) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:29:19.581701 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 1207/2448] 10.222.0.27 () {58 vars in 958 bytes} [Thu Oct 13 04:29:19 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 86 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 13 04:29:19.592004 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-26aff5dd-1560-4818-b2e2-7bb7441a1a24 admin admin] Authenticating user token {{(pid=81409) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:29:19.593398 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.fernet_utils [None req-26aff5dd-1560-4818-b2e2-7bb7441a1a24 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81409) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:29:19.677930 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-26aff5dd-1560-4818-b2e2-7bb7441a1a24 admin admin] Validating token access rules against request {{(pid=81409) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:29:19.681586 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-16331727-fd77-4649-a5bc-27780042c95a service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd8812c3974f84871bb2d761cf0016257', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dcec0431fb6849499d6aec35a234daa6', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81409) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:29:19.682185 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16331727-fd77-4649-a5bc-27780042c95a service neutron] REQUEST_METHOD: `GET` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:29:19.682494 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16331727-fd77-4649-a5bc-27780042c95a service neutron] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:29:19.682764 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16331727-fd77-4649-a5bc-27780042c95a service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:29:19.692024 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-16331727-fd77-4649-a5bc-27780042c95a service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:29:19.693490 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-16331727-fd77-4649-a5bc-27780042c95a service neutron] RBAC: Authorization granted {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:29:19.704179 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 1242/2449] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Oct 13 04:29:19 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 113 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 13 04:29:19.982465 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a73ba944-8724-48ea-8a56-4dd869246938 None None] Authenticating user token {{(pid=81408) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:29:19.983908 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.fernet_utils [None req-a73ba944-8724-48ea-8a56-4dd869246938 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81408) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:29:19.997184 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a73ba944-8724-48ea-8a56-4dd869246938 None None] Validating token access rules against request {{(pid=81408) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:29:20.001908 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9f1cb6d4-516b-40f3-a7ce-dddc8013bcc9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8ba823f313144da3b683bacef8fd62e0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81408) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:29:20.002580 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f1cb6d4-516b-40f3-a7ce-dddc8013bcc9 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:29:20.002852 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f1cb6d4-516b-40f3-a7ce-dddc8013bcc9 admin admin] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:29:20.003121 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f1cb6d4-516b-40f3-a7ce-dddc8013bcc9 admin admin] PATH_INFO: `/v3/projects/f8d6f417874147a7ba641ddc193eb891` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:29:20.004334 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9f1cb6d4-516b-40f3-a7ce-dddc8013bcc9 admin admin] RBAC: Authorizing `identity:delete_project(project_id=f8d6f417874147a7ba641ddc193eb891)` {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:29:20.006613 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9f1cb6d4-516b-40f3-a7ce-dddc8013bcc9 admin admin] RBAC: Authorization granted {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:29:20.055704 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.notifications [None req-9f1cb6d4-516b-40f3-a7ce-dddc8013bcc9 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'f8d6f417874147a7ba641ddc193eb891', 'request_id': 'req-9f1cb6d4-516b-40f3-a7ce-dddc8013bcc9'} {{(pid=81408) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 13 04:29:20.066037 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.notifications [None req-9f1cb6d4-516b-40f3-a7ce-dddc8013bcc9 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'f8d6f417874147a7ba641ddc193eb891', 'request_id': 'req-9f1cb6d4-516b-40f3-a7ce-dddc8013bcc9'} {{(pid=81408) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 13 04:29:20.075092 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.notifications [None req-9f1cb6d4-516b-40f3-a7ce-dddc8013bcc9 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'f8d6f417874147a7ba641ddc193eb891', 'request_id': 'req-9f1cb6d4-516b-40f3-a7ce-dddc8013bcc9'} {{(pid=81408) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 13 04:29:20.086102 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.notifications [None req-9f1cb6d4-516b-40f3-a7ce-dddc8013bcc9 admin admin] The token cache is being invalidate because project f8d6f417874147a7ba641ddc193eb891 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81408) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 13 04:29:20.086418 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.notifications [None req-9f1cb6d4-516b-40f3-a7ce-dddc8013bcc9 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81408) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 13 04:29:20.087067 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.notifications [None req-9f1cb6d4-516b-40f3-a7ce-dddc8013bcc9 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81408) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 13 04:29:20.088553 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 1208/2450] 10.222.0.27 () {58 vars in 1233 bytes} [Thu Oct 13 04:29:19 2022] DELETE /identity/v3/projects/f8d6f417874147a7ba641ddc193eb891 => generated 0 bytes in 107 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 13 04:29:20.093104 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-16331727-fd77-4649-a5bc-27780042c95a service neutron] Authenticating user token {{(pid=81409) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:29:20.103112 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-16331727-fd77-4649-a5bc-27780042c95a service neutron] Validating token access rules against request {{(pid=81409) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:29:20.107806 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG dbcounter [-] [81409] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81409) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:29:20.166017 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2899924a-e6eb-4b53-8a3b-70d478330c8c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8ba823f313144da3b683bacef8fd62e0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81409) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:29:20.166681 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2899924a-e6eb-4b53-8a3b-70d478330c8c admin admin] REQUEST_METHOD: `DELETE` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:29:20.166955 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2899924a-e6eb-4b53-8a3b-70d478330c8c admin admin] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:29:20.167296 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2899924a-e6eb-4b53-8a3b-70d478330c8c admin admin] PATH_INFO: `/v3/users/a25ba64e178740c5a9f10a791553f1dc` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:29:20.169317 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2899924a-e6eb-4b53-8a3b-70d478330c8c admin admin] RBAC: Authorizing `identity:delete_user(user_id=a25ba64e178740c5a9f10a791553f1dc)` {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:29:20.171449 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2899924a-e6eb-4b53-8a3b-70d478330c8c admin admin] RBAC: Authorization granted {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:29:20.218604 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.notifications [None req-2899924a-e6eb-4b53-8a3b-70d478330c8c admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'a25ba64e178740c5a9f10a791553f1dc'} {{(pid=81409) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 13 04:29:20.219253 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.notifications [None req-2899924a-e6eb-4b53-8a3b-70d478330c8c admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'a25ba64e178740c5a9f10a791553f1dc'} {{(pid=81409) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 13 04:29:20.226151 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.notifications [None req-2899924a-e6eb-4b53-8a3b-70d478330c8c admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'a25ba64e178740c5a9f10a791553f1dc'} {{(pid=81409) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 13 04:29:20.226793 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.notifications [None req-2899924a-e6eb-4b53-8a3b-70d478330c8c admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'a25ba64e178740c5a9f10a791553f1dc'} {{(pid=81409) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 13 04:29:20.235937 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.notifications [None req-2899924a-e6eb-4b53-8a3b-70d478330c8c admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'a25ba64e178740c5a9f10a791553f1dc'} {{(pid=81409) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 13 04:29:20.254499 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 1243/2451] 10.222.0.27 () {58 vars in 1224 bytes} [Thu Oct 13 04:29:20 2022] DELETE /identity/v3/users/a25ba64e178740c5a9f10a791553f1dc => generated 0 bytes in 162 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 13 04:29:20.555893 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9f1cb6d4-516b-40f3-a7ce-dddc8013bcc9 admin admin] Authenticating user token {{(pid=81408) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:29:20.557371 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.fernet_utils [None req-9f1cb6d4-516b-40f3-a7ce-dddc8013bcc9 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81408) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:29:20.572634 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9f1cb6d4-516b-40f3-a7ce-dddc8013bcc9 admin admin] Validating token access rules against request {{(pid=81408) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:29:20.633563 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e3f8065e-1c61-4e8d-bf7f-7cf3cad34725 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8ba823f313144da3b683bacef8fd62e0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81408) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:29:20.634186 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3f8065e-1c61-4e8d-bf7f-7cf3cad34725 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:29:20.634496 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3f8065e-1c61-4e8d-bf7f-7cf3cad34725 admin admin] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:29:20.634769 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3f8065e-1c61-4e8d-bf7f-7cf3cad34725 admin admin] PATH_INFO: `/v3/projects/9ff74d24f31643ce9f136ee4b099ead6` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:29:20.635908 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e3f8065e-1c61-4e8d-bf7f-7cf3cad34725 admin admin] RBAC: Authorizing `identity:delete_project(project_id=9ff74d24f31643ce9f136ee4b099ead6)` {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:29:20.638618 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e3f8065e-1c61-4e8d-bf7f-7cf3cad34725 admin admin] RBAC: Authorization granted {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:29:20.640390 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG dbcounter [-] [81408] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=81408) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:29:20.692647 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.notifications [None req-e3f8065e-1c61-4e8d-bf7f-7cf3cad34725 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '9ff74d24f31643ce9f136ee4b099ead6', 'request_id': 'req-e3f8065e-1c61-4e8d-bf7f-7cf3cad34725'} {{(pid=81408) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 13 04:29:20.700722 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.notifications [None req-e3f8065e-1c61-4e8d-bf7f-7cf3cad34725 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '9ff74d24f31643ce9f136ee4b099ead6', 'request_id': 'req-e3f8065e-1c61-4e8d-bf7f-7cf3cad34725'} {{(pid=81408) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 13 04:29:20.707175 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.notifications [None req-e3f8065e-1c61-4e8d-bf7f-7cf3cad34725 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '9ff74d24f31643ce9f136ee4b099ead6', 'request_id': 'req-e3f8065e-1c61-4e8d-bf7f-7cf3cad34725'} {{(pid=81408) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 13 04:29:20.717669 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.notifications [None req-e3f8065e-1c61-4e8d-bf7f-7cf3cad34725 admin admin] The token cache is being invalidate because project 9ff74d24f31643ce9f136ee4b099ead6 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81408) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 13 04:29:20.717958 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.notifications [None req-e3f8065e-1c61-4e8d-bf7f-7cf3cad34725 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81408) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 13 04:29:20.718615 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.notifications [None req-e3f8065e-1c61-4e8d-bf7f-7cf3cad34725 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81408) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 13 04:29:20.720100 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 1209/2452] 10.222.0.27 () {58 vars in 1233 bytes} [Thu Oct 13 04:29:20 2022] DELETE /identity/v3/projects/9ff74d24f31643ce9f136ee4b099ead6 => generated 0 bytes in 165 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 13 04:29:20.724966 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9baddbf1-eec8-49a5-a96d-34f5acac4358 None None] REQUEST_METHOD: `POST` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:29:20.725465 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9baddbf1-eec8-49a5-a96d-34f5acac4358 None None] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:29:20.725580 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9baddbf1-eec8-49a5-a96d-34f5acac4358 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:29:20.748316 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.auth.core [None req-9baddbf1-eec8-49a5-a96d-34f5acac4358 None None] MFA Rules not processed for user `0e2d979563d6421995451a0e86bf3022`. Rule list: `[]` (Enabled: `True`). {{(pid=81409) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 13 04:29:20.749518 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.fernet_utils [None req-9baddbf1-eec8-49a5-a96d-34f5acac4358 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81409) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:29:20.814163 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 1244/2453] 10.222.0.27 () {58 vars in 958 bytes} [Thu Oct 13 04:29:20 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 90 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 13 04:29:20.821976 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-86680291-412c-4a16-88b7-150e674de1cb None None] REQUEST_METHOD: `POST` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:29:20.822311 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-86680291-412c-4a16-88b7-150e674de1cb None None] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:29:20.822601 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-86680291-412c-4a16-88b7-150e674de1cb None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:29:20.843717 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.auth.core [None req-86680291-412c-4a16-88b7-150e674de1cb None None] MFA Rules not processed for user `0e2d979563d6421995451a0e86bf3022`. Rule list: `[]` (Enabled: `True`). {{(pid=81408) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 13 04:29:20.845046 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.fernet_utils [None req-86680291-412c-4a16-88b7-150e674de1cb None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81408) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:29:20.853532 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 1210/2454] 10.222.0.27 () {58 vars in 958 bytes} [Thu Oct 13 04:29:20 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 32 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 13 04:29:20.858762 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9baddbf1-eec8-49a5-a96d-34f5acac4358 None None] Authenticating user token {{(pid=81409) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:29:20.868981 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9baddbf1-eec8-49a5-a96d-34f5acac4358 None None] Validating token access rules against request {{(pid=81409) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:29:20.874585 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fb17f850-9f00-4e2f-add5-c6a6dbc45799 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8ba823f313144da3b683bacef8fd62e0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81409) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:29:20.875196 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb17f850-9f00-4e2f-add5-c6a6dbc45799 admin admin] REQUEST_METHOD: `GET` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:29:20.875473 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb17f850-9f00-4e2f-add5-c6a6dbc45799 admin admin] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:29:20.875743 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb17f850-9f00-4e2f-add5-c6a6dbc45799 admin admin] PATH_INFO: `/v3/domains` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:29:20.876393 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fb17f850-9f00-4e2f-add5-c6a6dbc45799 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81409) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 13 04:29:20.876694 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fb17f850-9f00-4e2f-add5-c6a6dbc45799 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:29:20.878304 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fb17f850-9f00-4e2f-add5-c6a6dbc45799 admin admin] RBAC: Authorization granted {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:29:20.891368 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 1245/2455] 10.222.0.27 () {58 vars in 1153 bytes} [Thu Oct 13 04:29:20 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 33 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 13 04:29:20.895831 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-86680291-412c-4a16-88b7-150e674de1cb None None] Authenticating user token {{(pid=81408) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:29:20.905617 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-86680291-412c-4a16-88b7-150e674de1cb None None] Validating token access rules against request {{(pid=81408) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:29:20.910408 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0e797e28-6ae6-4fe6-aaa5-1dccea95d410 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8ba823f313144da3b683bacef8fd62e0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81408) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:29:20.911158 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e797e28-6ae6-4fe6-aaa5-1dccea95d410 admin admin] REQUEST_METHOD: `POST` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:29:20.911446 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e797e28-6ae6-4fe6-aaa5-1dccea95d410 admin admin] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:29:20.911718 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e797e28-6ae6-4fe6-aaa5-1dccea95d410 admin admin] PATH_INFO: `/v3/projects` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:29:20.912372 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0e797e28-6ae6-4fe6-aaa5-1dccea95d410 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:29:20.914329 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0e797e28-6ae6-4fe6-aaa5-1dccea95d410 admin admin] RBAC: Authorization granted {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:29:20.941941 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 1211/2456] 10.222.0.27 () {60 vars in 1153 bytes} [Thu Oct 13 04:29:20 2022] POST /identity/v3/projects => generated 383 bytes in 47 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 13 04:29:20.946171 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fb17f850-9f00-4e2f-add5-c6a6dbc45799 admin admin] Authenticating user token {{(pid=81409) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:29:20.958533 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fb17f850-9f00-4e2f-add5-c6a6dbc45799 admin admin] Validating token access rules against request {{(pid=81409) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:29:20.968591 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG dbcounter [-] [81409] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81409) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:29:21.021378 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e8d868f9-d4c7-43f3-a3db-140144f9d180 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8ba823f313144da3b683bacef8fd62e0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81409) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:29:21.021980 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8d868f9-d4c7-43f3-a3db-140144f9d180 admin admin] REQUEST_METHOD: `POST` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:29:21.022304 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8d868f9-d4c7-43f3-a3db-140144f9d180 admin admin] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:29:21.022583 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8d868f9-d4c7-43f3-a3db-140144f9d180 admin admin] PATH_INFO: `/v3/users` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:29:21.023203 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e8d868f9-d4c7-43f3-a3db-140144f9d180 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:29:21.025764 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e8d868f9-d4c7-43f3-a3db-140144f9d180 admin admin] RBAC: Authorization granted {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:29:21.050871 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 1246/2457] 10.222.0.27 () {60 vars in 1144 bytes} [Thu Oct 13 04:29:20 2022] POST /identity/v3/users => generated 292 bytes in 105 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 13 04:29:21.055150 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0e797e28-6ae6-4fe6-aaa5-1dccea95d410 admin admin] Authenticating user token {{(pid=81408) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:29:21.066323 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0e797e28-6ae6-4fe6-aaa5-1dccea95d410 admin admin] Validating token access rules against request {{(pid=81408) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:29:21.071205 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7e7a0a02-71f7-4e3c-9e93-0f5b00eaa98a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8ba823f313144da3b683bacef8fd62e0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81408) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:29:21.071829 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e7a0a02-71f7-4e3c-9e93-0f5b00eaa98a admin admin] REQUEST_METHOD: `GET` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:29:21.072113 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e7a0a02-71f7-4e3c-9e93-0f5b00eaa98a admin admin] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:29:21.072382 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e7a0a02-71f7-4e3c-9e93-0f5b00eaa98a admin admin] PATH_INFO: `/v3/roles` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:29:21.073019 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7e7a0a02-71f7-4e3c-9e93-0f5b00eaa98a admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:29:21.074879 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7e7a0a02-71f7-4e3c-9e93-0f5b00eaa98a admin admin] RBAC: Authorization granted {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:29:21.084787 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 1212/2458] 10.222.0.27 () {58 vars in 1122 bytes} [Thu Oct 13 04:29:21 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 13 04:29:21.089742 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e8d868f9-d4c7-43f3-a3db-140144f9d180 admin admin] Authenticating user token {{(pid=81409) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:29:21.100854 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e8d868f9-d4c7-43f3-a3db-140144f9d180 admin admin] Validating token access rules against request {{(pid=81409) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:29:21.106572 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ebb1ed08-7056-4000-938e-de0de22ff300 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8ba823f313144da3b683bacef8fd62e0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81409) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:29:21.107270 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ebb1ed08-7056-4000-938e-de0de22ff300 admin admin] REQUEST_METHOD: `GET` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:29:21.107551 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ebb1ed08-7056-4000-938e-de0de22ff300 admin admin] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:29:21.107816 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ebb1ed08-7056-4000-938e-de0de22ff300 admin admin] PATH_INFO: `/v3/roles` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:29:21.108456 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ebb1ed08-7056-4000-938e-de0de22ff300 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:29:21.110029 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ebb1ed08-7056-4000-938e-de0de22ff300 admin admin] RBAC: Authorization granted {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:29:21.121826 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 1247/2459] 10.222.0.27 () {58 vars in 1122 bytes} [Thu Oct 13 04:29:21 2022] GET /identity/v3/roles => generated 1420 bytes in 33 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Oct 13 04:29:21.126170 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7e7a0a02-71f7-4e3c-9e93-0f5b00eaa98a admin admin] Authenticating user token {{(pid=81408) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:29:21.138003 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7e7a0a02-71f7-4e3c-9e93-0f5b00eaa98a admin admin] Validating token access rules against request {{(pid=81408) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:29:21.143213 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0228e5a3-1f57-4b0d-9c46-afdc9324c8a2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8ba823f313144da3b683bacef8fd62e0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81408) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:29:21.143872 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0228e5a3-1f57-4b0d-9c46-afdc9324c8a2 admin admin] REQUEST_METHOD: `PUT` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:29:21.144148 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0228e5a3-1f57-4b0d-9c46-afdc9324c8a2 admin admin] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:29:21.144422 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0228e5a3-1f57-4b0d-9c46-afdc9324c8a2 admin admin] PATH_INFO: `/v3/projects/39d8e9f0ad8643d889208be9800c548a/users/5885c66ee3604653bd61621f8306f8f8/roles/cc56c4ae13a7424ea771c15da4b153c1` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:29:21.161440 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0228e5a3-1f57-4b0d-9c46-afdc9324c8a2 admin admin] RBAC: Authorizing `identity:create_grant(project_id=39d8e9f0ad8643d889208be9800c548a, user_id=5885c66ee3604653bd61621f8306f8f8, role_id=cc56c4ae13a7424ea771c15da4b153c1)` {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:29:21.164160 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0228e5a3-1f57-4b0d-9c46-afdc9324c8a2 admin admin] RBAC: Authorization granted {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:29:21.177590 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 1213/2460] 10.222.0.27 () {60 vars in 1483 bytes} [Thu Oct 13 04:29:21 2022] PUT /identity/v3/projects/39d8e9f0ad8643d889208be9800c548a/users/5885c66ee3604653bd61621f8306f8f8/roles/cc56c4ae13a7424ea771c15da4b153c1 => generated 0 bytes in 52 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 13 04:29:21.189342 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6304e521-f3c4-497a-8c9d-d8b3e42fe936 None None] REQUEST_METHOD: `POST` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:29:21.189651 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6304e521-f3c4-497a-8c9d-d8b3e42fe936 None None] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:29:21.189927 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6304e521-f3c4-497a-8c9d-d8b3e42fe936 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:29:21.220301 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.auth.core [None req-6304e521-f3c4-497a-8c9d-d8b3e42fe936 None None] MFA Rules not processed for user `5885c66ee3604653bd61621f8306f8f8`. Rule list: `[]` (Enabled: `True`). {{(pid=81409) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 13 04:29:21.221541 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.fernet_utils [None req-6304e521-f3c4-497a-8c9d-d8b3e42fe936 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81409) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:29:21.289373 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 1248/2461] 10.222.0.27 () {58 vars in 958 bytes} [Thu Oct 13 04:29:21 2022] POST /identity/v3/auth/tokens => generated 3260 bytes in 101 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 13 04:29:21.293938 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0228e5a3-1f57-4b0d-9c46-afdc9324c8a2 admin admin] Authenticating user token {{(pid=81408) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:29:21.362158 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0228e5a3-1f57-4b0d-9c46-afdc9324c8a2 admin admin] Validating token access rules against request {{(pid=81408) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:29:21.366779 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-46c965e4-1099-469d-b934-65f51fdda8ee admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8ba823f313144da3b683bacef8fd62e0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81408) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:29:21.367510 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46c965e4-1099-469d-b934-65f51fdda8ee admin admin] REQUEST_METHOD: `POST` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:29:21.367998 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46c965e4-1099-469d-b934-65f51fdda8ee admin admin] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:29:21.368437 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46c965e4-1099-469d-b934-65f51fdda8ee admin admin] PATH_INFO: `/v3/projects` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:29:21.369272 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-46c965e4-1099-469d-b934-65f51fdda8ee admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:29:21.372637 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-46c965e4-1099-469d-b934-65f51fdda8ee admin admin] RBAC: Authorization granted {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:29:21.415324 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 1214/2462] 10.222.0.27 () {60 vars in 1153 bytes} [Thu Oct 13 04:29:21 2022] POST /identity/v3/projects => generated 383 bytes in 122 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 13 04:29:21.421507 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6304e521-f3c4-497a-8c9d-d8b3e42fe936 None None] Authenticating user token {{(pid=81409) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:29:21.432165 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6304e521-f3c4-497a-8c9d-d8b3e42fe936 None None] Validating token access rules against request {{(pid=81409) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:29:21.499902 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b522c2ce-a0ea-404e-a712-1de4a35529b1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8ba823f313144da3b683bacef8fd62e0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81409) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:29:21.500512 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b522c2ce-a0ea-404e-a712-1de4a35529b1 admin admin] REQUEST_METHOD: `POST` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:29:21.500789 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b522c2ce-a0ea-404e-a712-1de4a35529b1 admin admin] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:29:21.501059 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b522c2ce-a0ea-404e-a712-1de4a35529b1 admin admin] PATH_INFO: `/v3/users` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:29:21.501700 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b522c2ce-a0ea-404e-a712-1de4a35529b1 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:29:21.503652 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b522c2ce-a0ea-404e-a712-1de4a35529b1 admin admin] RBAC: Authorization granted {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:29:21.525715 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 1249/2463] 10.222.0.27 () {60 vars in 1144 bytes} [Thu Oct 13 04:29:21 2022] POST /identity/v3/users => generated 298 bytes in 106 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 13 04:29:21.529814 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-46c965e4-1099-469d-b934-65f51fdda8ee admin admin] Authenticating user token {{(pid=81408) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:29:21.540892 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-46c965e4-1099-469d-b934-65f51fdda8ee admin admin] Validating token access rules against request {{(pid=81408) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:29:21.546274 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d18ea271-50a0-486b-ab95-05859a48a40f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8ba823f313144da3b683bacef8fd62e0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81408) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:29:21.546895 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d18ea271-50a0-486b-ab95-05859a48a40f admin admin] REQUEST_METHOD: `GET` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:29:21.547170 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d18ea271-50a0-486b-ab95-05859a48a40f admin admin] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:29:21.547438 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d18ea271-50a0-486b-ab95-05859a48a40f admin admin] PATH_INFO: `/v3/roles` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:29:21.548078 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d18ea271-50a0-486b-ab95-05859a48a40f admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:29:21.549688 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d18ea271-50a0-486b-ab95-05859a48a40f admin admin] RBAC: Authorization granted {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:29:21.555977 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG dbcounter [-] [81408] Writing DB stats keystone:SELECT=89,keystone:DELETE=7,keystone:INSERT=4 {{(pid=81408) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:29:21.562019 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 1215/2464] 10.222.0.27 () {58 vars in 1122 bytes} [Thu Oct 13 04:29:21 2022] GET /identity/v3/roles => generated 1420 bytes in 33 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 13 04:29:21.566346 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b522c2ce-a0ea-404e-a712-1de4a35529b1 admin admin] Authenticating user token {{(pid=81409) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:29:21.577341 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b522c2ce-a0ea-404e-a712-1de4a35529b1 admin admin] Validating token access rules against request {{(pid=81409) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:29:21.583100 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6410b3b8-2c63-4d01-b43e-0aace38af84c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8ba823f313144da3b683bacef8fd62e0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81409) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:29:21.583765 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6410b3b8-2c63-4d01-b43e-0aace38af84c admin admin] REQUEST_METHOD: `PUT` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:29:21.584037 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6410b3b8-2c63-4d01-b43e-0aace38af84c admin admin] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:29:21.584305 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6410b3b8-2c63-4d01-b43e-0aace38af84c admin admin] PATH_INFO: `/v3/projects/3d15f86d949e42ba8798b370153cc99b/users/8b0d1960be3344c78ad7901386ee85db/roles/ef55fbd94e1040be83695754075a7ed5` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:29:21.593717 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG dbcounter [-] [81409] Writing DB stats keystone:SELECT=94,keystone:INSERT=6 {{(pid=81409) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:29:21.603269 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6410b3b8-2c63-4d01-b43e-0aace38af84c admin admin] RBAC: Authorizing `identity:create_grant(project_id=3d15f86d949e42ba8798b370153cc99b, user_id=8b0d1960be3344c78ad7901386ee85db, role_id=ef55fbd94e1040be83695754075a7ed5)` {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:29:21.606973 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6410b3b8-2c63-4d01-b43e-0aace38af84c admin admin] RBAC: Authorization granted {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:29:21.620376 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 1250/2465] 10.222.0.27 () {60 vars in 1483 bytes} [Thu Oct 13 04:29:21 2022] PUT /identity/v3/projects/3d15f86d949e42ba8798b370153cc99b/users/8b0d1960be3344c78ad7901386ee85db/roles/ef55fbd94e1040be83695754075a7ed5 => generated 0 bytes in 55 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 13 04:29:21.624911 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d18ea271-50a0-486b-ab95-05859a48a40f admin admin] Authenticating user token {{(pid=81408) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:29:21.635433 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d18ea271-50a0-486b-ab95-05859a48a40f admin admin] Validating token access rules against request {{(pid=81408) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:29:21.693158 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1f10648c-e262-45d3-b885-7669d771c14e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8ba823f313144da3b683bacef8fd62e0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81408) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:29:21.693763 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f10648c-e262-45d3-b885-7669d771c14e admin admin] REQUEST_METHOD: `GET` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:29:21.694041 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f10648c-e262-45d3-b885-7669d771c14e admin admin] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:29:21.694343 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f10648c-e262-45d3-b885-7669d771c14e admin admin] PATH_INFO: `/v3/roles` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:29:21.694986 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1f10648c-e262-45d3-b885-7669d771c14e admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:29:21.696575 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1f10648c-e262-45d3-b885-7669d771c14e admin admin] RBAC: Authorization granted {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:29:21.706329 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 1216/2466] 10.222.0.27 () {58 vars in 1122 bytes} [Thu Oct 13 04:29:21 2022] GET /identity/v3/roles => generated 1420 bytes in 82 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 13 04:29:21.710517 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6410b3b8-2c63-4d01-b43e-0aace38af84c admin admin] Authenticating user token {{(pid=81409) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:29:21.721879 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6410b3b8-2c63-4d01-b43e-0aace38af84c admin admin] Validating token access rules against request {{(pid=81409) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:29:21.726687 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f1877500-1e15-4ce6-83f2-d8be8c412015 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8ba823f313144da3b683bacef8fd62e0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81409) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:29:21.727354 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1877500-1e15-4ce6-83f2-d8be8c412015 admin admin] REQUEST_METHOD: `PUT` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:29:21.727633 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1877500-1e15-4ce6-83f2-d8be8c412015 admin admin] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:29:21.727901 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1877500-1e15-4ce6-83f2-d8be8c412015 admin admin] PATH_INFO: `/v3/projects/3d15f86d949e42ba8798b370153cc99b/users/8b0d1960be3344c78ad7901386ee85db/roles/ef55fbd94e1040be83695754075a7ed5` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:29:21.729832 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f1877500-1e15-4ce6-83f2-d8be8c412015 admin admin] RBAC: Authorizing `identity:create_grant(project_id=3d15f86d949e42ba8798b370153cc99b, user_id=8b0d1960be3344c78ad7901386ee85db, role_id=ef55fbd94e1040be83695754075a7ed5)` {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:29:21.733594 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f1877500-1e15-4ce6-83f2-d8be8c412015 admin admin] RBAC: Authorization granted {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:29:21.744249 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 1251/2467] 10.222.0.27 () {60 vars in 1483 bytes} [Thu Oct 13 04:29:21 2022] PUT /identity/v3/projects/3d15f86d949e42ba8798b370153cc99b/users/8b0d1960be3344c78ad7901386ee85db/roles/ef55fbd94e1040be83695754075a7ed5 => generated 0 bytes in 34 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 13 04:29:21.754443 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-967b1570-50e0-446e-8743-0e0a0fb0a0c9 None None] REQUEST_METHOD: `POST` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:29:21.754729 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-967b1570-50e0-446e-8743-0e0a0fb0a0c9 None None] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:29:21.754999 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-967b1570-50e0-446e-8743-0e0a0fb0a0c9 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:29:21.786269 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.auth.core [None req-967b1570-50e0-446e-8743-0e0a0fb0a0c9 None None] MFA Rules not processed for user `8b0d1960be3344c78ad7901386ee85db`. Rule list: `[]` (Enabled: `True`). {{(pid=81408) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 13 04:29:21.787732 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.fernet_utils [None req-967b1570-50e0-446e-8743-0e0a0fb0a0c9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81408) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:29:21.861702 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 1217/2468] 10.222.0.27 () {58 vars in 958 bytes} [Thu Oct 13 04:29:21 2022] POST /identity/v3/auth/tokens => generated 3327 bytes in 108 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 13 04:29:21.874047 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f1877500-1e15-4ce6-83f2-d8be8c412015 admin admin] Authenticating user token {{(pid=81409) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:29:21.876095 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.fernet_utils [None req-f1877500-1e15-4ce6-83f2-d8be8c412015 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81409) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:29:21.937789 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f1877500-1e15-4ce6-83f2-d8be8c412015 admin admin] Validating token access rules against request {{(pid=81409) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:29:21.941716 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-36b09e77-8982-4a80-bb64-05279d6e3b8f service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6141b8a6c3d54e179d6879bb019f4d12', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dcec0431fb6849499d6aec35a234daa6', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81409) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:29:21.942466 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36b09e77-8982-4a80-bb64-05279d6e3b8f service cinder] REQUEST_METHOD: `GET` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:29:21.942746 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36b09e77-8982-4a80-bb64-05279d6e3b8f service cinder] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:29:21.943013 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36b09e77-8982-4a80-bb64-05279d6e3b8f service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:29:22.002373 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-36b09e77-8982-4a80-bb64-05279d6e3b8f service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:29:22.003816 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-36b09e77-8982-4a80-bb64-05279d6e3b8f service cinder] RBAC: Authorization granted {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:29:22.014560 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 1252/2469] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Oct 13 04:29:21 2022] GET /identity/v3/auth/tokens => generated 3260 bytes in 141 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Oct 13 04:29:26.281669 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-967b1570-50e0-446e-8743-0e0a0fb0a0c9 None None] Authenticating user token {{(pid=81408) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:29:26.303011 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-967b1570-50e0-446e-8743-0e0a0fb0a0c9 None None] Validating token access rules against request {{(pid=81408) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:29:26.306642 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c2f1c5f3-2685-4aee-b55c-51c5bc741cb7 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6141b8a6c3d54e179d6879bb019f4d12', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dcec0431fb6849499d6aec35a234daa6', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81408) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:29:26.307241 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2f1c5f3-2685-4aee-b55c-51c5bc741cb7 service cinder] REQUEST_METHOD: `GET` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:29:26.307514 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2f1c5f3-2685-4aee-b55c-51c5bc741cb7 service cinder] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:29:26.307783 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2f1c5f3-2685-4aee-b55c-51c5bc741cb7 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:29:26.317005 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c2f1c5f3-2685-4aee-b55c-51c5bc741cb7 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:29:26.318483 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c2f1c5f3-2685-4aee-b55c-51c5bc741cb7 service cinder] RBAC: Authorization granted {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:29:26.328923 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 1218/2470] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Oct 13 04:29:26 2022] GET /identity/v3/auth/tokens => generated 3327 bytes in 48 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 13 04:29:31.241742 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-36b09e77-8982-4a80-bb64-05279d6e3b8f service cinder] Authenticating user token {{(pid=81409) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:29:31.305641 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-36b09e77-8982-4a80-bb64-05279d6e3b8f service cinder] Validating token access rules against request {{(pid=81409) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:29:31.310245 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a06461a4-9a33-4fc5-b82f-74e6ff26266d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8ba823f313144da3b683bacef8fd62e0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81409) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:29:31.311613 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a06461a4-9a33-4fc5-b82f-74e6ff26266d admin admin] REQUEST_METHOD: `DELETE` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:29:31.312058 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a06461a4-9a33-4fc5-b82f-74e6ff26266d admin admin] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:29:31.312515 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a06461a4-9a33-4fc5-b82f-74e6ff26266d admin admin] PATH_INFO: `/v3/users/5885c66ee3604653bd61621f8306f8f8` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:29:31.313649 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a06461a4-9a33-4fc5-b82f-74e6ff26266d admin admin] RBAC: Authorizing `identity:delete_user(user_id=5885c66ee3604653bd61621f8306f8f8)` {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:29:31.315588 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a06461a4-9a33-4fc5-b82f-74e6ff26266d admin admin] RBAC: Authorization granted {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:29:31.350225 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG dbcounter [-] [81409] Writing DB stats keystone:SELECT=93,keystone:INSERT=2,keystone:DELETE=5 {{(pid=81409) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:29:31.366874 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.notifications [None req-a06461a4-9a33-4fc5-b82f-74e6ff26266d admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '5885c66ee3604653bd61621f8306f8f8'} {{(pid=81409) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 13 04:29:31.367536 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.notifications [None req-a06461a4-9a33-4fc5-b82f-74e6ff26266d admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '5885c66ee3604653bd61621f8306f8f8'} {{(pid=81409) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 13 04:29:31.375147 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.notifications [None req-a06461a4-9a33-4fc5-b82f-74e6ff26266d admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '5885c66ee3604653bd61621f8306f8f8'} {{(pid=81409) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 13 04:29:31.375781 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.notifications [None req-a06461a4-9a33-4fc5-b82f-74e6ff26266d admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '5885c66ee3604653bd61621f8306f8f8'} {{(pid=81409) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 13 04:29:31.384818 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.notifications [None req-a06461a4-9a33-4fc5-b82f-74e6ff26266d admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '5885c66ee3604653bd61621f8306f8f8'} {{(pid=81409) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 13 04:29:31.402808 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 1253/2471] 10.222.0.27 () {58 vars in 1224 bytes} [Thu Oct 13 04:29:31 2022] DELETE /identity/v3/users/5885c66ee3604653bd61621f8306f8f8 => generated 0 bytes in 162 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 13 04:29:31.407895 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3619b714-2927-41e5-8578-2b869ca4ca11 None None] REQUEST_METHOD: `POST` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:29:31.408403 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3619b714-2927-41e5-8578-2b869ca4ca11 None None] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:29:31.408504 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3619b714-2927-41e5-8578-2b869ca4ca11 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:29:31.430342 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.auth.core [None req-3619b714-2927-41e5-8578-2b869ca4ca11 None None] MFA Rules not processed for user `0e2d979563d6421995451a0e86bf3022`. Rule list: `[]` (Enabled: `True`). {{(pid=81408) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 13 04:29:31.431548 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.fernet_utils [None req-3619b714-2927-41e5-8578-2b869ca4ca11 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81408) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:29:31.484234 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG dbcounter [-] [81408] Writing DB stats keystone:SELECT=100 {{(pid=81408) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:29:31.493643 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 1219/2472] 10.222.0.27 () {58 vars in 958 bytes} [Thu Oct 13 04:29:31 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 87 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 13 04:29:31.503731 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a06461a4-9a33-4fc5-b82f-74e6ff26266d admin admin] Authenticating user token {{(pid=81409) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:29:31.505125 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.fernet_utils [None req-a06461a4-9a33-4fc5-b82f-74e6ff26266d admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81409) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:29:31.562109 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a06461a4-9a33-4fc5-b82f-74e6ff26266d admin admin] Validating token access rules against request {{(pid=81409) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:29:31.565768 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-549fd2f8-2afc-44bb-8498-d344af37c955 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd8812c3974f84871bb2d761cf0016257', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dcec0431fb6849499d6aec35a234daa6', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81409) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:29:31.566407 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-549fd2f8-2afc-44bb-8498-d344af37c955 service neutron] REQUEST_METHOD: `GET` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:29:31.566681 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-549fd2f8-2afc-44bb-8498-d344af37c955 service neutron] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:29:31.566948 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-549fd2f8-2afc-44bb-8498-d344af37c955 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:29:31.575834 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-549fd2f8-2afc-44bb-8498-d344af37c955 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:29:31.577311 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-549fd2f8-2afc-44bb-8498-d344af37c955 service neutron] RBAC: Authorization granted {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:29:31.586668 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 1254/2473] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Oct 13 04:29:31 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 83 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 13 04:29:31.860525 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3619b714-2927-41e5-8578-2b869ca4ca11 None None] Authenticating user token {{(pid=81408) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:29:31.862122 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.fernet_utils [None req-3619b714-2927-41e5-8578-2b869ca4ca11 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81408) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:29:31.877783 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3619b714-2927-41e5-8578-2b869ca4ca11 None None] Validating token access rules against request {{(pid=81408) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:29:31.882767 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-38bbaa3a-3d40-4bc3-97b9-db82f2d0bdcb admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8ba823f313144da3b683bacef8fd62e0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81408) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:29:31.883599 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38bbaa3a-3d40-4bc3-97b9-db82f2d0bdcb admin admin] REQUEST_METHOD: `DELETE` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:29:31.883880 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38bbaa3a-3d40-4bc3-97b9-db82f2d0bdcb admin admin] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:29:31.884151 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38bbaa3a-3d40-4bc3-97b9-db82f2d0bdcb admin admin] PATH_INFO: `/v3/projects/39d8e9f0ad8643d889208be9800c548a` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:29:31.885294 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-38bbaa3a-3d40-4bc3-97b9-db82f2d0bdcb admin admin] RBAC: Authorizing `identity:delete_project(project_id=39d8e9f0ad8643d889208be9800c548a)` {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:29:31.887260 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-38bbaa3a-3d40-4bc3-97b9-db82f2d0bdcb admin admin] RBAC: Authorization granted {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:29:31.939022 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.notifications [None req-38bbaa3a-3d40-4bc3-97b9-db82f2d0bdcb admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '39d8e9f0ad8643d889208be9800c548a', 'request_id': 'req-38bbaa3a-3d40-4bc3-97b9-db82f2d0bdcb'} {{(pid=81408) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 13 04:29:31.948350 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.notifications [None req-38bbaa3a-3d40-4bc3-97b9-db82f2d0bdcb admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '39d8e9f0ad8643d889208be9800c548a', 'request_id': 'req-38bbaa3a-3d40-4bc3-97b9-db82f2d0bdcb'} {{(pid=81408) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 13 04:29:31.956238 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.notifications [None req-38bbaa3a-3d40-4bc3-97b9-db82f2d0bdcb admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '39d8e9f0ad8643d889208be9800c548a', 'request_id': 'req-38bbaa3a-3d40-4bc3-97b9-db82f2d0bdcb'} {{(pid=81408) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 13 04:29:31.972443 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.notifications [None req-38bbaa3a-3d40-4bc3-97b9-db82f2d0bdcb admin admin] The token cache is being invalidate because project 39d8e9f0ad8643d889208be9800c548a was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81408) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 13 04:29:31.972729 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.notifications [None req-38bbaa3a-3d40-4bc3-97b9-db82f2d0bdcb admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81408) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 13 04:29:31.973381 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.notifications [None req-38bbaa3a-3d40-4bc3-97b9-db82f2d0bdcb admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81408) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 13 04:29:31.974938 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 1220/2474] 10.222.0.27 () {58 vars in 1233 bytes} [Thu Oct 13 04:29:31 2022] DELETE /identity/v3/projects/39d8e9f0ad8643d889208be9800c548a => generated 0 bytes in 115 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 13 04:29:31.978832 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-549fd2f8-2afc-44bb-8498-d344af37c955 service neutron] Authenticating user token {{(pid=81409) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:29:31.990082 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-549fd2f8-2afc-44bb-8498-d344af37c955 service neutron] Validating token access rules against request {{(pid=81409) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:29:32.049368 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-90c80ef6-7a4f-4da9-9614-1039343110a7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8ba823f313144da3b683bacef8fd62e0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81409) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:29:32.049986 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90c80ef6-7a4f-4da9-9614-1039343110a7 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:29:32.050284 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90c80ef6-7a4f-4da9-9614-1039343110a7 admin admin] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:29:32.050551 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90c80ef6-7a4f-4da9-9614-1039343110a7 admin admin] PATH_INFO: `/v3/users/8b0d1960be3344c78ad7901386ee85db` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:29:32.051647 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-90c80ef6-7a4f-4da9-9614-1039343110a7 admin admin] RBAC: Authorizing `identity:delete_user(user_id=8b0d1960be3344c78ad7901386ee85db)` {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:29:32.053538 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-90c80ef6-7a4f-4da9-9614-1039343110a7 admin admin] RBAC: Authorization granted {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:29:32.101259 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.notifications [None req-90c80ef6-7a4f-4da9-9614-1039343110a7 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '8b0d1960be3344c78ad7901386ee85db'} {{(pid=81409) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 13 04:29:32.101902 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.notifications [None req-90c80ef6-7a4f-4da9-9614-1039343110a7 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '8b0d1960be3344c78ad7901386ee85db'} {{(pid=81409) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 13 04:29:32.106720 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG dbcounter [-] [81409] Writing DB stats keystone:SELECT=87,keystone:DELETE=12,keystone:INSERT=1 {{(pid=81409) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:29:32.112416 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.notifications [None req-90c80ef6-7a4f-4da9-9614-1039343110a7 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '8b0d1960be3344c78ad7901386ee85db'} {{(pid=81409) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 13 04:29:32.113338 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.notifications [None req-90c80ef6-7a4f-4da9-9614-1039343110a7 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '8b0d1960be3344c78ad7901386ee85db'} {{(pid=81409) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 13 04:29:32.121121 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.notifications [None req-90c80ef6-7a4f-4da9-9614-1039343110a7 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '8b0d1960be3344c78ad7901386ee85db'} {{(pid=81409) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 13 04:29:32.140963 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 1255/2475] 10.222.0.27 () {58 vars in 1224 bytes} [Thu Oct 13 04:29:31 2022] DELETE /identity/v3/users/8b0d1960be3344c78ad7901386ee85db => generated 0 bytes in 162 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 13 04:29:32.409426 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-38bbaa3a-3d40-4bc3-97b9-db82f2d0bdcb admin admin] Authenticating user token {{(pid=81408) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:29:32.410939 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.fernet_utils [None req-38bbaa3a-3d40-4bc3-97b9-db82f2d0bdcb admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81408) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:29:32.428892 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-38bbaa3a-3d40-4bc3-97b9-db82f2d0bdcb admin admin] Validating token access rules against request {{(pid=81408) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:29:32.491754 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f7ce8a72-242f-436f-a552-975829d101e3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8ba823f313144da3b683bacef8fd62e0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81408) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:29:32.492312 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7ce8a72-242f-436f-a552-975829d101e3 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:29:32.492593 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7ce8a72-242f-436f-a552-975829d101e3 admin admin] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:29:32.492867 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7ce8a72-242f-436f-a552-975829d101e3 admin admin] PATH_INFO: `/v3/projects/3d15f86d949e42ba8798b370153cc99b` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:29:32.494001 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f7ce8a72-242f-436f-a552-975829d101e3 admin admin] RBAC: Authorizing `identity:delete_project(project_id=3d15f86d949e42ba8798b370153cc99b)` {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:29:32.495950 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f7ce8a72-242f-436f-a552-975829d101e3 admin admin] RBAC: Authorization granted {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:29:32.541387 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.notifications [None req-f7ce8a72-242f-436f-a552-975829d101e3 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '3d15f86d949e42ba8798b370153cc99b', 'request_id': 'req-f7ce8a72-242f-436f-a552-975829d101e3'} {{(pid=81408) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 13 04:29:32.547890 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.notifications [None req-f7ce8a72-242f-436f-a552-975829d101e3 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '3d15f86d949e42ba8798b370153cc99b', 'request_id': 'req-f7ce8a72-242f-436f-a552-975829d101e3'} {{(pid=81408) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 13 04:29:32.554091 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.notifications [None req-f7ce8a72-242f-436f-a552-975829d101e3 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '3d15f86d949e42ba8798b370153cc99b', 'request_id': 'req-f7ce8a72-242f-436f-a552-975829d101e3'} {{(pid=81408) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 13 04:29:32.564820 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.notifications [None req-f7ce8a72-242f-436f-a552-975829d101e3 admin admin] The token cache is being invalidate because project 3d15f86d949e42ba8798b370153cc99b was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81408) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 13 04:29:32.565107 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.notifications [None req-f7ce8a72-242f-436f-a552-975829d101e3 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81408) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 13 04:29:32.565741 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.notifications [None req-f7ce8a72-242f-436f-a552-975829d101e3 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81408) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 13 04:29:32.567237 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 1221/2476] 10.222.0.27 () {58 vars in 1233 bytes} [Thu Oct 13 04:29:32 2022] DELETE /identity/v3/projects/3d15f86d949e42ba8798b370153cc99b => generated 0 bytes in 158 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 13 04:29:32.573337 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7771f8e7-151f-4823-af10-06a54a1cd1ce None None] REQUEST_METHOD: `POST` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:29:32.573822 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7771f8e7-151f-4823-af10-06a54a1cd1ce None None] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:29:32.573929 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7771f8e7-151f-4823-af10-06a54a1cd1ce None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:29:32.596036 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.auth.core [None req-7771f8e7-151f-4823-af10-06a54a1cd1ce None None] MFA Rules not processed for user `0e2d979563d6421995451a0e86bf3022`. Rule list: `[]` (Enabled: `True`). {{(pid=81409) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 13 04:29:32.597250 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.fernet_utils [None req-7771f8e7-151f-4823-af10-06a54a1cd1ce None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81409) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:29:32.658469 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 1256/2477] 10.222.0.27 () {58 vars in 958 bytes} [Thu Oct 13 04:29:32 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 86 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 13 04:29:32.667104 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2883a29e-711d-45ea-99cd-0741af80248c None None] REQUEST_METHOD: `POST` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:29:32.667390 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2883a29e-711d-45ea-99cd-0741af80248c None None] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:29:32.667662 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2883a29e-711d-45ea-99cd-0741af80248c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:29:32.688076 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.auth.core [None req-2883a29e-711d-45ea-99cd-0741af80248c None None] MFA Rules not processed for user `0e2d979563d6421995451a0e86bf3022`. Rule list: `[]` (Enabled: `True`). {{(pid=81408) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 13 04:29:32.689268 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.fernet_utils [None req-2883a29e-711d-45ea-99cd-0741af80248c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81408) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:29:32.697112 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 1222/2478] 10.222.0.27 () {58 vars in 958 bytes} [Thu Oct 13 04:29:32 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 31 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 13 04:29:32.701430 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7771f8e7-151f-4823-af10-06a54a1cd1ce None None] Authenticating user token {{(pid=81409) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:29:32.711158 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7771f8e7-151f-4823-af10-06a54a1cd1ce None None] Validating token access rules against request {{(pid=81409) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:29:32.715613 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0a08e7aa-a217-40e1-a384-5d7a86ec8852 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8ba823f313144da3b683bacef8fd62e0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81409) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:29:32.716206 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a08e7aa-a217-40e1-a384-5d7a86ec8852 admin admin] REQUEST_METHOD: `GET` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:29:32.716480 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a08e7aa-a217-40e1-a384-5d7a86ec8852 admin admin] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:29:32.716750 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a08e7aa-a217-40e1-a384-5d7a86ec8852 admin admin] PATH_INFO: `/v3/domains` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:29:32.717646 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0a08e7aa-a217-40e1-a384-5d7a86ec8852 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81409) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 13 04:29:32.717971 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0a08e7aa-a217-40e1-a384-5d7a86ec8852 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:29:32.720090 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0a08e7aa-a217-40e1-a384-5d7a86ec8852 admin admin] RBAC: Authorization granted {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:29:32.735794 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 1257/2479] 10.222.0.27 () {58 vars in 1153 bytes} [Thu Oct 13 04:29:32 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 35 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 13 04:29:32.740608 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2883a29e-711d-45ea-99cd-0741af80248c None None] Authenticating user token {{(pid=81408) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:29:32.750643 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2883a29e-711d-45ea-99cd-0741af80248c None None] Validating token access rules against request {{(pid=81408) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:29:32.755393 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d088a2bf-6461-4328-9cf2-4f6c1c01f98e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8ba823f313144da3b683bacef8fd62e0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81408) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:29:32.755984 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d088a2bf-6461-4328-9cf2-4f6c1c01f98e admin admin] REQUEST_METHOD: `POST` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:29:32.756262 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d088a2bf-6461-4328-9cf2-4f6c1c01f98e admin admin] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:29:32.756534 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d088a2bf-6461-4328-9cf2-4f6c1c01f98e admin admin] PATH_INFO: `/v3/projects` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:29:32.757157 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d088a2bf-6461-4328-9cf2-4f6c1c01f98e admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:29:32.759112 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d088a2bf-6461-4328-9cf2-4f6c1c01f98e admin admin] RBAC: Authorization granted {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:29:32.763620 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG dbcounter [-] [81408] Writing DB stats keystone:SELECT=84,keystone:DELETE=14,keystone:INSERT=2 {{(pid=81408) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:29:32.789167 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 1223/2480] 10.222.0.27 () {60 vars in 1153 bytes} [Thu Oct 13 04:29:32 2022] POST /identity/v3/projects => generated 373 bytes in 49 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 13 04:29:32.793654 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0a08e7aa-a217-40e1-a384-5d7a86ec8852 admin admin] Authenticating user token {{(pid=81409) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:29:32.804193 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0a08e7aa-a217-40e1-a384-5d7a86ec8852 admin admin] Validating token access rules against request {{(pid=81409) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:29:32.860079 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9de72a25-7c93-44be-a345-7ccde2307337 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8ba823f313144da3b683bacef8fd62e0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81409) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:29:32.860667 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9de72a25-7c93-44be-a345-7ccde2307337 admin admin] REQUEST_METHOD: `POST` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:29:32.860938 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9de72a25-7c93-44be-a345-7ccde2307337 admin admin] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:29:32.861207 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9de72a25-7c93-44be-a345-7ccde2307337 admin admin] PATH_INFO: `/v3/users` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:29:32.861829 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9de72a25-7c93-44be-a345-7ccde2307337 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:29:32.863744 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9de72a25-7c93-44be-a345-7ccde2307337 admin admin] RBAC: Authorization granted {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:29:32.886124 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 1258/2481] 10.222.0.27 () {60 vars in 1144 bytes} [Thu Oct 13 04:29:32 2022] POST /identity/v3/users => generated 287 bytes in 93 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 13 04:29:32.890813 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d088a2bf-6461-4328-9cf2-4f6c1c01f98e admin admin] Authenticating user token {{(pid=81408) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:29:32.901019 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d088a2bf-6461-4328-9cf2-4f6c1c01f98e admin admin] Validating token access rules against request {{(pid=81408) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:29:32.906014 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1bead241-9524-46f4-8315-deb74505ad37 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8ba823f313144da3b683bacef8fd62e0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81408) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:29:32.906645 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1bead241-9524-46f4-8315-deb74505ad37 admin admin] REQUEST_METHOD: `GET` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:29:32.906919 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1bead241-9524-46f4-8315-deb74505ad37 admin admin] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:29:32.907184 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1bead241-9524-46f4-8315-deb74505ad37 admin admin] PATH_INFO: `/v3/roles` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:29:32.907814 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1bead241-9524-46f4-8315-deb74505ad37 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:29:32.909397 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1bead241-9524-46f4-8315-deb74505ad37 admin admin] RBAC: Authorization granted {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:29:32.919170 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 1224/2482] 10.222.0.27 () {58 vars in 1122 bytes} [Thu Oct 13 04:29:32 2022] GET /identity/v3/roles => generated 1420 bytes in 29 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 13 04:29:32.923689 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9de72a25-7c93-44be-a345-7ccde2307337 admin admin] Authenticating user token {{(pid=81409) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:29:32.933353 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9de72a25-7c93-44be-a345-7ccde2307337 admin admin] Validating token access rules against request {{(pid=81409) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:29:32.938984 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-94b15afc-3bdb-498a-bc29-a6e2591b0aa9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8ba823f313144da3b683bacef8fd62e0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81409) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:29:32.939534 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94b15afc-3bdb-498a-bc29-a6e2591b0aa9 admin admin] REQUEST_METHOD: `GET` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:29:32.939808 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94b15afc-3bdb-498a-bc29-a6e2591b0aa9 admin admin] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:29:32.940072 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94b15afc-3bdb-498a-bc29-a6e2591b0aa9 admin admin] PATH_INFO: `/v3/roles` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:29:32.940699 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-94b15afc-3bdb-498a-bc29-a6e2591b0aa9 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:29:32.942298 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-94b15afc-3bdb-498a-bc29-a6e2591b0aa9 admin admin] RBAC: Authorization granted {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:29:32.957025 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 1259/2483] 10.222.0.27 () {58 vars in 1122 bytes} [Thu Oct 13 04:29:32 2022] GET /identity/v3/roles => generated 1420 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 13 04:29:32.959377 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1bead241-9524-46f4-8315-deb74505ad37 admin admin] Authenticating user token {{(pid=81408) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:29:32.970233 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1bead241-9524-46f4-8315-deb74505ad37 admin admin] Validating token access rules against request {{(pid=81408) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:29:32.975025 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-558b92dc-03d6-40f0-9a83-84e7c25b5cbb admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8ba823f313144da3b683bacef8fd62e0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81408) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:29:32.975840 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-558b92dc-03d6-40f0-9a83-84e7c25b5cbb admin admin] REQUEST_METHOD: `PUT` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:29:32.975948 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-558b92dc-03d6-40f0-9a83-84e7c25b5cbb admin admin] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:29:32.976180 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-558b92dc-03d6-40f0-9a83-84e7c25b5cbb admin admin] PATH_INFO: `/v3/projects/0f64ebdf74a64748a83d40d740bd09fe/users/bcfc63654952469b925eac5358fb27de/roles/cc56c4ae13a7424ea771c15da4b153c1` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:29:32.991575 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-558b92dc-03d6-40f0-9a83-84e7c25b5cbb admin admin] RBAC: Authorizing `identity:create_grant(project_id=0f64ebdf74a64748a83d40d740bd09fe, user_id=bcfc63654952469b925eac5358fb27de, role_id=cc56c4ae13a7424ea771c15da4b153c1)` {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:29:32.994249 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-558b92dc-03d6-40f0-9a83-84e7c25b5cbb admin admin] RBAC: Authorization granted {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:29:33.008818 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 1225/2484] 10.222.0.27 () {60 vars in 1483 bytes} [Thu Oct 13 04:29:32 2022] PUT /identity/v3/projects/0f64ebdf74a64748a83d40d740bd09fe/users/bcfc63654952469b925eac5358fb27de/roles/cc56c4ae13a7424ea771c15da4b153c1 => generated 0 bytes in 50 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 13 04:29:33.021521 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53eefef0-161a-43d2-b848-942ce6493b8e None None] REQUEST_METHOD: `POST` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:29:33.021800 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53eefef0-161a-43d2-b848-942ce6493b8e None None] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:29:33.022068 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53eefef0-161a-43d2-b848-942ce6493b8e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:29:33.051323 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.auth.core [None req-53eefef0-161a-43d2-b848-942ce6493b8e None None] MFA Rules not processed for user `bcfc63654952469b925eac5358fb27de`. Rule list: `[]` (Enabled: `True`). {{(pid=81409) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 13 04:29:33.052540 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.fernet_utils [None req-53eefef0-161a-43d2-b848-942ce6493b8e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81409) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:29:33.076361 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG dbcounter [-] [81409] Writing DB stats keystone:SELECT=93,keystone:INSERT=4,keystone:DELETE=3 {{(pid=81409) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:29:33.119581 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 1260/2485] 10.222.0.27 () {58 vars in 958 bytes} [Thu Oct 13 04:29:33 2022] POST /identity/v3/auth/tokens => generated 3250 bytes in 99 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 13 04:29:33.130747 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-558b92dc-03d6-40f0-9a83-84e7c25b5cbb admin admin] Authenticating user token {{(pid=81408) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:29:33.132436 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.fernet_utils [None req-558b92dc-03d6-40f0-9a83-84e7c25b5cbb admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81408) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:29:33.201396 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-558b92dc-03d6-40f0-9a83-84e7c25b5cbb admin admin] Validating token access rules against request {{(pid=81408) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:29:33.205143 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0d16d5e8-1a5e-494b-9e5e-9d76c6673cbb service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6141b8a6c3d54e179d6879bb019f4d12', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dcec0431fb6849499d6aec35a234daa6', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81408) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:29:33.205744 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d16d5e8-1a5e-494b-9e5e-9d76c6673cbb service cinder] REQUEST_METHOD: `GET` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:29:33.206018 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d16d5e8-1a5e-494b-9e5e-9d76c6673cbb service cinder] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:29:33.206317 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d16d5e8-1a5e-494b-9e5e-9d76c6673cbb service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:29:33.214482 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0d16d5e8-1a5e-494b-9e5e-9d76c6673cbb service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:29:33.215927 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0d16d5e8-1a5e-494b-9e5e-9d76c6673cbb service cinder] RBAC: Authorization granted {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:29:33.226818 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 1226/2486] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Oct 13 04:29:33 2022] GET /identity/v3/auth/tokens => generated 3250 bytes in 96 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 13 04:29:33.249611 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-53eefef0-161a-43d2-b848-942ce6493b8e None None] Authenticating user token {{(pid=81409) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:29:33.311756 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-53eefef0-161a-43d2-b848-942ce6493b8e None None] Validating token access rules against request {{(pid=81409) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:29:33.316287 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-96356b0f-e11c-40b0-b8d5-7b166430c5d6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8ba823f313144da3b683bacef8fd62e0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81409) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:29:33.316907 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96356b0f-e11c-40b0-b8d5-7b166430c5d6 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:29:33.317182 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96356b0f-e11c-40b0-b8d5-7b166430c5d6 admin admin] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:29:33.317452 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96356b0f-e11c-40b0-b8d5-7b166430c5d6 admin admin] PATH_INFO: `/v3/users/bcfc63654952469b925eac5358fb27de` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:29:33.318605 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-96356b0f-e11c-40b0-b8d5-7b166430c5d6 admin admin] RBAC: Authorizing `identity:delete_user(user_id=bcfc63654952469b925eac5358fb27de)` {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:29:33.320513 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-96356b0f-e11c-40b0-b8d5-7b166430c5d6 admin admin] RBAC: Authorization granted {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:29:33.366554 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.notifications [None req-96356b0f-e11c-40b0-b8d5-7b166430c5d6 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'bcfc63654952469b925eac5358fb27de'} {{(pid=81409) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 13 04:29:33.368190 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.notifications [None req-96356b0f-e11c-40b0-b8d5-7b166430c5d6 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'bcfc63654952469b925eac5358fb27de'} {{(pid=81409) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 13 04:29:33.375373 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.notifications [None req-96356b0f-e11c-40b0-b8d5-7b166430c5d6 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'bcfc63654952469b925eac5358fb27de'} {{(pid=81409) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 13 04:29:33.376028 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.notifications [None req-96356b0f-e11c-40b0-b8d5-7b166430c5d6 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'bcfc63654952469b925eac5358fb27de'} {{(pid=81409) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 13 04:29:33.384099 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.notifications [None req-96356b0f-e11c-40b0-b8d5-7b166430c5d6 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'bcfc63654952469b925eac5358fb27de'} {{(pid=81409) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 13 04:29:33.401853 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 1261/2487] 10.222.0.27 () {58 vars in 1224 bytes} [Thu Oct 13 04:29:33 2022] DELETE /identity/v3/users/bcfc63654952469b925eac5358fb27de => generated 0 bytes in 153 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 13 04:29:33.406966 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3dbf747-fb45-4609-9803-d337c607e619 None None] REQUEST_METHOD: `POST` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:29:33.407251 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3dbf747-fb45-4609-9803-d337c607e619 None None] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:29:33.407522 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3dbf747-fb45-4609-9803-d337c607e619 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:29:33.426814 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.auth.core [None req-a3dbf747-fb45-4609-9803-d337c607e619 None None] MFA Rules not processed for user `0e2d979563d6421995451a0e86bf3022`. Rule list: `[]` (Enabled: `True`). {{(pid=81408) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 13 04:29:33.428019 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.fernet_utils [None req-a3dbf747-fb45-4609-9803-d337c607e619 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81408) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:29:33.485695 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 1227/2488] 10.222.0.27 () {58 vars in 958 bytes} [Thu Oct 13 04:29:33 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 80 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 13 04:29:33.495593 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-96356b0f-e11c-40b0-b8d5-7b166430c5d6 admin admin] Authenticating user token {{(pid=81409) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:29:33.496983 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.fernet_utils [None req-96356b0f-e11c-40b0-b8d5-7b166430c5d6 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81409) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:29:33.545890 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG dbcounter [-] [81409] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81409) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:29:33.558607 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-96356b0f-e11c-40b0-b8d5-7b166430c5d6 admin admin] Validating token access rules against request {{(pid=81409) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:29:33.564033 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-036507e7-8f43-4065-ae98-590b787b7520 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd8812c3974f84871bb2d761cf0016257', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dcec0431fb6849499d6aec35a234daa6', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81409) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:29:33.564586 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-036507e7-8f43-4065-ae98-590b787b7520 service neutron] REQUEST_METHOD: `GET` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:29:33.564868 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-036507e7-8f43-4065-ae98-590b787b7520 service neutron] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:29:33.565167 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-036507e7-8f43-4065-ae98-590b787b7520 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:29:33.575163 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-036507e7-8f43-4065-ae98-590b787b7520 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:29:33.576814 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-036507e7-8f43-4065-ae98-590b787b7520 service neutron] RBAC: Authorization granted {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:29:33.586646 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 1262/2489] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Oct 13 04:29:33 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 92 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 13 04:29:33.855065 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a3dbf747-fb45-4609-9803-d337c607e619 None None] Authenticating user token {{(pid=81408) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:29:33.856790 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.fernet_utils [None req-a3dbf747-fb45-4609-9803-d337c607e619 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81408) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:29:33.871683 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a3dbf747-fb45-4609-9803-d337c607e619 None None] Validating token access rules against request {{(pid=81408) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:29:33.877136 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7033bfac-e291-415d-9853-d9c72520e25c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8ba823f313144da3b683bacef8fd62e0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81408) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:29:33.877764 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7033bfac-e291-415d-9853-d9c72520e25c admin admin] REQUEST_METHOD: `DELETE` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:29:33.878034 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7033bfac-e291-415d-9853-d9c72520e25c admin admin] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:29:33.878330 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7033bfac-e291-415d-9853-d9c72520e25c admin admin] PATH_INFO: `/v3/projects/0f64ebdf74a64748a83d40d740bd09fe` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:29:33.879967 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7033bfac-e291-415d-9853-d9c72520e25c admin admin] RBAC: Authorizing `identity:delete_project(project_id=0f64ebdf74a64748a83d40d740bd09fe)` {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:29:33.882062 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7033bfac-e291-415d-9853-d9c72520e25c admin admin] RBAC: Authorization granted {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:29:33.921243 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG dbcounter [-] [81408] Writing DB stats keystone:SELECT=95,keystone:INSERT=2,keystone:DELETE=3 {{(pid=81408) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:29:33.931557 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.notifications [None req-7033bfac-e291-415d-9853-d9c72520e25c admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '0f64ebdf74a64748a83d40d740bd09fe', 'request_id': 'req-7033bfac-e291-415d-9853-d9c72520e25c'} {{(pid=81408) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 13 04:29:33.938768 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.notifications [None req-7033bfac-e291-415d-9853-d9c72520e25c admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '0f64ebdf74a64748a83d40d740bd09fe', 'request_id': 'req-7033bfac-e291-415d-9853-d9c72520e25c'} {{(pid=81408) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 13 04:29:33.945035 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.notifications [None req-7033bfac-e291-415d-9853-d9c72520e25c admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '0f64ebdf74a64748a83d40d740bd09fe', 'request_id': 'req-7033bfac-e291-415d-9853-d9c72520e25c'} {{(pid=81408) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 13 04:29:33.955264 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.notifications [None req-7033bfac-e291-415d-9853-d9c72520e25c admin admin] The token cache is being invalidate because project 0f64ebdf74a64748a83d40d740bd09fe was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81408) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 13 04:29:33.955546 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.notifications [None req-7033bfac-e291-415d-9853-d9c72520e25c admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81408) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 13 04:29:33.956163 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.notifications [None req-7033bfac-e291-415d-9853-d9c72520e25c admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81408) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 13 04:29:33.957614 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 1228/2490] 10.222.0.27 () {58 vars in 1233 bytes} [Thu Oct 13 04:29:33 2022] DELETE /identity/v3/projects/0f64ebdf74a64748a83d40d740bd09fe => generated 0 bytes in 103 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 13 04:29:33.962671 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be310da6-d108-4df9-a572-4923622cad34 None None] REQUEST_METHOD: `POST` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:29:33.962955 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be310da6-d108-4df9-a572-4923622cad34 None None] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:29:33.963256 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be310da6-d108-4df9-a572-4923622cad34 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:29:33.984218 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.auth.core [None req-be310da6-d108-4df9-a572-4923622cad34 None None] MFA Rules not processed for user `0e2d979563d6421995451a0e86bf3022`. Rule list: `[]` (Enabled: `True`). {{(pid=81409) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 13 04:29:33.985507 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.fernet_utils [None req-be310da6-d108-4df9-a572-4923622cad34 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81409) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:29:34.044367 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 1263/2491] 10.222.0.27 () {58 vars in 958 bytes} [Thu Oct 13 04:29:33 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 83 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 13 04:29:34.052724 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01a26129-2b4b-4441-8f47-6e64e32aa455 None None] REQUEST_METHOD: `POST` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:29:34.053007 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01a26129-2b4b-4441-8f47-6e64e32aa455 None None] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:29:34.053287 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01a26129-2b4b-4441-8f47-6e64e32aa455 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:29:34.073080 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.auth.core [None req-01a26129-2b4b-4441-8f47-6e64e32aa455 None None] MFA Rules not processed for user `0e2d979563d6421995451a0e86bf3022`. Rule list: `[]` (Enabled: `True`). {{(pid=81408) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 13 04:29:34.074320 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.fernet_utils [None req-01a26129-2b4b-4441-8f47-6e64e32aa455 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81408) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:29:34.083548 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 1229/2492] 10.222.0.27 () {58 vars in 958 bytes} [Thu Oct 13 04:29:34 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 32 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 13 04:29:34.088098 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be310da6-d108-4df9-a572-4923622cad34 None None] Authenticating user token {{(pid=81409) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:29:34.098874 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be310da6-d108-4df9-a572-4923622cad34 None None] Validating token access rules against request {{(pid=81409) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:29:34.104220 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5fa34782-ee86-45b6-95d1-b603166e49c4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8ba823f313144da3b683bacef8fd62e0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81409) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:29:34.104765 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5fa34782-ee86-45b6-95d1-b603166e49c4 admin admin] REQUEST_METHOD: `GET` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:29:34.105036 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5fa34782-ee86-45b6-95d1-b603166e49c4 admin admin] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:29:34.105304 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5fa34782-ee86-45b6-95d1-b603166e49c4 admin admin] PATH_INFO: `/v3/domains` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:29:34.106240 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5fa34782-ee86-45b6-95d1-b603166e49c4 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81409) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 13 04:29:34.106544 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5fa34782-ee86-45b6-95d1-b603166e49c4 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:29:34.109411 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5fa34782-ee86-45b6-95d1-b603166e49c4 admin admin] RBAC: Authorization granted {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:29:34.121196 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 1264/2493] 10.222.0.27 () {58 vars in 1153 bytes} [Thu Oct 13 04:29:34 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 33 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 13 04:29:34.125670 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-01a26129-2b4b-4441-8f47-6e64e32aa455 None None] Authenticating user token {{(pid=81408) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:29:34.135148 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-01a26129-2b4b-4441-8f47-6e64e32aa455 None None] Validating token access rules against request {{(pid=81408) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:29:34.142846 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9349fea7-174f-45c9-872d-56c0f89c02de admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8ba823f313144da3b683bacef8fd62e0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81408) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:29:34.143815 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9349fea7-174f-45c9-872d-56c0f89c02de admin admin] REQUEST_METHOD: `POST` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:29:34.144273 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9349fea7-174f-45c9-872d-56c0f89c02de admin admin] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:29:34.144743 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9349fea7-174f-45c9-872d-56c0f89c02de admin admin] PATH_INFO: `/v3/projects` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:29:34.145844 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9349fea7-174f-45c9-872d-56c0f89c02de admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:29:34.148325 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9349fea7-174f-45c9-872d-56c0f89c02de admin admin] RBAC: Authorization granted {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:29:34.189897 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 1230/2494] 10.222.0.27 () {60 vars in 1153 bytes} [Thu Oct 13 04:29:34 2022] POST /identity/v3/projects => generated 359 bytes in 61 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 13 04:29:34.191335 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5fa34782-ee86-45b6-95d1-b603166e49c4 admin admin] Authenticating user token {{(pid=81409) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:29:34.202875 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5fa34782-ee86-45b6-95d1-b603166e49c4 admin admin] Validating token access rules against request {{(pid=81409) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:29:34.269584 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f146ceb4-c1ac-46ba-a258-fa58c43b9734 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8ba823f313144da3b683bacef8fd62e0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81409) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:29:34.270194 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f146ceb4-c1ac-46ba-a258-fa58c43b9734 admin admin] REQUEST_METHOD: `POST` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:29:34.270496 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f146ceb4-c1ac-46ba-a258-fa58c43b9734 admin admin] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:29:34.270765 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f146ceb4-c1ac-46ba-a258-fa58c43b9734 admin admin] PATH_INFO: `/v3/users` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:29:34.271387 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f146ceb4-c1ac-46ba-a258-fa58c43b9734 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:29:34.273282 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f146ceb4-c1ac-46ba-a258-fa58c43b9734 admin admin] RBAC: Authorization granted {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:29:34.295086 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 1265/2495] 10.222.0.27 () {60 vars in 1144 bytes} [Thu Oct 13 04:29:34 2022] POST /identity/v3/users => generated 280 bytes in 104 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 13 04:29:34.299317 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9349fea7-174f-45c9-872d-56c0f89c02de admin admin] Authenticating user token {{(pid=81408) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:29:34.310832 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9349fea7-174f-45c9-872d-56c0f89c02de admin admin] Validating token access rules against request {{(pid=81408) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:29:34.315625 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-68eea2dc-0b73-44aa-876d-83d13366e6bc admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8ba823f313144da3b683bacef8fd62e0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81408) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:29:34.316270 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68eea2dc-0b73-44aa-876d-83d13366e6bc admin admin] REQUEST_METHOD: `GET` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:29:34.316596 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68eea2dc-0b73-44aa-876d-83d13366e6bc admin admin] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:29:34.316919 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68eea2dc-0b73-44aa-876d-83d13366e6bc admin admin] PATH_INFO: `/v3/roles` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:29:34.317669 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-68eea2dc-0b73-44aa-876d-83d13366e6bc admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:29:34.320047 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-68eea2dc-0b73-44aa-876d-83d13366e6bc admin admin] RBAC: Authorization granted {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:29:34.331958 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 1231/2496] 10.222.0.27 () {58 vars in 1122 bytes} [Thu Oct 13 04:29:34 2022] GET /identity/v3/roles => generated 1420 bytes in 33 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 13 04:29:34.336068 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f146ceb4-c1ac-46ba-a258-fa58c43b9734 admin admin] Authenticating user token {{(pid=81409) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:29:34.346855 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f146ceb4-c1ac-46ba-a258-fa58c43b9734 admin admin] Validating token access rules against request {{(pid=81409) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:29:34.351443 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c7350c83-428f-46ef-b41a-d0317f4a49b7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8ba823f313144da3b683bacef8fd62e0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81409) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:29:34.351978 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7350c83-428f-46ef-b41a-d0317f4a49b7 admin admin] REQUEST_METHOD: `GET` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:29:34.352248 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7350c83-428f-46ef-b41a-d0317f4a49b7 admin admin] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:29:34.352514 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7350c83-428f-46ef-b41a-d0317f4a49b7 admin admin] PATH_INFO: `/v3/roles` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:29:34.353143 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c7350c83-428f-46ef-b41a-d0317f4a49b7 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:29:34.354749 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c7350c83-428f-46ef-b41a-d0317f4a49b7 admin admin] RBAC: Authorization granted {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:29:34.368753 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 1266/2497] 10.222.0.27 () {58 vars in 1122 bytes} [Thu Oct 13 04:29:34 2022] GET /identity/v3/roles => generated 1420 bytes in 33 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 13 04:29:34.373172 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-68eea2dc-0b73-44aa-876d-83d13366e6bc admin admin] Authenticating user token {{(pid=81408) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:29:34.384798 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-68eea2dc-0b73-44aa-876d-83d13366e6bc admin admin] Validating token access rules against request {{(pid=81408) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:29:34.390054 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-15846292-f2e5-4f0d-94ec-be61cbe25047 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8ba823f313144da3b683bacef8fd62e0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81408) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:29:34.390728 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15846292-f2e5-4f0d-94ec-be61cbe25047 admin admin] REQUEST_METHOD: `PUT` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:29:34.391008 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15846292-f2e5-4f0d-94ec-be61cbe25047 admin admin] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:29:34.391280 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15846292-f2e5-4f0d-94ec-be61cbe25047 admin admin] PATH_INFO: `/v3/projects/efc320e9794a45308fa0e627d94cb9a8/users/cb8b80a3b7634a03b648e76384d70526/roles/cc56c4ae13a7424ea771c15da4b153c1` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:29:34.406357 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-15846292-f2e5-4f0d-94ec-be61cbe25047 admin admin] RBAC: Authorizing `identity:create_grant(project_id=efc320e9794a45308fa0e627d94cb9a8, user_id=cb8b80a3b7634a03b648e76384d70526, role_id=cc56c4ae13a7424ea771c15da4b153c1)` {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:29:34.409053 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-15846292-f2e5-4f0d-94ec-be61cbe25047 admin admin] RBAC: Authorization granted {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:29:34.425698 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 1232/2498] 10.222.0.27 () {60 vars in 1483 bytes} [Thu Oct 13 04:29:34 2022] PUT /identity/v3/projects/efc320e9794a45308fa0e627d94cb9a8/users/cb8b80a3b7634a03b648e76384d70526/roles/cc56c4ae13a7424ea771c15da4b153c1 => generated 0 bytes in 53 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 13 04:29:34.436282 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec508698-578b-4898-82a4-bb6d1647d818 None None] REQUEST_METHOD: `POST` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:29:34.436558 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec508698-578b-4898-82a4-bb6d1647d818 None None] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:29:34.436843 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec508698-578b-4898-82a4-bb6d1647d818 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:29:34.464558 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.auth.core [None req-ec508698-578b-4898-82a4-bb6d1647d818 None None] MFA Rules not processed for user `cb8b80a3b7634a03b648e76384d70526`. Rule list: `[]` (Enabled: `True`). {{(pid=81409) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 13 04:29:34.465935 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.fernet_utils [None req-ec508698-578b-4898-82a4-bb6d1647d818 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81409) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:29:34.511886 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG dbcounter [-] [81409] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81409) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:29:34.536861 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 1267/2499] 10.222.0.27 () {58 vars in 958 bytes} [Thu Oct 13 04:29:34 2022] POST /identity/v3/auth/tokens => generated 3236 bytes in 102 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 13 04:29:34.572748 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-15846292-f2e5-4f0d-94ec-be61cbe25047 admin admin] Authenticating user token {{(pid=81408) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:29:34.574475 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.fernet_utils [None req-15846292-f2e5-4f0d-94ec-be61cbe25047 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81408) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:29:34.631318 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-15846292-f2e5-4f0d-94ec-be61cbe25047 admin admin] Validating token access rules against request {{(pid=81408) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:29:34.635391 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f1a8497d-9251-4914-8dab-5666431d1b52 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6141b8a6c3d54e179d6879bb019f4d12', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dcec0431fb6849499d6aec35a234daa6', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81408) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:29:34.635995 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1a8497d-9251-4914-8dab-5666431d1b52 service cinder] REQUEST_METHOD: `GET` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:29:34.636283 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1a8497d-9251-4914-8dab-5666431d1b52 service cinder] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:29:34.636545 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1a8497d-9251-4914-8dab-5666431d1b52 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:29:34.645665 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f1a8497d-9251-4914-8dab-5666431d1b52 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:29:34.647127 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f1a8497d-9251-4914-8dab-5666431d1b52 service cinder] RBAC: Authorization granted {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:29:34.655869 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 1233/2500] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Oct 13 04:29:34 2022] GET /identity/v3/auth/tokens => generated 3236 bytes in 84 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 13 04:29:34.671430 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ec508698-578b-4898-82a4-bb6d1647d818 None None] Authenticating user token {{(pid=81409) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:29:34.734078 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ec508698-578b-4898-82a4-bb6d1647d818 None None] Validating token access rules against request {{(pid=81409) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:29:34.738589 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b15f8a2a-d19d-434c-ba4f-c96786176916 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8ba823f313144da3b683bacef8fd62e0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81409) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:29:34.739198 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b15f8a2a-d19d-434c-ba4f-c96786176916 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:29:34.739471 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b15f8a2a-d19d-434c-ba4f-c96786176916 admin admin] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:29:34.739736 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b15f8a2a-d19d-434c-ba4f-c96786176916 admin admin] PATH_INFO: `/v3/users/cb8b80a3b7634a03b648e76384d70526` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:29:34.740838 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b15f8a2a-d19d-434c-ba4f-c96786176916 admin admin] RBAC: Authorizing `identity:delete_user(user_id=cb8b80a3b7634a03b648e76384d70526)` {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:29:34.742770 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b15f8a2a-d19d-434c-ba4f-c96786176916 admin admin] RBAC: Authorization granted {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:29:34.788763 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.notifications [None req-b15f8a2a-d19d-434c-ba4f-c96786176916 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'cb8b80a3b7634a03b648e76384d70526'} {{(pid=81409) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 13 04:29:34.789407 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.notifications [None req-b15f8a2a-d19d-434c-ba4f-c96786176916 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'cb8b80a3b7634a03b648e76384d70526'} {{(pid=81409) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 13 04:29:34.796384 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.notifications [None req-b15f8a2a-d19d-434c-ba4f-c96786176916 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'cb8b80a3b7634a03b648e76384d70526'} {{(pid=81409) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 13 04:29:34.796997 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.notifications [None req-b15f8a2a-d19d-434c-ba4f-c96786176916 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'cb8b80a3b7634a03b648e76384d70526'} {{(pid=81409) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 13 04:29:34.803977 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.notifications [None req-b15f8a2a-d19d-434c-ba4f-c96786176916 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'cb8b80a3b7634a03b648e76384d70526'} {{(pid=81409) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 13 04:29:34.822595 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 1268/2501] 10.222.0.27 () {58 vars in 1224 bytes} [Thu Oct 13 04:29:34 2022] DELETE /identity/v3/users/cb8b80a3b7634a03b648e76384d70526 => generated 0 bytes in 152 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 13 04:29:34.827727 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8237dba-69f3-495e-b049-ba4bcbdb9f64 None None] REQUEST_METHOD: `POST` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:29:34.828011 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8237dba-69f3-495e-b049-ba4bcbdb9f64 None None] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:29:34.828283 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8237dba-69f3-495e-b049-ba4bcbdb9f64 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:29:34.847026 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.auth.core [None req-f8237dba-69f3-495e-b049-ba4bcbdb9f64 None None] MFA Rules not processed for user `0e2d979563d6421995451a0e86bf3022`. Rule list: `[]` (Enabled: `True`). {{(pid=81408) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 13 04:29:34.848227 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.fernet_utils [None req-f8237dba-69f3-495e-b049-ba4bcbdb9f64 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81408) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:29:34.906608 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 1234/2502] 10.222.0.27 () {58 vars in 958 bytes} [Thu Oct 13 04:29:34 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 80 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 13 04:29:34.916151 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b15f8a2a-d19d-434c-ba4f-c96786176916 admin admin] Authenticating user token {{(pid=81409) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:29:34.917527 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.fernet_utils [None req-b15f8a2a-d19d-434c-ba4f-c96786176916 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81409) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:29:34.974865 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b15f8a2a-d19d-434c-ba4f-c96786176916 admin admin] Validating token access rules against request {{(pid=81409) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:29:34.978630 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-664067bd-a937-43f3-af8a-6906651bca64 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd8812c3974f84871bb2d761cf0016257', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dcec0431fb6849499d6aec35a234daa6', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81409) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:29:34.979226 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-664067bd-a937-43f3-af8a-6906651bca64 service neutron] REQUEST_METHOD: `GET` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:29:34.979502 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-664067bd-a937-43f3-af8a-6906651bca64 service neutron] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:29:34.979772 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-664067bd-a937-43f3-af8a-6906651bca64 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:29:34.989680 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-664067bd-a937-43f3-af8a-6906651bca64 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:29:34.991300 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-664067bd-a937-43f3-af8a-6906651bca64 service neutron] RBAC: Authorization granted {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:29:35.001680 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 1269/2503] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Oct 13 04:29:34 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 86 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 13 04:29:35.267609 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f8237dba-69f3-495e-b049-ba4bcbdb9f64 None None] Authenticating user token {{(pid=81408) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:29:35.269143 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.fernet_utils [None req-f8237dba-69f3-495e-b049-ba4bcbdb9f64 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81408) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:29:35.277421 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG dbcounter [-] [81408] Writing DB stats keystone:SELECT=93,keystone:DELETE=4,keystone:INSERT=3 {{(pid=81408) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:29:35.282169 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f8237dba-69f3-495e-b049-ba4bcbdb9f64 None None] Validating token access rules against request {{(pid=81408) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:29:35.290942 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-48b5fc9e-240d-4907-96f0-1e4bd2b485e1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8ba823f313144da3b683bacef8fd62e0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81408) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:29:35.291776 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48b5fc9e-240d-4907-96f0-1e4bd2b485e1 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:29:35.292234 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48b5fc9e-240d-4907-96f0-1e4bd2b485e1 admin admin] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:29:35.292683 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48b5fc9e-240d-4907-96f0-1e4bd2b485e1 admin admin] PATH_INFO: `/v3/projects/efc320e9794a45308fa0e627d94cb9a8` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:29:35.293912 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-48b5fc9e-240d-4907-96f0-1e4bd2b485e1 admin admin] RBAC: Authorizing `identity:delete_project(project_id=efc320e9794a45308fa0e627d94cb9a8)` {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:29:35.296596 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-48b5fc9e-240d-4907-96f0-1e4bd2b485e1 admin admin] RBAC: Authorization granted {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:29:35.350782 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.notifications [None req-48b5fc9e-240d-4907-96f0-1e4bd2b485e1 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'efc320e9794a45308fa0e627d94cb9a8', 'request_id': 'req-48b5fc9e-240d-4907-96f0-1e4bd2b485e1'} {{(pid=81408) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 13 04:29:35.359704 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.notifications [None req-48b5fc9e-240d-4907-96f0-1e4bd2b485e1 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'efc320e9794a45308fa0e627d94cb9a8', 'request_id': 'req-48b5fc9e-240d-4907-96f0-1e4bd2b485e1'} {{(pid=81408) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 13 04:29:35.366198 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.notifications [None req-48b5fc9e-240d-4907-96f0-1e4bd2b485e1 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'efc320e9794a45308fa0e627d94cb9a8', 'request_id': 'req-48b5fc9e-240d-4907-96f0-1e4bd2b485e1'} {{(pid=81408) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 13 04:29:35.378235 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.notifications [None req-48b5fc9e-240d-4907-96f0-1e4bd2b485e1 admin admin] The token cache is being invalidate because project efc320e9794a45308fa0e627d94cb9a8 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81408) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 13 04:29:35.378545 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.notifications [None req-48b5fc9e-240d-4907-96f0-1e4bd2b485e1 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81408) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 13 04:29:35.379207 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.notifications [None req-48b5fc9e-240d-4907-96f0-1e4bd2b485e1 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81408) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 13 04:29:35.380777 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 1235/2504] 10.222.0.27 () {58 vars in 1233 bytes} [Thu Oct 13 04:29:35 2022] DELETE /identity/v3/projects/efc320e9794a45308fa0e627d94cb9a8 => generated 0 bytes in 114 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 13 04:29:35.386036 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad4663ca-0ace-4eba-9fe1-ad449ab998f7 None None] REQUEST_METHOD: `POST` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:29:35.386389 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad4663ca-0ace-4eba-9fe1-ad449ab998f7 None None] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:29:35.386632 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad4663ca-0ace-4eba-9fe1-ad449ab998f7 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:29:35.396704 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG dbcounter [-] [81409] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81409) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:29:35.414433 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.auth.core [None req-ad4663ca-0ace-4eba-9fe1-ad449ab998f7 None None] MFA Rules not processed for user `0e2d979563d6421995451a0e86bf3022`. Rule list: `[]` (Enabled: `True`). {{(pid=81409) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 13 04:29:35.415720 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.fernet_utils [None req-ad4663ca-0ace-4eba-9fe1-ad449ab998f7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81409) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:29:35.476787 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 1270/2505] 10.222.0.27 () {58 vars in 958 bytes} [Thu Oct 13 04:29:35 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 92 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 13 04:29:35.485209 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-372df106-7ab7-4eae-a3b5-3fe9c7edb191 None None] REQUEST_METHOD: `POST` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:29:35.485573 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-372df106-7ab7-4eae-a3b5-3fe9c7edb191 None None] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:29:35.485905 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-372df106-7ab7-4eae-a3b5-3fe9c7edb191 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:29:35.506128 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.auth.core [None req-372df106-7ab7-4eae-a3b5-3fe9c7edb191 None None] MFA Rules not processed for user `0e2d979563d6421995451a0e86bf3022`. Rule list: `[]` (Enabled: `True`). {{(pid=81408) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 13 04:29:35.507377 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.fernet_utils [None req-372df106-7ab7-4eae-a3b5-3fe9c7edb191 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81408) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:29:35.515789 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 1236/2506] 10.222.0.27 () {58 vars in 958 bytes} [Thu Oct 13 04:29:35 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 31 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 13 04:29:35.520865 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ad4663ca-0ace-4eba-9fe1-ad449ab998f7 None None] Authenticating user token {{(pid=81409) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:29:35.532682 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ad4663ca-0ace-4eba-9fe1-ad449ab998f7 None None] Validating token access rules against request {{(pid=81409) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:29:35.537274 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3aa0ba94-f01a-4e25-a0ca-e7b4e7ca362e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8ba823f313144da3b683bacef8fd62e0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81409) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:29:35.537875 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3aa0ba94-f01a-4e25-a0ca-e7b4e7ca362e admin admin] REQUEST_METHOD: `GET` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:29:35.538162 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3aa0ba94-f01a-4e25-a0ca-e7b4e7ca362e admin admin] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:29:35.538446 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3aa0ba94-f01a-4e25-a0ca-e7b4e7ca362e admin admin] PATH_INFO: `/v3/domains` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:29:35.539083 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3aa0ba94-f01a-4e25-a0ca-e7b4e7ca362e admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81409) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 13 04:29:35.540025 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3aa0ba94-f01a-4e25-a0ca-e7b4e7ca362e admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:29:35.541726 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3aa0ba94-f01a-4e25-a0ca-e7b4e7ca362e admin admin] RBAC: Authorization granted {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:29:35.555791 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 1271/2507] 10.222.0.27 () {58 vars in 1153 bytes} [Thu Oct 13 04:29:35 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 36 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 13 04:29:35.561274 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-372df106-7ab7-4eae-a3b5-3fe9c7edb191 None None] Authenticating user token {{(pid=81408) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:29:35.570857 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-372df106-7ab7-4eae-a3b5-3fe9c7edb191 None None] Validating token access rules against request {{(pid=81408) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:29:35.575766 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a50ce645-63c3-4a05-a419-b5c479a62b88 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8ba823f313144da3b683bacef8fd62e0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81408) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:29:35.576547 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a50ce645-63c3-4a05-a419-b5c479a62b88 admin admin] REQUEST_METHOD: `POST` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:29:35.576856 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a50ce645-63c3-4a05-a419-b5c479a62b88 admin admin] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:29:35.577114 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a50ce645-63c3-4a05-a419-b5c479a62b88 admin admin] PATH_INFO: `/v3/projects` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:29:35.577792 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a50ce645-63c3-4a05-a419-b5c479a62b88 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:29:35.580206 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a50ce645-63c3-4a05-a419-b5c479a62b88 admin admin] RBAC: Authorization granted {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:29:35.613374 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 1237/2508] 10.222.0.27 () {60 vars in 1153 bytes} [Thu Oct 13 04:29:35 2022] POST /identity/v3/projects => generated 373 bytes in 50 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 13 04:29:35.614537 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3aa0ba94-f01a-4e25-a0ca-e7b4e7ca362e admin admin] Authenticating user token {{(pid=81409) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:29:35.624907 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3aa0ba94-f01a-4e25-a0ca-e7b4e7ca362e admin admin] Validating token access rules against request {{(pid=81409) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:29:35.708779 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7e230240-a267-455a-bbf9-1dc91d1e7aef admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8ba823f313144da3b683bacef8fd62e0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81409) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:29:35.709521 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e230240-a267-455a-bbf9-1dc91d1e7aef admin admin] REQUEST_METHOD: `POST` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:29:35.709914 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e230240-a267-455a-bbf9-1dc91d1e7aef admin admin] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:29:35.710275 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e230240-a267-455a-bbf9-1dc91d1e7aef admin admin] PATH_INFO: `/v3/users` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:29:35.710981 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7e230240-a267-455a-bbf9-1dc91d1e7aef admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:29:35.712988 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7e230240-a267-455a-bbf9-1dc91d1e7aef admin admin] RBAC: Authorization granted {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:29:35.735463 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 1272/2509] 10.222.0.27 () {60 vars in 1144 bytes} [Thu Oct 13 04:29:35 2022] POST /identity/v3/users => generated 287 bytes in 121 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 13 04:29:35.739782 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a50ce645-63c3-4a05-a419-b5c479a62b88 admin admin] Authenticating user token {{(pid=81408) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:29:35.750109 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a50ce645-63c3-4a05-a419-b5c479a62b88 admin admin] Validating token access rules against request {{(pid=81408) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:29:35.754642 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b75581a9-30f9-4ddb-83c5-b69fc4a205db admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8ba823f313144da3b683bacef8fd62e0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81408) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:29:35.755225 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b75581a9-30f9-4ddb-83c5-b69fc4a205db admin admin] REQUEST_METHOD: `GET` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:29:35.755499 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b75581a9-30f9-4ddb-83c5-b69fc4a205db admin admin] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:29:35.755765 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b75581a9-30f9-4ddb-83c5-b69fc4a205db admin admin] PATH_INFO: `/v3/roles` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:29:35.756395 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b75581a9-30f9-4ddb-83c5-b69fc4a205db admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:29:35.757977 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b75581a9-30f9-4ddb-83c5-b69fc4a205db admin admin] RBAC: Authorization granted {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:29:35.768544 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 1238/2510] 10.222.0.27 () {58 vars in 1122 bytes} [Thu Oct 13 04:29:35 2022] GET /identity/v3/roles => generated 1420 bytes in 29 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 13 04:29:35.772443 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7e230240-a267-455a-bbf9-1dc91d1e7aef admin admin] Authenticating user token {{(pid=81409) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:29:35.783048 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7e230240-a267-455a-bbf9-1dc91d1e7aef admin admin] Validating token access rules against request {{(pid=81409) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:29:35.787698 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f0cf5849-e7d2-4fee-bd4c-c753f5c558da admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8ba823f313144da3b683bacef8fd62e0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81409) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:29:35.788291 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0cf5849-e7d2-4fee-bd4c-c753f5c558da admin admin] REQUEST_METHOD: `GET` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:29:35.788571 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0cf5849-e7d2-4fee-bd4c-c753f5c558da admin admin] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:29:35.788842 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0cf5849-e7d2-4fee-bd4c-c753f5c558da admin admin] PATH_INFO: `/v3/roles` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:29:35.789481 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f0cf5849-e7d2-4fee-bd4c-c753f5c558da admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:29:35.791129 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f0cf5849-e7d2-4fee-bd4c-c753f5c558da admin admin] RBAC: Authorization granted {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:29:35.801992 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 1273/2511] 10.222.0.27 () {58 vars in 1122 bytes} [Thu Oct 13 04:29:35 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 13 04:29:35.806115 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b75581a9-30f9-4ddb-83c5-b69fc4a205db admin admin] Authenticating user token {{(pid=81408) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:29:35.816014 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b75581a9-30f9-4ddb-83c5-b69fc4a205db admin admin] Validating token access rules against request {{(pid=81408) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:29:35.820532 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3b8e536e-720f-43c2-84b0-f34cf3c662a5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8ba823f313144da3b683bacef8fd62e0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81408) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:29:35.821162 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b8e536e-720f-43c2-84b0-f34cf3c662a5 admin admin] REQUEST_METHOD: `PUT` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:29:35.821436 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b8e536e-720f-43c2-84b0-f34cf3c662a5 admin admin] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:29:35.821703 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b8e536e-720f-43c2-84b0-f34cf3c662a5 admin admin] PATH_INFO: `/v3/projects/6f7a2834d7ac4860a64148e75bbd52e2/users/2a7fd87975f5432e95f5cb77930bcf06/roles/cc56c4ae13a7424ea771c15da4b153c1` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:29:35.836255 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3b8e536e-720f-43c2-84b0-f34cf3c662a5 admin admin] RBAC: Authorizing `identity:create_grant(project_id=6f7a2834d7ac4860a64148e75bbd52e2, user_id=2a7fd87975f5432e95f5cb77930bcf06, role_id=cc56c4ae13a7424ea771c15da4b153c1)` {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:29:35.838923 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3b8e536e-720f-43c2-84b0-f34cf3c662a5 admin admin] RBAC: Authorization granted {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:29:35.852120 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 1239/2512] 10.222.0.27 () {60 vars in 1483 bytes} [Thu Oct 13 04:29:35 2022] PUT /identity/v3/projects/6f7a2834d7ac4860a64148e75bbd52e2/users/2a7fd87975f5432e95f5cb77930bcf06/roles/cc56c4ae13a7424ea771c15da4b153c1 => generated 0 bytes in 46 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 13 04:29:35.862692 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d52c60c-f34a-4b29-ba8c-faf50b2d1e0c None None] REQUEST_METHOD: `POST` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:29:35.863011 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d52c60c-f34a-4b29-ba8c-faf50b2d1e0c None None] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:29:35.863277 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d52c60c-f34a-4b29-ba8c-faf50b2d1e0c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:29:35.892219 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.auth.core [None req-4d52c60c-f34a-4b29-ba8c-faf50b2d1e0c None None] MFA Rules not processed for user `2a7fd87975f5432e95f5cb77930bcf06`. Rule list: `[]` (Enabled: `True`). {{(pid=81409) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 13 04:29:35.893423 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.fernet_utils [None req-4d52c60c-f34a-4b29-ba8c-faf50b2d1e0c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81409) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:29:35.954050 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG dbcounter [-] [81409] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81409) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:29:35.958334 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 1274/2513] 10.222.0.27 () {58 vars in 958 bytes} [Thu Oct 13 04:29:35 2022] POST /identity/v3/auth/tokens => generated 3250 bytes in 96 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 13 04:29:35.970555 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3b8e536e-720f-43c2-84b0-f34cf3c662a5 admin admin] Authenticating user token {{(pid=81408) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:29:35.972282 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.fernet_utils [None req-3b8e536e-720f-43c2-84b0-f34cf3c662a5 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81408) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:29:36.028834 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3b8e536e-720f-43c2-84b0-f34cf3c662a5 admin admin] Validating token access rules against request {{(pid=81408) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:29:36.032525 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-66cd210f-46f8-415e-bde9-1bfe93b5b1ad service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6141b8a6c3d54e179d6879bb019f4d12', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dcec0431fb6849499d6aec35a234daa6', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81408) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:29:36.033122 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66cd210f-46f8-415e-bde9-1bfe93b5b1ad service cinder] REQUEST_METHOD: `GET` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:29:36.033397 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66cd210f-46f8-415e-bde9-1bfe93b5b1ad service cinder] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:29:36.033667 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66cd210f-46f8-415e-bde9-1bfe93b5b1ad service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:29:36.042108 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-66cd210f-46f8-415e-bde9-1bfe93b5b1ad service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:29:36.043610 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-66cd210f-46f8-415e-bde9-1bfe93b5b1ad service cinder] RBAC: Authorization granted {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:29:36.052662 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 1240/2514] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Oct 13 04:29:35 2022] GET /identity/v3/auth/tokens => generated 3250 bytes in 83 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 13 04:29:41.175810 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4d52c60c-f34a-4b29-ba8c-faf50b2d1e0c None None] Authenticating user token {{(pid=81409) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:29:41.241593 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4d52c60c-f34a-4b29-ba8c-faf50b2d1e0c None None] Validating token access rules against request {{(pid=81409) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:29:41.246060 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-61c235ea-ecb7-4fa3-9019-386d08af42c9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8ba823f313144da3b683bacef8fd62e0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81409) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:29:41.246713 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61c235ea-ecb7-4fa3-9019-386d08af42c9 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:29:41.246990 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61c235ea-ecb7-4fa3-9019-386d08af42c9 admin admin] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:29:41.247260 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61c235ea-ecb7-4fa3-9019-386d08af42c9 admin admin] PATH_INFO: `/v3/users/2a7fd87975f5432e95f5cb77930bcf06` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:29:41.248369 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-61c235ea-ecb7-4fa3-9019-386d08af42c9 admin admin] RBAC: Authorizing `identity:delete_user(user_id=2a7fd87975f5432e95f5cb77930bcf06)` {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:29:41.250314 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-61c235ea-ecb7-4fa3-9019-386d08af42c9 admin admin] RBAC: Authorization granted {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:29:41.296036 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.notifications [None req-61c235ea-ecb7-4fa3-9019-386d08af42c9 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '2a7fd87975f5432e95f5cb77930bcf06'} {{(pid=81409) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 13 04:29:41.296605 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.notifications [None req-61c235ea-ecb7-4fa3-9019-386d08af42c9 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '2a7fd87975f5432e95f5cb77930bcf06'} {{(pid=81409) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 13 04:29:41.304161 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.notifications [None req-61c235ea-ecb7-4fa3-9019-386d08af42c9 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '2a7fd87975f5432e95f5cb77930bcf06'} {{(pid=81409) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 13 04:29:41.304788 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.notifications [None req-61c235ea-ecb7-4fa3-9019-386d08af42c9 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '2a7fd87975f5432e95f5cb77930bcf06'} {{(pid=81409) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 13 04:29:41.311238 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.notifications [None req-61c235ea-ecb7-4fa3-9019-386d08af42c9 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '2a7fd87975f5432e95f5cb77930bcf06'} {{(pid=81409) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 13 04:29:41.328206 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 1275/2515] 10.222.0.27 () {58 vars in 1224 bytes} [Thu Oct 13 04:29:41 2022] DELETE /identity/v3/users/2a7fd87975f5432e95f5cb77930bcf06 => generated 0 bytes in 153 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 13 04:29:41.332777 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-803c35fb-65d8-4b11-a5f7-b44a0ddc423f None None] REQUEST_METHOD: `POST` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:29:41.333268 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-803c35fb-65d8-4b11-a5f7-b44a0ddc423f None None] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:29:41.333368 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-803c35fb-65d8-4b11-a5f7-b44a0ddc423f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:29:41.352603 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.auth.core [None req-803c35fb-65d8-4b11-a5f7-b44a0ddc423f None None] MFA Rules not processed for user `0e2d979563d6421995451a0e86bf3022`. Rule list: `[]` (Enabled: `True`). {{(pid=81408) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 13 04:29:41.353806 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.fernet_utils [None req-803c35fb-65d8-4b11-a5f7-b44a0ddc423f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81408) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:29:41.373934 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG dbcounter [-] [81408] Writing DB stats keystone:SELECT=90,keystone:DELETE=7,keystone:INSERT=3 {{(pid=81408) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:29:41.415561 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 1241/2516] 10.222.0.27 () {58 vars in 958 bytes} [Thu Oct 13 04:29:41 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 83 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 13 04:29:41.425618 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-61c235ea-ecb7-4fa3-9019-386d08af42c9 admin admin] Authenticating user token {{(pid=81409) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:29:41.427076 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.fernet_utils [None req-61c235ea-ecb7-4fa3-9019-386d08af42c9 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81409) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:29:41.484636 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-61c235ea-ecb7-4fa3-9019-386d08af42c9 admin admin] Validating token access rules against request {{(pid=81409) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:29:41.488948 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dde97e23-7b05-4fb0-9f6f-d6bd343cb2f2 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd8812c3974f84871bb2d761cf0016257', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dcec0431fb6849499d6aec35a234daa6', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81409) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:29:41.489556 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dde97e23-7b05-4fb0-9f6f-d6bd343cb2f2 service neutron] REQUEST_METHOD: `GET` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:29:41.489835 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dde97e23-7b05-4fb0-9f6f-d6bd343cb2f2 service neutron] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:29:41.490105 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dde97e23-7b05-4fb0-9f6f-d6bd343cb2f2 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:29:41.499763 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dde97e23-7b05-4fb0-9f6f-d6bd343cb2f2 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:29:41.501369 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dde97e23-7b05-4fb0-9f6f-d6bd343cb2f2 service neutron] RBAC: Authorization granted {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:29:41.511363 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 1276/2517] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Oct 13 04:29:41 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 86 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 13 04:29:41.780525 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-803c35fb-65d8-4b11-a5f7-b44a0ddc423f None None] Authenticating user token {{(pid=81408) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:29:41.782051 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.fernet_utils [None req-803c35fb-65d8-4b11-a5f7-b44a0ddc423f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81408) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:29:41.801544 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-803c35fb-65d8-4b11-a5f7-b44a0ddc423f None None] Validating token access rules against request {{(pid=81408) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:29:41.807475 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8c831df9-1ebb-43ee-aaed-d5d352a99c74 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8ba823f313144da3b683bacef8fd62e0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81408) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:29:41.814084 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c831df9-1ebb-43ee-aaed-d5d352a99c74 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:29:41.814660 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c831df9-1ebb-43ee-aaed-d5d352a99c74 admin admin] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:29:41.814979 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c831df9-1ebb-43ee-aaed-d5d352a99c74 admin admin] PATH_INFO: `/v3/projects/6f7a2834d7ac4860a64148e75bbd52e2` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:29:41.817882 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8c831df9-1ebb-43ee-aaed-d5d352a99c74 admin admin] RBAC: Authorizing `identity:delete_project(project_id=6f7a2834d7ac4860a64148e75bbd52e2)` {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:29:41.819852 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8c831df9-1ebb-43ee-aaed-d5d352a99c74 admin admin] RBAC: Authorization granted {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:29:41.866430 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.notifications [None req-8c831df9-1ebb-43ee-aaed-d5d352a99c74 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '6f7a2834d7ac4860a64148e75bbd52e2', 'request_id': 'req-8c831df9-1ebb-43ee-aaed-d5d352a99c74'} {{(pid=81408) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 13 04:29:41.873277 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.notifications [None req-8c831df9-1ebb-43ee-aaed-d5d352a99c74 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '6f7a2834d7ac4860a64148e75bbd52e2', 'request_id': 'req-8c831df9-1ebb-43ee-aaed-d5d352a99c74'} {{(pid=81408) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 13 04:29:41.879760 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.notifications [None req-8c831df9-1ebb-43ee-aaed-d5d352a99c74 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '6f7a2834d7ac4860a64148e75bbd52e2', 'request_id': 'req-8c831df9-1ebb-43ee-aaed-d5d352a99c74'} {{(pid=81408) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 13 04:29:41.890016 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.notifications [None req-8c831df9-1ebb-43ee-aaed-d5d352a99c74 admin admin] The token cache is being invalidate because project 6f7a2834d7ac4860a64148e75bbd52e2 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81408) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 13 04:29:41.890336 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.notifications [None req-8c831df9-1ebb-43ee-aaed-d5d352a99c74 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81408) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 13 04:29:41.890970 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.notifications [None req-8c831df9-1ebb-43ee-aaed-d5d352a99c74 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81408) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 13 04:29:41.893509 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 1242/2518] 10.222.0.27 () {58 vars in 1233 bytes} [Thu Oct 13 04:29:41 2022] DELETE /identity/v3/projects/6f7a2834d7ac4860a64148e75bbd52e2 => generated 0 bytes in 114 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 13 04:29:41.898737 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71b26681-8646-47bc-8b55-b63762d62eb8 None None] REQUEST_METHOD: `POST` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:29:41.899021 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71b26681-8646-47bc-8b55-b63762d62eb8 None None] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:29:41.899293 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71b26681-8646-47bc-8b55-b63762d62eb8 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:29:41.919269 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.auth.core [None req-71b26681-8646-47bc-8b55-b63762d62eb8 None None] MFA Rules not processed for user `0e2d979563d6421995451a0e86bf3022`. Rule list: `[]` (Enabled: `True`). {{(pid=81409) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 13 04:29:41.920473 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.fernet_utils [None req-71b26681-8646-47bc-8b55-b63762d62eb8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81409) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:29:41.937800 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG dbcounter [-] [81409] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81409) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:29:41.981694 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 1277/2519] 10.222.0.27 () {58 vars in 958 bytes} [Thu Oct 13 04:29:41 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 84 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 13 04:29:41.991293 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e5a38ba-264e-4379-bd19-024237ba0968 None None] REQUEST_METHOD: `POST` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:29:41.991603 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e5a38ba-264e-4379-bd19-024237ba0968 None None] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:29:41.991876 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e5a38ba-264e-4379-bd19-024237ba0968 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:29:42.013622 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.auth.core [None req-7e5a38ba-264e-4379-bd19-024237ba0968 None None] MFA Rules not processed for user `0e2d979563d6421995451a0e86bf3022`. Rule list: `[]` (Enabled: `True`). {{(pid=81408) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 13 04:29:42.014848 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.fernet_utils [None req-7e5a38ba-264e-4379-bd19-024237ba0968 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81408) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:29:42.023385 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 1243/2520] 10.222.0.27 () {58 vars in 958 bytes} [Thu Oct 13 04:29:41 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 33 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 13 04:29:42.027425 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-71b26681-8646-47bc-8b55-b63762d62eb8 None None] Authenticating user token {{(pid=81409) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:29:42.037907 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-71b26681-8646-47bc-8b55-b63762d62eb8 None None] Validating token access rules against request {{(pid=81409) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:29:42.042425 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1c012a45-e00d-4661-8a5f-005c12137a0f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8ba823f313144da3b683bacef8fd62e0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81409) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:29:42.043015 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c012a45-e00d-4661-8a5f-005c12137a0f admin admin] REQUEST_METHOD: `GET` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:29:42.043286 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c012a45-e00d-4661-8a5f-005c12137a0f admin admin] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:29:42.043552 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c012a45-e00d-4661-8a5f-005c12137a0f admin admin] PATH_INFO: `/v3/domains` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:29:42.044188 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1c012a45-e00d-4661-8a5f-005c12137a0f admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81409) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 13 04:29:42.044483 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1c012a45-e00d-4661-8a5f-005c12137a0f admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:29:42.046047 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1c012a45-e00d-4661-8a5f-005c12137a0f admin admin] RBAC: Authorization granted {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:29:42.058248 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 1278/2521] 10.222.0.27 () {58 vars in 1153 bytes} [Thu Oct 13 04:29:42 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 13 04:29:42.062430 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7e5a38ba-264e-4379-bd19-024237ba0968 None None] Authenticating user token {{(pid=81408) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:29:42.071729 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7e5a38ba-264e-4379-bd19-024237ba0968 None None] Validating token access rules against request {{(pid=81408) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:29:42.076366 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-55b60dc0-9f1c-4d70-8e72-2e36f54b5649 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8ba823f313144da3b683bacef8fd62e0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81408) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:29:42.076954 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55b60dc0-9f1c-4d70-8e72-2e36f54b5649 admin admin] REQUEST_METHOD: `POST` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:29:42.077232 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55b60dc0-9f1c-4d70-8e72-2e36f54b5649 admin admin] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:29:42.077499 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55b60dc0-9f1c-4d70-8e72-2e36f54b5649 admin admin] PATH_INFO: `/v3/projects` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:29:42.078126 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-55b60dc0-9f1c-4d70-8e72-2e36f54b5649 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:29:42.080350 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-55b60dc0-9f1c-4d70-8e72-2e36f54b5649 admin admin] RBAC: Authorization granted {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:29:42.112685 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 1244/2522] 10.222.0.27 () {60 vars in 1153 bytes} [Thu Oct 13 04:29:42 2022] POST /identity/v3/projects => generated 377 bytes in 47 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 13 04:29:42.113852 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1c012a45-e00d-4661-8a5f-005c12137a0f admin admin] Authenticating user token {{(pid=81409) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:29:42.123896 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1c012a45-e00d-4661-8a5f-005c12137a0f admin admin] Validating token access rules against request {{(pid=81409) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:29:42.181266 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-82d4da69-c37f-4522-9715-8b9b45987842 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8ba823f313144da3b683bacef8fd62e0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81409) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:29:42.181918 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82d4da69-c37f-4522-9715-8b9b45987842 admin admin] REQUEST_METHOD: `POST` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:29:42.182123 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82d4da69-c37f-4522-9715-8b9b45987842 admin admin] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:29:42.182418 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82d4da69-c37f-4522-9715-8b9b45987842 admin admin] PATH_INFO: `/v3/users` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:29:42.183034 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-82d4da69-c37f-4522-9715-8b9b45987842 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:29:42.184926 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-82d4da69-c37f-4522-9715-8b9b45987842 admin admin] RBAC: Authorization granted {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:29:42.207402 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 1279/2523] 10.222.0.27 () {60 vars in 1144 bytes} [Thu Oct 13 04:29:42 2022] POST /identity/v3/users => generated 289 bytes in 94 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 13 04:29:42.211483 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-55b60dc0-9f1c-4d70-8e72-2e36f54b5649 admin admin] Authenticating user token {{(pid=81408) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:29:42.222338 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-55b60dc0-9f1c-4d70-8e72-2e36f54b5649 admin admin] Validating token access rules against request {{(pid=81408) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:29:42.227308 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b05323de-ac1a-471f-a98f-ef88fb4f6b36 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8ba823f313144da3b683bacef8fd62e0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81408) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:29:42.228114 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b05323de-ac1a-471f-a98f-ef88fb4f6b36 admin admin] REQUEST_METHOD: `GET` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:29:42.228462 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b05323de-ac1a-471f-a98f-ef88fb4f6b36 admin admin] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:29:42.228797 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b05323de-ac1a-471f-a98f-ef88fb4f6b36 admin admin] PATH_INFO: `/v3/roles` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:29:42.229499 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b05323de-ac1a-471f-a98f-ef88fb4f6b36 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:29:42.231521 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b05323de-ac1a-471f-a98f-ef88fb4f6b36 admin admin] RBAC: Authorization granted {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:29:42.245537 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 1245/2524] 10.222.0.27 () {58 vars in 1122 bytes} [Thu Oct 13 04:29:42 2022] GET /identity/v3/roles => generated 1420 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 13 04:29:42.246699 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-82d4da69-c37f-4522-9715-8b9b45987842 admin admin] Authenticating user token {{(pid=81409) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:29:42.257081 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-82d4da69-c37f-4522-9715-8b9b45987842 admin admin] Validating token access rules against request {{(pid=81409) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:29:42.261735 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9cdc0356-8c93-48c5-b355-63b218ea83a7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8ba823f313144da3b683bacef8fd62e0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81409) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:29:42.262359 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9cdc0356-8c93-48c5-b355-63b218ea83a7 admin admin] REQUEST_METHOD: `GET` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:29:42.262635 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9cdc0356-8c93-48c5-b355-63b218ea83a7 admin admin] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:29:42.262902 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9cdc0356-8c93-48c5-b355-63b218ea83a7 admin admin] PATH_INFO: `/v3/roles` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:29:42.263531 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9cdc0356-8c93-48c5-b355-63b218ea83a7 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:29:42.265201 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9cdc0356-8c93-48c5-b355-63b218ea83a7 admin admin] RBAC: Authorization granted {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:29:42.277131 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 1280/2525] 10.222.0.27 () {58 vars in 1122 bytes} [Thu Oct 13 04:29:42 2022] GET /identity/v3/roles => generated 1420 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 13 04:29:42.281441 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b05323de-ac1a-471f-a98f-ef88fb4f6b36 admin admin] Authenticating user token {{(pid=81408) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:29:42.292128 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b05323de-ac1a-471f-a98f-ef88fb4f6b36 admin admin] Validating token access rules against request {{(pid=81408) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:29:42.297297 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0196dd1e-1436-478b-b155-849ae3856837 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8ba823f313144da3b683bacef8fd62e0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81408) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:29:42.298652 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0196dd1e-1436-478b-b155-849ae3856837 admin admin] REQUEST_METHOD: `PUT` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:29:42.299148 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0196dd1e-1436-478b-b155-849ae3856837 admin admin] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:29:42.299427 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0196dd1e-1436-478b-b155-849ae3856837 admin admin] PATH_INFO: `/v3/projects/b70ff895a0594ede86db4211a2d0238d/users/41c9d68bdd49472abf7cf70f4085aa0c/roles/cc56c4ae13a7424ea771c15da4b153c1` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:29:42.315284 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0196dd1e-1436-478b-b155-849ae3856837 admin admin] RBAC: Authorizing `identity:create_grant(project_id=b70ff895a0594ede86db4211a2d0238d, user_id=41c9d68bdd49472abf7cf70f4085aa0c, role_id=cc56c4ae13a7424ea771c15da4b153c1)` {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:29:42.318117 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0196dd1e-1436-478b-b155-849ae3856837 admin admin] RBAC: Authorization granted {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:29:42.333235 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 1246/2526] 10.222.0.27 () {60 vars in 1483 bytes} [Thu Oct 13 04:29:42 2022] PUT /identity/v3/projects/b70ff895a0594ede86db4211a2d0238d/users/41c9d68bdd49472abf7cf70f4085aa0c/roles/cc56c4ae13a7424ea771c15da4b153c1 => generated 0 bytes in 52 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 13 04:29:42.400868 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6d1b6c3-f784-4a38-b857-d372f6189a5d None None] REQUEST_METHOD: `POST` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:29:42.401150 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6d1b6c3-f784-4a38-b857-d372f6189a5d None None] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:29:42.401424 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6d1b6c3-f784-4a38-b857-d372f6189a5d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:29:42.430164 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.auth.core [None req-e6d1b6c3-f784-4a38-b857-d372f6189a5d None None] MFA Rules not processed for user `41c9d68bdd49472abf7cf70f4085aa0c`. Rule list: `[]` (Enabled: `True`). {{(pid=81409) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 13 04:29:42.431401 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.fernet_utils [None req-e6d1b6c3-f784-4a38-b857-d372f6189a5d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81409) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:29:42.497377 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 1281/2527] 10.222.0.27 () {58 vars in 958 bytes} [Thu Oct 13 04:29:42 2022] POST /identity/v3/auth/tokens => generated 3254 bytes in 97 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 13 04:29:42.501872 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0196dd1e-1436-478b-b155-849ae3856837 admin admin] Authenticating user token {{(pid=81408) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:29:42.556126 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG dbcounter [-] [81408] Writing DB stats keystone:SELECT=90,keystone:DELETE=7,keystone:INSERT=3 {{(pid=81408) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:29:42.573576 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0196dd1e-1436-478b-b155-849ae3856837 admin admin] Validating token access rules against request {{(pid=81408) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:29:42.578313 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ffc1c8b1-38a3-4571-9f8f-039d4efa408c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8ba823f313144da3b683bacef8fd62e0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81408) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:29:42.578909 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ffc1c8b1-38a3-4571-9f8f-039d4efa408c admin admin] REQUEST_METHOD: `POST` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:29:42.579876 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ffc1c8b1-38a3-4571-9f8f-039d4efa408c admin admin] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:29:42.580196 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ffc1c8b1-38a3-4571-9f8f-039d4efa408c admin admin] PATH_INFO: `/v3/projects` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:29:42.581202 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ffc1c8b1-38a3-4571-9f8f-039d4efa408c admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:29:42.583183 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ffc1c8b1-38a3-4571-9f8f-039d4efa408c admin admin] RBAC: Authorization granted {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:29:42.609789 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 1247/2528] 10.222.0.27 () {60 vars in 1153 bytes} [Thu Oct 13 04:29:42 2022] POST /identity/v3/projects => generated 377 bytes in 108 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 13 04:29:42.613952 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e6d1b6c3-f784-4a38-b857-d372f6189a5d None None] Authenticating user token {{(pid=81409) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:29:42.624511 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e6d1b6c3-f784-4a38-b857-d372f6189a5d None None] Validating token access rules against request {{(pid=81409) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:29:42.636740 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG dbcounter [-] [81409] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81409) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:29:42.682704 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e80a09f7-236f-47a8-a266-3da3e13c054b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8ba823f313144da3b683bacef8fd62e0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81409) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:29:42.683308 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e80a09f7-236f-47a8-a266-3da3e13c054b admin admin] REQUEST_METHOD: `POST` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:29:42.683584 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e80a09f7-236f-47a8-a266-3da3e13c054b admin admin] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:29:42.683856 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e80a09f7-236f-47a8-a266-3da3e13c054b admin admin] PATH_INFO: `/v3/users` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:29:42.684482 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e80a09f7-236f-47a8-a266-3da3e13c054b admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:29:42.686420 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e80a09f7-236f-47a8-a266-3da3e13c054b admin admin] RBAC: Authorization granted {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:29:42.708327 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 1282/2529] 10.222.0.27 () {60 vars in 1144 bytes} [Thu Oct 13 04:29:42 2022] POST /identity/v3/users => generated 289 bytes in 95 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 13 04:29:42.712486 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ffc1c8b1-38a3-4571-9f8f-039d4efa408c admin admin] Authenticating user token {{(pid=81408) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:29:42.722685 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ffc1c8b1-38a3-4571-9f8f-039d4efa408c admin admin] Validating token access rules against request {{(pid=81408) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:29:42.727279 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-12bdd46d-da9d-45d6-b627-6efcf0adc37c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8ba823f313144da3b683bacef8fd62e0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81408) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:29:42.727871 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12bdd46d-da9d-45d6-b627-6efcf0adc37c admin admin] REQUEST_METHOD: `GET` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:29:42.728148 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12bdd46d-da9d-45d6-b627-6efcf0adc37c admin admin] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:29:42.728419 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12bdd46d-da9d-45d6-b627-6efcf0adc37c admin admin] PATH_INFO: `/v3/roles` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:29:42.729056 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-12bdd46d-da9d-45d6-b627-6efcf0adc37c admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:29:42.730945 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-12bdd46d-da9d-45d6-b627-6efcf0adc37c admin admin] RBAC: Authorization granted {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:29:42.742608 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 1248/2530] 10.222.0.27 () {58 vars in 1122 bytes} [Thu Oct 13 04:29:42 2022] GET /identity/v3/roles => generated 1420 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 13 04:29:42.746513 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e80a09f7-236f-47a8-a266-3da3e13c054b admin admin] Authenticating user token {{(pid=81409) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:29:42.756961 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e80a09f7-236f-47a8-a266-3da3e13c054b admin admin] Validating token access rules against request {{(pid=81409) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:29:42.761523 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ac3d4212-9b92-4ea6-b456-89b3a8a3c738 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8ba823f313144da3b683bacef8fd62e0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81409) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:29:42.762151 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac3d4212-9b92-4ea6-b456-89b3a8a3c738 admin admin] REQUEST_METHOD: `GET` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:29:42.762457 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac3d4212-9b92-4ea6-b456-89b3a8a3c738 admin admin] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:29:42.762728 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac3d4212-9b92-4ea6-b456-89b3a8a3c738 admin admin] PATH_INFO: `/v3/roles` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:29:42.763364 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ac3d4212-9b92-4ea6-b456-89b3a8a3c738 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:29:42.764983 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ac3d4212-9b92-4ea6-b456-89b3a8a3c738 admin admin] RBAC: Authorization granted {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:29:42.775759 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 1283/2531] 10.222.0.27 () {58 vars in 1122 bytes} [Thu Oct 13 04:29:42 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 13 04:29:42.779616 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-12bdd46d-da9d-45d6-b627-6efcf0adc37c admin admin] Authenticating user token {{(pid=81408) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:29:42.790235 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-12bdd46d-da9d-45d6-b627-6efcf0adc37c admin admin] Validating token access rules against request {{(pid=81408) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:29:42.795093 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ae7ca389-8c78-4237-80fd-7fd447cbcc3f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8ba823f313144da3b683bacef8fd62e0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81408) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:29:42.795737 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae7ca389-8c78-4237-80fd-7fd447cbcc3f admin admin] REQUEST_METHOD: `PUT` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:29:42.796015 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae7ca389-8c78-4237-80fd-7fd447cbcc3f admin admin] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:29:42.796312 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae7ca389-8c78-4237-80fd-7fd447cbcc3f admin admin] PATH_INFO: `/v3/projects/cbac7832aade4cb4a46afd8c49db38a6/users/42b8edbbaebb4202b3eee6802e190dc0/roles/cc56c4ae13a7424ea771c15da4b153c1` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:29:42.811087 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ae7ca389-8c78-4237-80fd-7fd447cbcc3f admin admin] RBAC: Authorizing `identity:create_grant(project_id=cbac7832aade4cb4a46afd8c49db38a6, user_id=42b8edbbaebb4202b3eee6802e190dc0, role_id=cc56c4ae13a7424ea771c15da4b153c1)` {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:29:42.814292 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ae7ca389-8c78-4237-80fd-7fd447cbcc3f admin admin] RBAC: Authorization granted {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:29:42.827734 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 1249/2532] 10.222.0.27 () {60 vars in 1483 bytes} [Thu Oct 13 04:29:42 2022] PUT /identity/v3/projects/cbac7832aade4cb4a46afd8c49db38a6/users/42b8edbbaebb4202b3eee6802e190dc0/roles/cc56c4ae13a7424ea771c15da4b153c1 => generated 0 bytes in 49 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 13 04:29:42.838239 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3bbe417d-6ab6-4a62-9c6c-dd0edcede1c5 None None] REQUEST_METHOD: `POST` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:29:42.838523 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3bbe417d-6ab6-4a62-9c6c-dd0edcede1c5 None None] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:29:42.838798 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3bbe417d-6ab6-4a62-9c6c-dd0edcede1c5 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:29:42.867425 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.auth.core [None req-3bbe417d-6ab6-4a62-9c6c-dd0edcede1c5 None None] MFA Rules not processed for user `42b8edbbaebb4202b3eee6802e190dc0`. Rule list: `[]` (Enabled: `True`). {{(pid=81409) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 13 04:29:42.868637 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.fernet_utils [None req-3bbe417d-6ab6-4a62-9c6c-dd0edcede1c5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81409) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:29:42.932173 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 1284/2533] 10.222.0.27 () {58 vars in 958 bytes} [Thu Oct 13 04:29:42 2022] POST /identity/v3/auth/tokens => generated 3254 bytes in 95 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 13 04:29:42.936563 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ae7ca389-8c78-4237-80fd-7fd447cbcc3f admin admin] Authenticating user token {{(pid=81408) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:29:42.998259 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ae7ca389-8c78-4237-80fd-7fd447cbcc3f admin admin] Validating token access rules against request {{(pid=81408) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:29:43.003040 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f8847a32-1713-477a-a74d-3375e43a4b7e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8ba823f313144da3b683bacef8fd62e0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81408) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:29:43.003667 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8847a32-1713-477a-a74d-3375e43a4b7e admin admin] REQUEST_METHOD: `POST` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:29:43.003947 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8847a32-1713-477a-a74d-3375e43a4b7e admin admin] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:29:43.004220 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8847a32-1713-477a-a74d-3375e43a4b7e admin admin] PATH_INFO: `/v3/projects` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:29:43.004889 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f8847a32-1713-477a-a74d-3375e43a4b7e admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:29:43.006839 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f8847a32-1713-477a-a74d-3375e43a4b7e admin admin] RBAC: Authorization granted {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:29:43.032750 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 1250/2534] 10.222.0.27 () {60 vars in 1153 bytes} [Thu Oct 13 04:29:42 2022] POST /identity/v3/projects => generated 377 bytes in 96 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 13 04:29:43.036933 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3bbe417d-6ab6-4a62-9c6c-dd0edcede1c5 None None] Authenticating user token {{(pid=81409) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:29:43.047692 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3bbe417d-6ab6-4a62-9c6c-dd0edcede1c5 None None] Validating token access rules against request {{(pid=81409) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:29:43.105268 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c26c833b-e858-465d-bae4-2220cb81a0ca admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8ba823f313144da3b683bacef8fd62e0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81409) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:29:43.106136 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c26c833b-e858-465d-bae4-2220cb81a0ca admin admin] REQUEST_METHOD: `POST` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:29:43.106437 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c26c833b-e858-465d-bae4-2220cb81a0ca admin admin] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:29:43.106706 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c26c833b-e858-465d-bae4-2220cb81a0ca admin admin] PATH_INFO: `/v3/users` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:29:43.107340 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c26c833b-e858-465d-bae4-2220cb81a0ca admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:29:43.109231 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c26c833b-e858-465d-bae4-2220cb81a0ca admin admin] RBAC: Authorization granted {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:29:43.130461 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 1285/2535] 10.222.0.27 () {60 vars in 1144 bytes} [Thu Oct 13 04:29:43 2022] POST /identity/v3/users => generated 295 bytes in 94 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 13 04:29:43.134746 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f8847a32-1713-477a-a74d-3375e43a4b7e admin admin] Authenticating user token {{(pid=81408) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:29:43.145541 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f8847a32-1713-477a-a74d-3375e43a4b7e admin admin] Validating token access rules against request {{(pid=81408) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:29:43.150470 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-db846c6a-9019-475b-8698-3a38e51a57e0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8ba823f313144da3b683bacef8fd62e0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81408) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:29:43.151086 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db846c6a-9019-475b-8698-3a38e51a57e0 admin admin] REQUEST_METHOD: `GET` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:29:43.151372 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db846c6a-9019-475b-8698-3a38e51a57e0 admin admin] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:29:43.151642 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db846c6a-9019-475b-8698-3a38e51a57e0 admin admin] PATH_INFO: `/v3/roles` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:29:43.152282 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-db846c6a-9019-475b-8698-3a38e51a57e0 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:29:43.153913 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-db846c6a-9019-475b-8698-3a38e51a57e0 admin admin] RBAC: Authorization granted {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:29:43.167956 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 1251/2536] 10.222.0.27 () {58 vars in 1122 bytes} [Thu Oct 13 04:29:43 2022] GET /identity/v3/roles => generated 1420 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 13 04:29:43.169060 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c26c833b-e858-465d-bae4-2220cb81a0ca admin admin] Authenticating user token {{(pid=81409) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:29:43.180123 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c26c833b-e858-465d-bae4-2220cb81a0ca admin admin] Validating token access rules against request {{(pid=81409) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:29:43.185048 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-62fa4622-326b-44fb-b871-b840594af64f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8ba823f313144da3b683bacef8fd62e0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81409) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:29:43.185705 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62fa4622-326b-44fb-b871-b840594af64f admin admin] REQUEST_METHOD: `PUT` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:29:43.185966 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62fa4622-326b-44fb-b871-b840594af64f admin admin] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:29:43.186262 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62fa4622-326b-44fb-b871-b840594af64f admin admin] PATH_INFO: `/v3/projects/81cce0b4bce645ecb2afa01000c715ef/users/f40202607e124a73a975069c36ee7741/roles/ef55fbd94e1040be83695754075a7ed5` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:29:43.199439 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG dbcounter [-] [81409] Writing DB stats keystone:SELECT=94,keystone:INSERT=6 {{(pid=81409) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:29:43.203154 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-62fa4622-326b-44fb-b871-b840594af64f admin admin] RBAC: Authorizing `identity:create_grant(project_id=81cce0b4bce645ecb2afa01000c715ef, user_id=f40202607e124a73a975069c36ee7741, role_id=ef55fbd94e1040be83695754075a7ed5)` {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:29:43.207001 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-62fa4622-326b-44fb-b871-b840594af64f admin admin] RBAC: Authorization granted {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:29:43.225929 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 1286/2537] 10.222.0.27 () {60 vars in 1483 bytes} [Thu Oct 13 04:29:43 2022] PUT /identity/v3/projects/81cce0b4bce645ecb2afa01000c715ef/users/f40202607e124a73a975069c36ee7741/roles/ef55fbd94e1040be83695754075a7ed5 => generated 0 bytes in 54 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 13 04:29:43.226938 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-db846c6a-9019-475b-8698-3a38e51a57e0 admin admin] Authenticating user token {{(pid=81408) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:29:43.237998 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-db846c6a-9019-475b-8698-3a38e51a57e0 admin admin] Validating token access rules against request {{(pid=81408) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:29:43.293636 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e026716d-4c0f-4ade-a2ff-d57c0e6c1708 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8ba823f313144da3b683bacef8fd62e0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81408) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:29:43.294256 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e026716d-4c0f-4ade-a2ff-d57c0e6c1708 admin admin] REQUEST_METHOD: `GET` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:29:43.294540 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e026716d-4c0f-4ade-a2ff-d57c0e6c1708 admin admin] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:29:43.294809 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e026716d-4c0f-4ade-a2ff-d57c0e6c1708 admin admin] PATH_INFO: `/v3/roles` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:29:43.295442 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e026716d-4c0f-4ade-a2ff-d57c0e6c1708 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:29:43.297026 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e026716d-4c0f-4ade-a2ff-d57c0e6c1708 admin admin] RBAC: Authorization granted {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:29:43.307075 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 1252/2538] 10.222.0.27 () {58 vars in 1122 bytes} [Thu Oct 13 04:29:43 2022] GET /identity/v3/roles => generated 1420 bytes in 81 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 13 04:29:43.311365 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-62fa4622-326b-44fb-b871-b840594af64f admin admin] Authenticating user token {{(pid=81409) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:29:43.321480 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-62fa4622-326b-44fb-b871-b840594af64f admin admin] Validating token access rules against request {{(pid=81409) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:29:43.326142 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5e01e3f2-626c-4501-9adc-276d85c3b531 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8ba823f313144da3b683bacef8fd62e0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81409) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:29:43.326820 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e01e3f2-626c-4501-9adc-276d85c3b531 admin admin] REQUEST_METHOD: `PUT` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:29:43.327097 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e01e3f2-626c-4501-9adc-276d85c3b531 admin admin] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:29:43.327362 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e01e3f2-626c-4501-9adc-276d85c3b531 admin admin] PATH_INFO: `/v3/projects/81cce0b4bce645ecb2afa01000c715ef/users/f40202607e124a73a975069c36ee7741/roles/ef55fbd94e1040be83695754075a7ed5` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:29:43.329894 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5e01e3f2-626c-4501-9adc-276d85c3b531 admin admin] RBAC: Authorizing `identity:create_grant(project_id=81cce0b4bce645ecb2afa01000c715ef, user_id=f40202607e124a73a975069c36ee7741, role_id=ef55fbd94e1040be83695754075a7ed5)` {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:29:43.332670 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5e01e3f2-626c-4501-9adc-276d85c3b531 admin admin] RBAC: Authorization granted {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:29:43.342649 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 1287/2539] 10.222.0.27 () {60 vars in 1483 bytes} [Thu Oct 13 04:29:43 2022] PUT /identity/v3/projects/81cce0b4bce645ecb2afa01000c715ef/users/f40202607e124a73a975069c36ee7741/roles/ef55fbd94e1040be83695754075a7ed5 => generated 0 bytes in 32 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 13 04:29:43.354615 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95c96a33-c58c-4cbc-bcb4-665908439166 None None] REQUEST_METHOD: `POST` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:29:43.355109 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95c96a33-c58c-4cbc-bcb4-665908439166 None None] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:29:43.355208 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95c96a33-c58c-4cbc-bcb4-665908439166 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:29:43.369092 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG dbcounter [-] [81408] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81408) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:29:43.383879 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.auth.core [None req-95c96a33-c58c-4cbc-bcb4-665908439166 None None] MFA Rules not processed for user `f40202607e124a73a975069c36ee7741`. Rule list: `[]` (Enabled: `True`). {{(pid=81408) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 13 04:29:43.385095 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.fernet_utils [None req-95c96a33-c58c-4cbc-bcb4-665908439166 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81408) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:29:43.458461 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 1253/2540] 10.222.0.27 () {58 vars in 958 bytes} [Thu Oct 13 04:29:43 2022] POST /identity/v3/auth/tokens => generated 3321 bytes in 105 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 13 04:29:43.469951 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5e01e3f2-626c-4501-9adc-276d85c3b531 admin admin] Authenticating user token {{(pid=81409) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:29:43.471724 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.fernet_utils [None req-5e01e3f2-626c-4501-9adc-276d85c3b531 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81409) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:29:43.536285 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5e01e3f2-626c-4501-9adc-276d85c3b531 admin admin] Validating token access rules against request {{(pid=81409) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:29:43.539943 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ca295d90-477d-4275-b6ca-2a16c31c6e3f service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6141b8a6c3d54e179d6879bb019f4d12', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dcec0431fb6849499d6aec35a234daa6', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81409) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:29:43.540543 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca295d90-477d-4275-b6ca-2a16c31c6e3f service cinder] REQUEST_METHOD: `GET` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:29:43.540817 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca295d90-477d-4275-b6ca-2a16c31c6e3f service cinder] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:29:43.541084 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca295d90-477d-4275-b6ca-2a16c31c6e3f service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:29:43.597310 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ca295d90-477d-4275-b6ca-2a16c31c6e3f service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:29:43.598779 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ca295d90-477d-4275-b6ca-2a16c31c6e3f service cinder] RBAC: Authorization granted {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:29:43.608264 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 1288/2541] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Oct 13 04:29:43 2022] GET /identity/v3/auth/tokens => generated 3254 bytes in 139 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 13 04:29:45.128940 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-95c96a33-c58c-4cbc-bcb4-665908439166 None None] Authenticating user token {{(pid=81408) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:29:45.130423 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.fernet_utils [None req-95c96a33-c58c-4cbc-bcb4-665908439166 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81408) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:29:45.148307 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-95c96a33-c58c-4cbc-bcb4-665908439166 None None] Validating token access rules against request {{(pid=81408) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:29:45.151852 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4e40f1bb-62e3-4a22-8ae0-cf9b142d9219 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6141b8a6c3d54e179d6879bb019f4d12', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dcec0431fb6849499d6aec35a234daa6', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81408) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:29:45.152453 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e40f1bb-62e3-4a22-8ae0-cf9b142d9219 service cinder] REQUEST_METHOD: `GET` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:29:45.152724 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e40f1bb-62e3-4a22-8ae0-cf9b142d9219 service cinder] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:29:45.152990 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e40f1bb-62e3-4a22-8ae0-cf9b142d9219 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:29:45.208546 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4e40f1bb-62e3-4a22-8ae0-cf9b142d9219 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:29:45.209870 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4e40f1bb-62e3-4a22-8ae0-cf9b142d9219 service cinder] RBAC: Authorization granted {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:29:45.218495 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 1254/2542] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Oct 13 04:29:45 2022] GET /identity/v3/auth/tokens => generated 3254 bytes in 90 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 13 04:29:45.690468 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ca295d90-477d-4275-b6ca-2a16c31c6e3f service cinder] Authenticating user token {{(pid=81409) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:29:45.691965 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.fernet_utils [None req-ca295d90-477d-4275-b6ca-2a16c31c6e3f service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81409) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:29:45.746197 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4e40f1bb-62e3-4a22-8ae0-cf9b142d9219 service cinder] Authenticating user token {{(pid=81408) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:29:45.755536 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4e40f1bb-62e3-4a22-8ae0-cf9b142d9219 service cinder] Validating token access rules against request {{(pid=81408) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:29:45.759168 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7b80aaaa-94b5-46b9-b53d-96850d8cb21d service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6141b8a6c3d54e179d6879bb019f4d12', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dcec0431fb6849499d6aec35a234daa6', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81408) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:29:45.759643 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ca295d90-477d-4275-b6ca-2a16c31c6e3f service cinder] Validating token access rules against request {{(pid=81409) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:29:45.759769 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b80aaaa-94b5-46b9-b53d-96850d8cb21d service cinder] REQUEST_METHOD: `GET` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:29:45.760016 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b80aaaa-94b5-46b9-b53d-96850d8cb21d service cinder] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:29:45.760289 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b80aaaa-94b5-46b9-b53d-96850d8cb21d service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:29:45.764120 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2ea873c7-ac53-4cb4-b505-d9d833823a8a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8ba823f313144da3b683bacef8fd62e0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81409) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:29:45.764745 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ea873c7-ac53-4cb4-b505-d9d833823a8a admin admin] REQUEST_METHOD: `DELETE` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:29:45.765022 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ea873c7-ac53-4cb4-b505-d9d833823a8a admin admin] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:29:45.765293 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ea873c7-ac53-4cb4-b505-d9d833823a8a admin admin] PATH_INFO: `/v3/users/d8c58849da3a41d7b1c068b0bb9a512c` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:29:45.766524 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2ea873c7-ac53-4cb4-b505-d9d833823a8a admin admin] RBAC: Authorizing `identity:delete_user(user_id=d8c58849da3a41d7b1c068b0bb9a512c)` {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:29:45.768956 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2ea873c7-ac53-4cb4-b505-d9d833823a8a admin admin] RBAC: Authorization granted {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:29:45.769806 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7b80aaaa-94b5-46b9-b53d-96850d8cb21d service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:29:45.771269 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7b80aaaa-94b5-46b9-b53d-96850d8cb21d service cinder] RBAC: Authorization granted {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:29:45.780753 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 1255/2543] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Oct 13 04:29:45 2022] GET /identity/v3/auth/tokens => generated 3321 bytes in 35 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 13 04:29:45.806384 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG dbcounter [-] [81409] Writing DB stats keystone:SELECT=93,keystone:INSERT=2,keystone:DELETE=5 {{(pid=81409) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:29:45.818191 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.notifications [None req-2ea873c7-ac53-4cb4-b505-d9d833823a8a admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'd8c58849da3a41d7b1c068b0bb9a512c'} {{(pid=81409) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 13 04:29:45.818960 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.notifications [None req-2ea873c7-ac53-4cb4-b505-d9d833823a8a admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'd8c58849da3a41d7b1c068b0bb9a512c'} {{(pid=81409) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 13 04:29:45.826820 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.notifications [None req-2ea873c7-ac53-4cb4-b505-d9d833823a8a admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'd8c58849da3a41d7b1c068b0bb9a512c'} {{(pid=81409) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 13 04:29:45.827578 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.notifications [None req-2ea873c7-ac53-4cb4-b505-d9d833823a8a admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'd8c58849da3a41d7b1c068b0bb9a512c'} {{(pid=81409) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 13 04:29:45.835553 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.notifications [None req-2ea873c7-ac53-4cb4-b505-d9d833823a8a admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'd8c58849da3a41d7b1c068b0bb9a512c'} {{(pid=81409) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 13 04:29:45.857293 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 1289/2544] 10.222.0.27 () {58 vars in 1224 bytes} [Thu Oct 13 04:29:45 2022] DELETE /identity/v3/users/d8c58849da3a41d7b1c068b0bb9a512c => generated 0 bytes in 167 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 13 04:29:45.862021 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70cfdb63-2421-40a8-ada6-aa5063c997d9 None None] REQUEST_METHOD: `POST` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:29:45.862328 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70cfdb63-2421-40a8-ada6-aa5063c997d9 None None] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:29:45.862600 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70cfdb63-2421-40a8-ada6-aa5063c997d9 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:29:45.882981 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.auth.core [None req-70cfdb63-2421-40a8-ada6-aa5063c997d9 None None] MFA Rules not processed for user `0e2d979563d6421995451a0e86bf3022`. Rule list: `[]` (Enabled: `True`). {{(pid=81408) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 13 04:29:45.884193 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.fernet_utils [None req-70cfdb63-2421-40a8-ada6-aa5063c997d9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81408) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:29:45.942545 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 1256/2545] 10.222.0.27 () {58 vars in 958 bytes} [Thu Oct 13 04:29:45 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 82 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 13 04:29:45.952731 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2ea873c7-ac53-4cb4-b505-d9d833823a8a admin admin] Authenticating user token {{(pid=81409) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:29:45.955094 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.fernet_utils [None req-2ea873c7-ac53-4cb4-b505-d9d833823a8a admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81409) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:29:46.011936 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2ea873c7-ac53-4cb4-b505-d9d833823a8a admin admin] Validating token access rules against request {{(pid=81409) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:29:46.015776 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e8221069-973a-4c14-8821-1fab1c230648 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd8812c3974f84871bb2d761cf0016257', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dcec0431fb6849499d6aec35a234daa6', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81409) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:29:46.016376 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8221069-973a-4c14-8821-1fab1c230648 service neutron] REQUEST_METHOD: `GET` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:29:46.017477 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8221069-973a-4c14-8821-1fab1c230648 service neutron] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:29:46.017904 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8221069-973a-4c14-8821-1fab1c230648 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:29:46.027360 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e8221069-973a-4c14-8821-1fab1c230648 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:29:46.028861 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e8221069-973a-4c14-8821-1fab1c230648 service neutron] RBAC: Authorization granted {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:29:46.038535 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 1290/2546] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Oct 13 04:29:45 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 86 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 13 04:29:46.306320 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-70cfdb63-2421-40a8-ada6-aa5063c997d9 None None] Authenticating user token {{(pid=81408) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:29:46.307868 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.fernet_utils [None req-70cfdb63-2421-40a8-ada6-aa5063c997d9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81408) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:29:46.322620 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-70cfdb63-2421-40a8-ada6-aa5063c997d9 None None] Validating token access rules against request {{(pid=81408) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:29:46.327450 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aa452f13-d351-4e01-89da-6352b373fe8b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8ba823f313144da3b683bacef8fd62e0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81408) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:29:46.328119 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa452f13-d351-4e01-89da-6352b373fe8b admin admin] REQUEST_METHOD: `DELETE` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:29:46.328395 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa452f13-d351-4e01-89da-6352b373fe8b admin admin] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:29:46.328665 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa452f13-d351-4e01-89da-6352b373fe8b admin admin] PATH_INFO: `/v3/projects/8df7ef685ef049afa8de293738c98406` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:29:46.329834 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aa452f13-d351-4e01-89da-6352b373fe8b admin admin] RBAC: Authorizing `identity:delete_project(project_id=8df7ef685ef049afa8de293738c98406)` {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:29:46.331790 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aa452f13-d351-4e01-89da-6352b373fe8b admin admin] RBAC: Authorization granted {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:29:46.343529 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG dbcounter [-] [81408] Writing DB stats keystone:SELECT=100 {{(pid=81408) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:29:46.382457 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.notifications [None req-aa452f13-d351-4e01-89da-6352b373fe8b admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '8df7ef685ef049afa8de293738c98406', 'request_id': 'req-aa452f13-d351-4e01-89da-6352b373fe8b'} {{(pid=81408) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 13 04:29:46.390921 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.notifications [None req-aa452f13-d351-4e01-89da-6352b373fe8b admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '8df7ef685ef049afa8de293738c98406', 'request_id': 'req-aa452f13-d351-4e01-89da-6352b373fe8b'} {{(pid=81408) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 13 04:29:46.397656 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.notifications [None req-aa452f13-d351-4e01-89da-6352b373fe8b admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '8df7ef685ef049afa8de293738c98406', 'request_id': 'req-aa452f13-d351-4e01-89da-6352b373fe8b'} {{(pid=81408) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 13 04:29:46.408648 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.notifications [None req-aa452f13-d351-4e01-89da-6352b373fe8b admin admin] The token cache is being invalidate because project 8df7ef685ef049afa8de293738c98406 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81408) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 13 04:29:46.408961 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.notifications [None req-aa452f13-d351-4e01-89da-6352b373fe8b admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81408) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 13 04:29:46.409617 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.notifications [None req-aa452f13-d351-4e01-89da-6352b373fe8b admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81408) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 13 04:29:46.411135 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 1257/2547] 10.222.0.27 () {58 vars in 1233 bytes} [Thu Oct 13 04:29:46 2022] DELETE /identity/v3/projects/8df7ef685ef049afa8de293738c98406 => generated 0 bytes in 105 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 13 04:29:46.416313 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a1cd731-9a82-4971-baa6-54ae7ceba616 None None] REQUEST_METHOD: `POST` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:29:46.416677 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a1cd731-9a82-4971-baa6-54ae7ceba616 None None] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:29:46.417004 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a1cd731-9a82-4971-baa6-54ae7ceba616 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:29:46.436797 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.auth.core [None req-8a1cd731-9a82-4971-baa6-54ae7ceba616 None None] MFA Rules not processed for user `0e2d979563d6421995451a0e86bf3022`. Rule list: `[]` (Enabled: `True`). {{(pid=81409) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 13 04:29:46.437983 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.fernet_utils [None req-8a1cd731-9a82-4971-baa6-54ae7ceba616 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81409) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:29:46.497070 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 1291/2548] 10.222.0.27 () {58 vars in 958 bytes} [Thu Oct 13 04:29:46 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 82 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 13 04:29:46.505918 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e6b9801-db87-40c3-9867-df844dd4addc None None] REQUEST_METHOD: `POST` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:29:46.506227 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e6b9801-db87-40c3-9867-df844dd4addc None None] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:29:46.506505 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e6b9801-db87-40c3-9867-df844dd4addc None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:29:46.526024 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.auth.core [None req-6e6b9801-db87-40c3-9867-df844dd4addc None None] MFA Rules not processed for user `0e2d979563d6421995451a0e86bf3022`. Rule list: `[]` (Enabled: `True`). {{(pid=81408) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 13 04:29:46.527250 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.fernet_utils [None req-6e6b9801-db87-40c3-9867-df844dd4addc None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81408) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:29:46.536061 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 1258/2549] 10.222.0.27 () {58 vars in 958 bytes} [Thu Oct 13 04:29:46 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 31 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 13 04:29:46.541120 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8a1cd731-9a82-4971-baa6-54ae7ceba616 None None] Authenticating user token {{(pid=81409) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:29:46.552204 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8a1cd731-9a82-4971-baa6-54ae7ceba616 None None] Validating token access rules against request {{(pid=81409) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:29:46.556755 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3c2d9685-eace-4057-acbf-ba1724c06304 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8ba823f313144da3b683bacef8fd62e0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81409) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:29:46.557357 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c2d9685-eace-4057-acbf-ba1724c06304 admin admin] REQUEST_METHOD: `GET` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:29:46.557632 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c2d9685-eace-4057-acbf-ba1724c06304 admin admin] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:29:46.557901 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c2d9685-eace-4057-acbf-ba1724c06304 admin admin] PATH_INFO: `/v3/domains` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:29:46.558580 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3c2d9685-eace-4057-acbf-ba1724c06304 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81409) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 13 04:29:46.558880 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3c2d9685-eace-4057-acbf-ba1724c06304 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:29:46.560470 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3c2d9685-eace-4057-acbf-ba1724c06304 admin admin] RBAC: Authorization granted {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:29:46.573018 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 1292/2550] 10.222.0.27 () {58 vars in 1153 bytes} [Thu Oct 13 04:29:46 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 33 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 13 04:29:46.577223 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6e6b9801-db87-40c3-9867-df844dd4addc None None] Authenticating user token {{(pid=81408) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:29:46.587188 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6e6b9801-db87-40c3-9867-df844dd4addc None None] Validating token access rules against request {{(pid=81408) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:29:46.591796 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-350f9a0b-21fc-45c2-81b3-7f7d8b8f2344 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8ba823f313144da3b683bacef8fd62e0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81408) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:29:46.592392 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-350f9a0b-21fc-45c2-81b3-7f7d8b8f2344 admin admin] REQUEST_METHOD: `POST` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:29:46.592667 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-350f9a0b-21fc-45c2-81b3-7f7d8b8f2344 admin admin] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:29:46.592935 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-350f9a0b-21fc-45c2-81b3-7f7d8b8f2344 admin admin] PATH_INFO: `/v3/projects` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:29:46.593573 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-350f9a0b-21fc-45c2-81b3-7f7d8b8f2344 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:29:46.595584 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-350f9a0b-21fc-45c2-81b3-7f7d8b8f2344 admin admin] RBAC: Authorization granted {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:29:46.622947 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 1259/2551] 10.222.0.27 () {60 vars in 1153 bytes} [Thu Oct 13 04:29:46 2022] POST /identity/v3/projects => generated 379 bytes in 46 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 13 04:29:46.627611 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3c2d9685-eace-4057-acbf-ba1724c06304 admin admin] Authenticating user token {{(pid=81409) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:29:46.637706 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3c2d9685-eace-4057-acbf-ba1724c06304 admin admin] Validating token access rules against request {{(pid=81409) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:29:46.677402 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG dbcounter [-] [81409] Writing DB stats keystone:SELECT=94,keystone:DELETE=5,keystone:INSERT=1 {{(pid=81409) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:29:46.696538 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-03fbf3eb-2114-4a5a-8fc6-37b616d503ac admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8ba823f313144da3b683bacef8fd62e0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81409) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:29:46.697156 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03fbf3eb-2114-4a5a-8fc6-37b616d503ac admin admin] REQUEST_METHOD: `POST` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:29:46.697403 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03fbf3eb-2114-4a5a-8fc6-37b616d503ac admin admin] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:29:46.697672 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03fbf3eb-2114-4a5a-8fc6-37b616d503ac admin admin] PATH_INFO: `/v3/users` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:29:46.698325 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-03fbf3eb-2114-4a5a-8fc6-37b616d503ac admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:29:46.700292 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-03fbf3eb-2114-4a5a-8fc6-37b616d503ac admin admin] RBAC: Authorization granted {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:29:46.721827 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 1293/2552] 10.222.0.27 () {60 vars in 1144 bytes} [Thu Oct 13 04:29:46 2022] POST /identity/v3/users => generated 290 bytes in 95 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 13 04:29:46.725912 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-350f9a0b-21fc-45c2-81b3-7f7d8b8f2344 admin admin] Authenticating user token {{(pid=81408) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:29:46.736534 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-350f9a0b-21fc-45c2-81b3-7f7d8b8f2344 admin admin] Validating token access rules against request {{(pid=81408) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:29:46.741143 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-32c18498-0663-4109-98ed-dcd76a81c5df admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8ba823f313144da3b683bacef8fd62e0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81408) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:29:46.741732 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32c18498-0663-4109-98ed-dcd76a81c5df admin admin] REQUEST_METHOD: `GET` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:29:46.742004 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32c18498-0663-4109-98ed-dcd76a81c5df admin admin] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:29:46.742306 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32c18498-0663-4109-98ed-dcd76a81c5df admin admin] PATH_INFO: `/v3/roles` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:29:46.742939 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-32c18498-0663-4109-98ed-dcd76a81c5df admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:29:46.744551 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-32c18498-0663-4109-98ed-dcd76a81c5df admin admin] RBAC: Authorization granted {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:29:46.755529 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 1260/2553] 10.222.0.27 () {58 vars in 1122 bytes} [Thu Oct 13 04:29:46 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 13 04:29:46.759312 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-03fbf3eb-2114-4a5a-8fc6-37b616d503ac admin admin] Authenticating user token {{(pid=81409) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:29:46.769984 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-03fbf3eb-2114-4a5a-8fc6-37b616d503ac admin admin] Validating token access rules against request {{(pid=81409) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:29:46.774579 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-632db809-3bc0-4f74-87aa-de2f356aa7bf admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8ba823f313144da3b683bacef8fd62e0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81409) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:29:46.775172 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-632db809-3bc0-4f74-87aa-de2f356aa7bf admin admin] REQUEST_METHOD: `GET` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:29:46.775474 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-632db809-3bc0-4f74-87aa-de2f356aa7bf admin admin] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:29:46.775744 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-632db809-3bc0-4f74-87aa-de2f356aa7bf admin admin] PATH_INFO: `/v3/roles` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:29:46.776380 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-632db809-3bc0-4f74-87aa-de2f356aa7bf admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:29:46.777961 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-632db809-3bc0-4f74-87aa-de2f356aa7bf admin admin] RBAC: Authorization granted {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:29:46.789137 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 1294/2554] 10.222.0.27 () {58 vars in 1122 bytes} [Thu Oct 13 04:29:46 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Oct 13 04:29:46.793193 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-32c18498-0663-4109-98ed-dcd76a81c5df admin admin] Authenticating user token {{(pid=81408) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:29:46.803864 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-32c18498-0663-4109-98ed-dcd76a81c5df admin admin] Validating token access rules against request {{(pid=81408) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:29:46.808370 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3437c576-7e70-45d2-87ee-1e9a69cce46d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8ba823f313144da3b683bacef8fd62e0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81408) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:29:46.809206 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3437c576-7e70-45d2-87ee-1e9a69cce46d admin admin] REQUEST_METHOD: `PUT` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:29:46.809483 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3437c576-7e70-45d2-87ee-1e9a69cce46d admin admin] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:29:46.809755 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3437c576-7e70-45d2-87ee-1e9a69cce46d admin admin] PATH_INFO: `/v3/projects/eb360c69b8c240408cccc0cfbbb709f1/users/4cd203fe04644ace8be512ea2c455774/roles/cc56c4ae13a7424ea771c15da4b153c1` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:29:46.824167 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3437c576-7e70-45d2-87ee-1e9a69cce46d admin admin] RBAC: Authorizing `identity:create_grant(project_id=eb360c69b8c240408cccc0cfbbb709f1, user_id=4cd203fe04644ace8be512ea2c455774, role_id=cc56c4ae13a7424ea771c15da4b153c1)` {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:29:46.826857 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3437c576-7e70-45d2-87ee-1e9a69cce46d admin admin] RBAC: Authorization granted {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:29:46.839962 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 1261/2555] 10.222.0.27 () {60 vars in 1483 bytes} [Thu Oct 13 04:29:46 2022] PUT /identity/v3/projects/eb360c69b8c240408cccc0cfbbb709f1/users/4cd203fe04644ace8be512ea2c455774/roles/cc56c4ae13a7424ea771c15da4b153c1 => generated 0 bytes in 47 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 13 04:29:46.845453 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b32709ee-122d-4995-af53-19683e1089f7 None None] REQUEST_METHOD: `POST` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:29:46.845909 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b32709ee-122d-4995-af53-19683e1089f7 None None] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:29:46.846184 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b32709ee-122d-4995-af53-19683e1089f7 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:29:46.866123 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.auth.core [None req-b32709ee-122d-4995-af53-19683e1089f7 None None] MFA Rules not processed for user `0e2d979563d6421995451a0e86bf3022`. Rule list: `[]` (Enabled: `True`). {{(pid=81409) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 13 04:29:46.867355 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.fernet_utils [None req-b32709ee-122d-4995-af53-19683e1089f7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81409) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:29:46.929011 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 1295/2556] 10.222.0.27 () {58 vars in 958 bytes} [Thu Oct 13 04:29:46 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 85 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 13 04:29:46.938897 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3437c576-7e70-45d2-87ee-1e9a69cce46d admin admin] Authenticating user token {{(pid=81408) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:29:46.940599 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.fernet_utils [None req-3437c576-7e70-45d2-87ee-1e9a69cce46d admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81408) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:29:46.999335 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3437c576-7e70-45d2-87ee-1e9a69cce46d admin admin] Validating token access rules against request {{(pid=81408) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:29:47.003225 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2d30dc30-8be2-496a-8c87-8f72d58c83c9 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd8812c3974f84871bb2d761cf0016257', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dcec0431fb6849499d6aec35a234daa6', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81408) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:29:47.003833 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d30dc30-8be2-496a-8c87-8f72d58c83c9 service neutron] REQUEST_METHOD: `GET` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:29:47.004116 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d30dc30-8be2-496a-8c87-8f72d58c83c9 service neutron] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:29:47.004387 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d30dc30-8be2-496a-8c87-8f72d58c83c9 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:29:47.014224 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2d30dc30-8be2-496a-8c87-8f72d58c83c9 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:29:47.015658 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2d30dc30-8be2-496a-8c87-8f72d58c83c9 service neutron] RBAC: Authorization granted {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:29:47.025906 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 1262/2557] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Oct 13 04:29:46 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 88 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 13 04:29:49.804913 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b32709ee-122d-4995-af53-19683e1089f7 None None] Authenticating user token {{(pid=81409) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:29:49.806597 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.fernet_utils [None req-b32709ee-122d-4995-af53-19683e1089f7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81409) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:29:49.820691 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b32709ee-122d-4995-af53-19683e1089f7 None None] Validating token access rules against request {{(pid=81409) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:29:49.825974 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1cb29b52-d45d-4ee2-82c5-265a3cfbd245 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8ba823f313144da3b683bacef8fd62e0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81409) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:29:49.826992 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1cb29b52-d45d-4ee2-82c5-265a3cfbd245 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:29:49.827475 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1cb29b52-d45d-4ee2-82c5-265a3cfbd245 admin admin] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:29:49.828085 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1cb29b52-d45d-4ee2-82c5-265a3cfbd245 admin admin] PATH_INFO: `/v3/users/41c9d68bdd49472abf7cf70f4085aa0c` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:29:49.830541 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1cb29b52-d45d-4ee2-82c5-265a3cfbd245 admin admin] RBAC: Authorizing `identity:delete_user(user_id=41c9d68bdd49472abf7cf70f4085aa0c)` {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:29:49.832790 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1cb29b52-d45d-4ee2-82c5-265a3cfbd245 admin admin] RBAC: Authorization granted {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:29:49.901238 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.notifications [None req-1cb29b52-d45d-4ee2-82c5-265a3cfbd245 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '41c9d68bdd49472abf7cf70f4085aa0c'} {{(pid=81409) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 13 04:29:49.902570 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.notifications [None req-1cb29b52-d45d-4ee2-82c5-265a3cfbd245 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '41c9d68bdd49472abf7cf70f4085aa0c'} {{(pid=81409) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 13 04:29:49.915147 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.notifications [None req-1cb29b52-d45d-4ee2-82c5-265a3cfbd245 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '41c9d68bdd49472abf7cf70f4085aa0c'} {{(pid=81409) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 13 04:29:49.916018 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.notifications [None req-1cb29b52-d45d-4ee2-82c5-265a3cfbd245 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '41c9d68bdd49472abf7cf70f4085aa0c'} {{(pid=81409) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 13 04:29:49.926043 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.notifications [None req-1cb29b52-d45d-4ee2-82c5-265a3cfbd245 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '41c9d68bdd49472abf7cf70f4085aa0c'} {{(pid=81409) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 13 04:29:49.956403 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 1296/2558] 10.222.0.27 () {58 vars in 1224 bytes} [Thu Oct 13 04:29:49 2022] DELETE /identity/v3/users/41c9d68bdd49472abf7cf70f4085aa0c => generated 0 bytes in 147 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 13 04:29:49.957050 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9998484-2779-4ff3-9f60-57de2aea08ba None None] REQUEST_METHOD: `POST` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:29:49.957499 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9998484-2779-4ff3-9f60-57de2aea08ba None None] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:29:49.957615 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9998484-2779-4ff3-9f60-57de2aea08ba None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:29:49.983414 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.auth.core [None req-a9998484-2779-4ff3-9f60-57de2aea08ba None None] MFA Rules not processed for user `0e2d979563d6421995451a0e86bf3022`. Rule list: `[]` (Enabled: `True`). {{(pid=81408) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 13 04:29:49.985548 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.fernet_utils [None req-a9998484-2779-4ff3-9f60-57de2aea08ba None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81408) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:29:50.030305 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG dbcounter [-] [81408] Writing DB stats keystone:SELECT=90,keystone:DELETE=7,keystone:INSERT=3 {{(pid=81408) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:29:50.069829 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 1263/2559] 10.222.0.27 () {58 vars in 958 bytes} [Thu Oct 13 04:29:49 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 111 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 13 04:29:50.077587 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1cb29b52-d45d-4ee2-82c5-265a3cfbd245 admin admin] Authenticating user token {{(pid=81409) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:29:50.078875 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.fernet_utils [None req-1cb29b52-d45d-4ee2-82c5-265a3cfbd245 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81409) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:29:50.119573 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG dbcounter [-] [81409] Writing DB stats keystone:SELECT=86,keystone:INSERT=4,keystone:DELETE=10 {{(pid=81409) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:29:50.153626 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1cb29b52-d45d-4ee2-82c5-265a3cfbd245 admin admin] Validating token access rules against request {{(pid=81409) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:29:50.158897 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a875c501-0581-4a22-9523-032492c1b20d service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd8812c3974f84871bb2d761cf0016257', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dcec0431fb6849499d6aec35a234daa6', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81409) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:29:50.159473 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a875c501-0581-4a22-9523-032492c1b20d service neutron] REQUEST_METHOD: `GET` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:29:50.159883 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a875c501-0581-4a22-9523-032492c1b20d service neutron] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:29:50.160331 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a875c501-0581-4a22-9523-032492c1b20d service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:29:50.174583 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a875c501-0581-4a22-9523-032492c1b20d service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:29:50.177874 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a875c501-0581-4a22-9523-032492c1b20d service neutron] RBAC: Authorization granted {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:29:50.188992 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 1297/2560] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Oct 13 04:29:50 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 112 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 13 04:29:50.565842 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a9998484-2779-4ff3-9f60-57de2aea08ba None None] Authenticating user token {{(pid=81408) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:29:50.569612 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.fernet_utils [None req-a9998484-2779-4ff3-9f60-57de2aea08ba None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81408) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:29:50.591133 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a9998484-2779-4ff3-9f60-57de2aea08ba None None] Validating token access rules against request {{(pid=81408) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:29:50.602299 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-401fb899-0804-407c-9682-6a0f9b3e3070 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8ba823f313144da3b683bacef8fd62e0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81408) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:29:50.603241 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-401fb899-0804-407c-9682-6a0f9b3e3070 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:29:50.603778 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-401fb899-0804-407c-9682-6a0f9b3e3070 admin admin] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:29:50.604206 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-401fb899-0804-407c-9682-6a0f9b3e3070 admin admin] PATH_INFO: `/v3/projects/b70ff895a0594ede86db4211a2d0238d` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:29:50.606327 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-401fb899-0804-407c-9682-6a0f9b3e3070 admin admin] RBAC: Authorizing `identity:delete_project(project_id=b70ff895a0594ede86db4211a2d0238d)` {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:29:50.608436 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-401fb899-0804-407c-9682-6a0f9b3e3070 admin admin] RBAC: Authorization granted {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:29:50.676392 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.notifications [None req-401fb899-0804-407c-9682-6a0f9b3e3070 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'b70ff895a0594ede86db4211a2d0238d', 'request_id': 'req-401fb899-0804-407c-9682-6a0f9b3e3070'} {{(pid=81408) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 13 04:29:50.692336 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.notifications [None req-401fb899-0804-407c-9682-6a0f9b3e3070 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'b70ff895a0594ede86db4211a2d0238d', 'request_id': 'req-401fb899-0804-407c-9682-6a0f9b3e3070'} {{(pid=81408) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 13 04:29:50.702473 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.notifications [None req-401fb899-0804-407c-9682-6a0f9b3e3070 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'b70ff895a0594ede86db4211a2d0238d', 'request_id': 'req-401fb899-0804-407c-9682-6a0f9b3e3070'} {{(pid=81408) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 13 04:29:50.728298 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.notifications [None req-401fb899-0804-407c-9682-6a0f9b3e3070 admin admin] The token cache is being invalidate because project b70ff895a0594ede86db4211a2d0238d was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81408) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 13 04:29:50.728593 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.notifications [None req-401fb899-0804-407c-9682-6a0f9b3e3070 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81408) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 13 04:29:50.729384 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.notifications [None req-401fb899-0804-407c-9682-6a0f9b3e3070 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81408) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 13 04:29:50.733933 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 1264/2561] 10.222.0.27 () {58 vars in 1233 bytes} [Thu Oct 13 04:29:50 2022] DELETE /identity/v3/projects/b70ff895a0594ede86db4211a2d0238d => generated 0 bytes in 166 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 13 04:29:50.735861 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a875c501-0581-4a22-9523-032492c1b20d service neutron] Authenticating user token {{(pid=81409) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:29:50.749123 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a875c501-0581-4a22-9523-032492c1b20d service neutron] Validating token access rules against request {{(pid=81409) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:29:50.834922 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-25d0687c-e17b-462e-931c-c788f08d8b48 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8ba823f313144da3b683bacef8fd62e0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81409) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:29:50.835657 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25d0687c-e17b-462e-931c-c788f08d8b48 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:29:50.835941 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25d0687c-e17b-462e-931c-c788f08d8b48 admin admin] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:29:50.836214 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25d0687c-e17b-462e-931c-c788f08d8b48 admin admin] PATH_INFO: `/v3/users/42b8edbbaebb4202b3eee6802e190dc0` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:29:50.837548 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-25d0687c-e17b-462e-931c-c788f08d8b48 admin admin] RBAC: Authorizing `identity:delete_user(user_id=42b8edbbaebb4202b3eee6802e190dc0)` {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:29:50.839708 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-25d0687c-e17b-462e-931c-c788f08d8b48 admin admin] RBAC: Authorization granted {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:29:50.905413 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.notifications [None req-25d0687c-e17b-462e-931c-c788f08d8b48 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '42b8edbbaebb4202b3eee6802e190dc0'} {{(pid=81409) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 13 04:29:50.906530 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.notifications [None req-25d0687c-e17b-462e-931c-c788f08d8b48 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '42b8edbbaebb4202b3eee6802e190dc0'} {{(pid=81409) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 13 04:29:50.917090 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.notifications [None req-25d0687c-e17b-462e-931c-c788f08d8b48 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '42b8edbbaebb4202b3eee6802e190dc0'} {{(pid=81409) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 13 04:29:50.918109 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.notifications [None req-25d0687c-e17b-462e-931c-c788f08d8b48 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '42b8edbbaebb4202b3eee6802e190dc0'} {{(pid=81409) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 13 04:29:50.928002 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.notifications [None req-25d0687c-e17b-462e-931c-c788f08d8b48 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '42b8edbbaebb4202b3eee6802e190dc0'} {{(pid=81409) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 13 04:29:50.951790 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 1298/2562] 10.222.0.27 () {58 vars in 1224 bytes} [Thu Oct 13 04:29:50 2022] DELETE /identity/v3/users/42b8edbbaebb4202b3eee6802e190dc0 => generated 0 bytes in 217 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 13 04:29:51.433835 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e600f63b-dff3-482e-88ba-b3a1189369f3 None None] REQUEST_METHOD: `POST` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:29:51.435982 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e600f63b-dff3-482e-88ba-b3a1189369f3 None None] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:29:51.436345 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e600f63b-dff3-482e-88ba-b3a1189369f3 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:29:51.464541 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-25d0687c-e17b-462e-931c-c788f08d8b48 admin admin] Authenticating user token {{(pid=81409) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:29:51.465986 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.fernet_utils [None req-25d0687c-e17b-462e-931c-c788f08d8b48 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81409) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:29:51.488763 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.auth.core [None req-e600f63b-dff3-482e-88ba-b3a1189369f3 None None] MFA Rules not processed for user `4cd203fe04644ace8be512ea2c455774`. Rule list: `[]` (Enabled: `True`). {{(pid=81408) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 13 04:29:51.490014 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.fernet_utils [None req-e600f63b-dff3-482e-88ba-b3a1189369f3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81408) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:29:51.557309 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG dbcounter [-] [81409] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81409) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:29:51.570384 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-25d0687c-e17b-462e-931c-c788f08d8b48 admin admin] Validating token access rules against request {{(pid=81409) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:29:51.576222 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-48c5a64b-89b0-4a1d-9d3b-84319d24e2c4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8ba823f313144da3b683bacef8fd62e0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81409) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:29:51.576992 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48c5a64b-89b0-4a1d-9d3b-84319d24e2c4 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:29:51.577363 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48c5a64b-89b0-4a1d-9d3b-84319d24e2c4 admin admin] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:29:51.577734 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48c5a64b-89b0-4a1d-9d3b-84319d24e2c4 admin admin] PATH_INFO: `/v3/projects/cbac7832aade4cb4a46afd8c49db38a6` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:29:51.579712 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-48c5a64b-89b0-4a1d-9d3b-84319d24e2c4 admin admin] RBAC: Authorizing `identity:delete_project(project_id=cbac7832aade4cb4a46afd8c49db38a6)` {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:29:51.581764 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-48c5a64b-89b0-4a1d-9d3b-84319d24e2c4 admin admin] RBAC: Authorization granted {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:29:51.596718 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 1265/2563] 10.222.0.27 () {58 vars in 958 bytes} [Thu Oct 13 04:29:51 2022] POST /identity/v3/auth/tokens => generated 3256 bytes in 165 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 13 04:29:51.601009 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e600f63b-dff3-482e-88ba-b3a1189369f3 None None] Authenticating user token {{(pid=81408) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:29:51.602683 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.fernet_utils [None req-e600f63b-dff3-482e-88ba-b3a1189369f3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81408) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:29:51.631218 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e600f63b-dff3-482e-88ba-b3a1189369f3 None None] Validating token access rules against request {{(pid=81408) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:29:51.643544 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.notifications [None req-48c5a64b-89b0-4a1d-9d3b-84319d24e2c4 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'cbac7832aade4cb4a46afd8c49db38a6', 'request_id': 'req-48c5a64b-89b0-4a1d-9d3b-84319d24e2c4'} {{(pid=81409) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 13 04:29:51.653529 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.notifications [None req-48c5a64b-89b0-4a1d-9d3b-84319d24e2c4 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'cbac7832aade4cb4a46afd8c49db38a6', 'request_id': 'req-48c5a64b-89b0-4a1d-9d3b-84319d24e2c4'} {{(pid=81409) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 13 04:29:51.663383 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.notifications [None req-48c5a64b-89b0-4a1d-9d3b-84319d24e2c4 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'cbac7832aade4cb4a46afd8c49db38a6', 'request_id': 'req-48c5a64b-89b0-4a1d-9d3b-84319d24e2c4'} {{(pid=81409) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 13 04:29:51.676979 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.notifications [None req-48c5a64b-89b0-4a1d-9d3b-84319d24e2c4 admin admin] The token cache is being invalidate because project cbac7832aade4cb4a46afd8c49db38a6 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81409) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 13 04:29:51.676979 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.notifications [None req-48c5a64b-89b0-4a1d-9d3b-84319d24e2c4 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81409) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 13 04:29:51.677349 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.notifications [None req-48c5a64b-89b0-4a1d-9d3b-84319d24e2c4 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81409) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 13 04:29:51.679129 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 1299/2564] 10.222.0.27 () {58 vars in 1233 bytes} [Thu Oct 13 04:29:51 2022] DELETE /identity/v3/projects/cbac7832aade4cb4a46afd8c49db38a6 => generated 0 bytes in 215 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 13 04:29:51.682795 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-48c5a64b-89b0-4a1d-9d3b-84319d24e2c4 admin admin] Authenticating user token {{(pid=81409) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:29:51.684280 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.fernet_utils [None req-48c5a64b-89b0-4a1d-9d3b-84319d24e2c4 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81409) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:29:51.693918 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG dbcounter [-] [81408] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=81408) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:29:51.708267 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a606fa24-1c24-421b-a67c-ebf7d31b0b8e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8ba823f313144da3b683bacef8fd62e0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81408) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:29:51.709349 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a606fa24-1c24-421b-a67c-ebf7d31b0b8e admin admin] REQUEST_METHOD: `POST` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:29:51.709632 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a606fa24-1c24-421b-a67c-ebf7d31b0b8e admin admin] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:29:51.709903 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a606fa24-1c24-421b-a67c-ebf7d31b0b8e admin admin] PATH_INFO: `/v3/projects` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:29:51.710579 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a606fa24-1c24-421b-a67c-ebf7d31b0b8e admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:29:51.712495 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a606fa24-1c24-421b-a67c-ebf7d31b0b8e admin admin] RBAC: Authorization granted {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:29:51.741602 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 1266/2565] 10.222.0.27 () {60 vars in 1153 bytes} [Thu Oct 13 04:29:51 2022] POST /identity/v3/projects => generated 377 bytes in 141 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 13 04:29:51.745454 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a606fa24-1c24-421b-a67c-ebf7d31b0b8e admin admin] Authenticating user token {{(pid=81408) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:29:51.747120 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.fernet_utils [None req-a606fa24-1c24-421b-a67c-ebf7d31b0b8e admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81408) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:29:51.760752 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-48c5a64b-89b0-4a1d-9d3b-84319d24e2c4 admin admin] Validating token access rules against request {{(pid=81409) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:29:51.833462 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a606fa24-1c24-421b-a67c-ebf7d31b0b8e admin admin] Validating token access rules against request {{(pid=81408) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:29:51.836765 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aead17b0-f49f-48ab-9110-a8f67e93c4a2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8ba823f313144da3b683bacef8fd62e0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81409) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:29:51.837420 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aead17b0-f49f-48ab-9110-a8f67e93c4a2 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:29:51.837922 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aead17b0-f49f-48ab-9110-a8f67e93c4a2 admin admin] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:29:51.838343 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aead17b0-f49f-48ab-9110-a8f67e93c4a2 admin admin] PATH_INFO: `/v3/users/f40202607e124a73a975069c36ee7741` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:29:51.839643 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aead17b0-f49f-48ab-9110-a8f67e93c4a2 admin admin] RBAC: Authorizing `identity:delete_user(user_id=f40202607e124a73a975069c36ee7741)` {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:29:51.839939 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e17ccdac-5f1e-4457-8a20-303e9b81063b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8ba823f313144da3b683bacef8fd62e0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81408) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:29:51.840653 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e17ccdac-5f1e-4457-8a20-303e9b81063b admin admin] REQUEST_METHOD: `POST` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:29:51.841032 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e17ccdac-5f1e-4457-8a20-303e9b81063b admin admin] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:29:51.841431 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e17ccdac-5f1e-4457-8a20-303e9b81063b admin admin] PATH_INFO: `/v3/users` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:29:51.841576 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aead17b0-f49f-48ab-9110-a8f67e93c4a2 admin admin] RBAC: Authorization granted {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:29:51.842356 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e17ccdac-5f1e-4457-8a20-303e9b81063b admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:29:51.845850 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e17ccdac-5f1e-4457-8a20-303e9b81063b admin admin] RBAC: Authorization granted {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:29:51.870281 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 1267/2566] 10.222.0.27 () {60 vars in 1144 bytes} [Thu Oct 13 04:29:51 2022] POST /identity/v3/users => generated 295 bytes in 124 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 13 04:29:51.873672 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e17ccdac-5f1e-4457-8a20-303e9b81063b admin admin] Authenticating user token {{(pid=81408) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:29:51.885551 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e17ccdac-5f1e-4457-8a20-303e9b81063b admin admin] Validating token access rules against request {{(pid=81408) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:29:51.885959 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG dbcounter [-] [81409] Writing DB stats keystone:SELECT=87,keystone:DELETE=12,keystone:INSERT=1 {{(pid=81409) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:29:51.890643 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a9c5abd5-d4ad-432d-aa60-594a012ba5ff admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8ba823f313144da3b683bacef8fd62e0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81408) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:29:51.891921 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9c5abd5-d4ad-432d-aa60-594a012ba5ff admin admin] REQUEST_METHOD: `GET` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:29:51.892143 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9c5abd5-d4ad-432d-aa60-594a012ba5ff admin admin] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:29:51.892415 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9c5abd5-d4ad-432d-aa60-594a012ba5ff admin admin] PATH_INFO: `/v3/roles` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:29:51.893678 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a9c5abd5-d4ad-432d-aa60-594a012ba5ff admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:29:51.895306 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a9c5abd5-d4ad-432d-aa60-594a012ba5ff admin admin] RBAC: Authorization granted {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:29:51.900579 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.notifications [None req-aead17b0-f49f-48ab-9110-a8f67e93c4a2 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'f40202607e124a73a975069c36ee7741'} {{(pid=81409) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 13 04:29:51.901250 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.notifications [None req-aead17b0-f49f-48ab-9110-a8f67e93c4a2 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'f40202607e124a73a975069c36ee7741'} {{(pid=81409) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 13 04:29:51.910098 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.notifications [None req-aead17b0-f49f-48ab-9110-a8f67e93c4a2 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'f40202607e124a73a975069c36ee7741'} {{(pid=81409) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 13 04:29:51.910963 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.notifications [None req-aead17b0-f49f-48ab-9110-a8f67e93c4a2 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'f40202607e124a73a975069c36ee7741'} {{(pid=81409) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 13 04:29:51.912318 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 1268/2567] 10.222.0.27 () {58 vars in 1122 bytes} [Thu Oct 13 04:29:51 2022] GET /identity/v3/roles => generated 1420 bytes in 39 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 13 04:29:51.916274 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a9c5abd5-d4ad-432d-aa60-594a012ba5ff admin admin] Authenticating user token {{(pid=81408) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:29:51.918735 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.fernet_utils [None req-a9c5abd5-d4ad-432d-aa60-594a012ba5ff admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81408) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:29:51.921855 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.notifications [None req-aead17b0-f49f-48ab-9110-a8f67e93c4a2 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'f40202607e124a73a975069c36ee7741'} {{(pid=81409) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 13 04:29:51.932816 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a9c5abd5-d4ad-432d-aa60-594a012ba5ff admin admin] Validating token access rules against request {{(pid=81408) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:29:51.937652 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-25b61bb8-107b-4d27-b5aa-35cfbec9e22d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8ba823f313144da3b683bacef8fd62e0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81408) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:29:51.939246 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25b61bb8-107b-4d27-b5aa-35cfbec9e22d admin admin] REQUEST_METHOD: `PUT` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:29:51.939583 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25b61bb8-107b-4d27-b5aa-35cfbec9e22d admin admin] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:29:51.940028 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25b61bb8-107b-4d27-b5aa-35cfbec9e22d admin admin] PATH_INFO: `/v3/projects/372da12f356748f28a1261ee8b643e8d/users/e37c37daa5dc4119b0e36508856719d2/roles/ef55fbd94e1040be83695754075a7ed5` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:29:51.941532 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 1300/2568] 10.222.0.27 () {58 vars in 1224 bytes} [Thu Oct 13 04:29:51 2022] DELETE /identity/v3/users/f40202607e124a73a975069c36ee7741 => generated 0 bytes in 259 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 13 04:29:51.959634 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-25b61bb8-107b-4d27-b5aa-35cfbec9e22d admin admin] RBAC: Authorizing `identity:create_grant(project_id=372da12f356748f28a1261ee8b643e8d, user_id=e37c37daa5dc4119b0e36508856719d2, role_id=ef55fbd94e1040be83695754075a7ed5)` {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:29:51.962549 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-25b61bb8-107b-4d27-b5aa-35cfbec9e22d admin admin] RBAC: Authorization granted {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:29:51.977531 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 1269/2569] 10.222.0.27 () {60 vars in 1483 bytes} [Thu Oct 13 04:29:51 2022] PUT /identity/v3/projects/372da12f356748f28a1261ee8b643e8d/users/e37c37daa5dc4119b0e36508856719d2/roles/ef55fbd94e1040be83695754075a7ed5 => generated 0 bytes in 62 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 13 04:29:51.984122 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aead17b0-f49f-48ab-9110-a8f67e93c4a2 admin admin] Authenticating user token {{(pid=81409) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:29:52.060574 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aead17b0-f49f-48ab-9110-a8f67e93c4a2 admin admin] Validating token access rules against request {{(pid=81409) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:29:52.159853 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4a7c6eda-3588-4c01-b356-6d8dd452b154 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8ba823f313144da3b683bacef8fd62e0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81409) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:29:52.160463 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a7c6eda-3588-4c01-b356-6d8dd452b154 admin admin] REQUEST_METHOD: `GET` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:29:52.160737 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a7c6eda-3588-4c01-b356-6d8dd452b154 admin admin] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:29:52.161003 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a7c6eda-3588-4c01-b356-6d8dd452b154 admin admin] PATH_INFO: `/v3/roles` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:29:52.161643 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4a7c6eda-3588-4c01-b356-6d8dd452b154 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:29:52.163275 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4a7c6eda-3588-4c01-b356-6d8dd452b154 admin admin] RBAC: Authorization granted {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:29:52.174561 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 1301/2570] 10.222.0.27 () {58 vars in 1122 bytes} [Thu Oct 13 04:29:51 2022] GET /identity/v3/roles => generated 1420 bytes in 192 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 13 04:29:52.178797 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-25b61bb8-107b-4d27-b5aa-35cfbec9e22d admin admin] Authenticating user token {{(pid=81408) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:29:52.190020 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-25b61bb8-107b-4d27-b5aa-35cfbec9e22d admin admin] Validating token access rules against request {{(pid=81408) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:29:52.195151 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-11439d7e-17ee-476d-86c8-d6837a5312ef admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8ba823f313144da3b683bacef8fd62e0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81408) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:29:52.196451 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11439d7e-17ee-476d-86c8-d6837a5312ef admin admin] REQUEST_METHOD: `PUT` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:29:52.196681 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11439d7e-17ee-476d-86c8-d6837a5312ef admin admin] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:29:52.197003 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11439d7e-17ee-476d-86c8-d6837a5312ef admin admin] PATH_INFO: `/v3/projects/372da12f356748f28a1261ee8b643e8d/users/e37c37daa5dc4119b0e36508856719d2/roles/ef55fbd94e1040be83695754075a7ed5` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:29:52.199089 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-11439d7e-17ee-476d-86c8-d6837a5312ef admin admin] RBAC: Authorizing `identity:create_grant(project_id=372da12f356748f28a1261ee8b643e8d, user_id=e37c37daa5dc4119b0e36508856719d2, role_id=ef55fbd94e1040be83695754075a7ed5)` {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:29:52.202996 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-11439d7e-17ee-476d-86c8-d6837a5312ef admin admin] RBAC: Authorization granted {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:29:52.222276 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 1270/2571] 10.222.0.27 () {60 vars in 1483 bytes} [Thu Oct 13 04:29:52 2022] PUT /identity/v3/projects/372da12f356748f28a1261ee8b643e8d/users/e37c37daa5dc4119b0e36508856719d2/roles/ef55fbd94e1040be83695754075a7ed5 => generated 0 bytes in 41 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 13 04:29:52.265188 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4a7c6eda-3588-4c01-b356-6d8dd452b154 admin admin] Authenticating user token {{(pid=81409) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:29:52.267276 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.fernet_utils [None req-4a7c6eda-3588-4c01-b356-6d8dd452b154 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81409) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:29:52.292445 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4a7c6eda-3588-4c01-b356-6d8dd452b154 admin admin] Validating token access rules against request {{(pid=81409) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:29:52.373266 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG dbcounter [-] [81409] Writing DB stats keystone:SELECT=94,keystone:DELETE=5,keystone:INSERT=1 {{(pid=81409) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:29:52.383889 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-78836a25-ee5e-4d33-887a-b55937df02b6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8ba823f313144da3b683bacef8fd62e0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81409) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:29:52.385158 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78836a25-ee5e-4d33-887a-b55937df02b6 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:29:52.385688 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78836a25-ee5e-4d33-887a-b55937df02b6 admin admin] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:29:52.386288 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78836a25-ee5e-4d33-887a-b55937df02b6 admin admin] PATH_INFO: `/v3/projects/81cce0b4bce645ecb2afa01000c715ef` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:29:52.388556 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-78836a25-ee5e-4d33-887a-b55937df02b6 admin admin] RBAC: Authorizing `identity:delete_project(project_id=81cce0b4bce645ecb2afa01000c715ef)` {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:29:52.390981 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-78836a25-ee5e-4d33-887a-b55937df02b6 admin admin] RBAC: Authorization granted {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:29:52.456327 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.notifications [None req-78836a25-ee5e-4d33-887a-b55937df02b6 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '81cce0b4bce645ecb2afa01000c715ef', 'request_id': 'req-78836a25-ee5e-4d33-887a-b55937df02b6'} {{(pid=81409) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 13 04:29:52.467616 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.notifications [None req-78836a25-ee5e-4d33-887a-b55937df02b6 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '81cce0b4bce645ecb2afa01000c715ef', 'request_id': 'req-78836a25-ee5e-4d33-887a-b55937df02b6'} {{(pid=81409) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 13 04:29:52.476048 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.notifications [None req-78836a25-ee5e-4d33-887a-b55937df02b6 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '81cce0b4bce645ecb2afa01000c715ef', 'request_id': 'req-78836a25-ee5e-4d33-887a-b55937df02b6'} {{(pid=81409) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 13 04:29:52.493242 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.notifications [None req-78836a25-ee5e-4d33-887a-b55937df02b6 admin admin] The token cache is being invalidate because project 81cce0b4bce645ecb2afa01000c715ef was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81409) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 13 04:29:52.493854 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.notifications [None req-78836a25-ee5e-4d33-887a-b55937df02b6 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81409) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 13 04:29:52.494496 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.notifications [None req-78836a25-ee5e-4d33-887a-b55937df02b6 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81409) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 13 04:29:52.499980 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 1302/2572] 10.222.0.27 () {58 vars in 1233 bytes} [Thu Oct 13 04:29:52 2022] DELETE /identity/v3/projects/81cce0b4bce645ecb2afa01000c715ef => generated 0 bytes in 232 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 13 04:29:52.503627 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e204394c-5bc1-4697-ab58-17f116fbfc66 None None] REQUEST_METHOD: `POST` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:29:52.504101 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e204394c-5bc1-4697-ab58-17f116fbfc66 None None] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:29:52.504204 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e204394c-5bc1-4697-ab58-17f116fbfc66 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:29:52.529649 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.auth.core [None req-e204394c-5bc1-4697-ab58-17f116fbfc66 None None] MFA Rules not processed for user `0e2d979563d6421995451a0e86bf3022`. Rule list: `[]` (Enabled: `True`). {{(pid=81408) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 13 04:29:52.530941 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.fernet_utils [None req-e204394c-5bc1-4697-ab58-17f116fbfc66 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81408) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:29:52.620198 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 1271/2573] 10.222.0.27 () {58 vars in 958 bytes} [Thu Oct 13 04:29:52 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 118 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 13 04:29:52.629328 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3125fbd6-8008-4956-8431-b0cbf8a0f0c1 None None] REQUEST_METHOD: `POST` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:29:52.629670 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3125fbd6-8008-4956-8431-b0cbf8a0f0c1 None None] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:29:52.629947 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3125fbd6-8008-4956-8431-b0cbf8a0f0c1 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:29:52.651786 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.auth.core [None req-3125fbd6-8008-4956-8431-b0cbf8a0f0c1 None None] MFA Rules not processed for user `0e2d979563d6421995451a0e86bf3022`. Rule list: `[]` (Enabled: `True`). {{(pid=81409) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 13 04:29:52.653135 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.fernet_utils [None req-3125fbd6-8008-4956-8431-b0cbf8a0f0c1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81409) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:29:52.671873 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 1303/2574] 10.222.0.27 () {58 vars in 958 bytes} [Thu Oct 13 04:29:52 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 43 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 13 04:29:52.673395 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e204394c-5bc1-4697-ab58-17f116fbfc66 None None] Authenticating user token {{(pid=81408) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:29:52.685679 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e204394c-5bc1-4697-ab58-17f116fbfc66 None None] Validating token access rules against request {{(pid=81408) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:29:52.692254 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bf9d3b87-e299-42ec-b051-ed4ba16748fd admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8ba823f313144da3b683bacef8fd62e0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81408) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:29:52.693488 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf9d3b87-e299-42ec-b051-ed4ba16748fd admin admin] REQUEST_METHOD: `GET` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:29:52.693856 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf9d3b87-e299-42ec-b051-ed4ba16748fd admin admin] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:29:52.695463 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf9d3b87-e299-42ec-b051-ed4ba16748fd admin admin] PATH_INFO: `/v3/domains` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:29:52.696148 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bf9d3b87-e299-42ec-b051-ed4ba16748fd admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81408) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 13 04:29:52.696717 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bf9d3b87-e299-42ec-b051-ed4ba16748fd admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:29:52.698971 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bf9d3b87-e299-42ec-b051-ed4ba16748fd admin admin] RBAC: Authorization granted {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:29:52.711418 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 1272/2575] 10.222.0.27 () {58 vars in 1153 bytes} [Thu Oct 13 04:29:52 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 39 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 13 04:29:52.715965 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3125fbd6-8008-4956-8431-b0cbf8a0f0c1 None None] Authenticating user token {{(pid=81409) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:29:52.726112 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3125fbd6-8008-4956-8431-b0cbf8a0f0c1 None None] Validating token access rules against request {{(pid=81409) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:29:52.731035 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-130f25e2-2fff-44f0-9151-9fb6af38f229 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8ba823f313144da3b683bacef8fd62e0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81409) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:29:52.731630 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-130f25e2-2fff-44f0-9151-9fb6af38f229 admin admin] REQUEST_METHOD: `POST` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:29:52.731900 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-130f25e2-2fff-44f0-9151-9fb6af38f229 admin admin] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:29:52.732161 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-130f25e2-2fff-44f0-9151-9fb6af38f229 admin admin] PATH_INFO: `/v3/projects` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:29:52.732782 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-130f25e2-2fff-44f0-9151-9fb6af38f229 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:29:52.734770 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-130f25e2-2fff-44f0-9151-9fb6af38f229 admin admin] RBAC: Authorization granted {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:29:52.771395 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 1304/2576] 10.222.0.27 () {60 vars in 1153 bytes} [Thu Oct 13 04:29:52 2022] POST /identity/v3/projects => generated 389 bytes in 53 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 13 04:29:52.772681 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bf9d3b87-e299-42ec-b051-ed4ba16748fd admin admin] Authenticating user token {{(pid=81408) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:29:52.785049 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bf9d3b87-e299-42ec-b051-ed4ba16748fd admin admin] Validating token access rules against request {{(pid=81408) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:29:52.789300 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG dbcounter [-] [81408] Writing DB stats keystone:SELECT=94,keystone:INSERT=6 {{(pid=81408) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:29:52.853185 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0c9579dd-729c-459c-963a-65ec9407d600 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8ba823f313144da3b683bacef8fd62e0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81408) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:29:52.853853 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c9579dd-729c-459c-963a-65ec9407d600 admin admin] REQUEST_METHOD: `POST` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:29:52.854135 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c9579dd-729c-459c-963a-65ec9407d600 admin admin] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:29:52.854433 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c9579dd-729c-459c-963a-65ec9407d600 admin admin] PATH_INFO: `/v3/users` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:29:52.855072 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0c9579dd-729c-459c-963a-65ec9407d600 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:29:52.857255 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0c9579dd-729c-459c-963a-65ec9407d600 admin admin] RBAC: Authorization granted {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:29:52.882745 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 1273/2577] 10.222.0.27 () {60 vars in 1144 bytes} [Thu Oct 13 04:29:52 2022] POST /identity/v3/users => generated 295 bytes in 111 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 13 04:29:52.887702 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-130f25e2-2fff-44f0-9151-9fb6af38f229 admin admin] Authenticating user token {{(pid=81409) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:29:52.900061 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-130f25e2-2fff-44f0-9151-9fb6af38f229 admin admin] Validating token access rules against request {{(pid=81409) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:29:52.909444 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e0fa290d-a472-43e8-950c-0c68b56ef31e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8ba823f313144da3b683bacef8fd62e0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81409) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:29:52.910502 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0fa290d-a472-43e8-950c-0c68b56ef31e admin admin] REQUEST_METHOD: `GET` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:29:52.911561 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0fa290d-a472-43e8-950c-0c68b56ef31e admin admin] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:29:52.912368 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0fa290d-a472-43e8-950c-0c68b56ef31e admin admin] PATH_INFO: `/v3/roles` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:29:52.913129 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e0fa290d-a472-43e8-950c-0c68b56ef31e admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:29:52.915362 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e0fa290d-a472-43e8-950c-0c68b56ef31e admin admin] RBAC: Authorization granted {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:29:52.933303 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 1305/2578] 10.222.0.27 () {58 vars in 1122 bytes} [Thu Oct 13 04:29:52 2022] GET /identity/v3/roles => generated 1420 bytes in 44 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 13 04:29:52.934981 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0c9579dd-729c-459c-963a-65ec9407d600 admin admin] Authenticating user token {{(pid=81408) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:29:52.946598 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0c9579dd-729c-459c-963a-65ec9407d600 admin admin] Validating token access rules against request {{(pid=81408) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:29:52.954185 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-57ef3153-277f-4ddb-af1d-fd5e3c4f1506 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8ba823f313144da3b683bacef8fd62e0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81408) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:29:52.955238 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57ef3153-277f-4ddb-af1d-fd5e3c4f1506 admin admin] REQUEST_METHOD: `GET` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:29:52.955650 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57ef3153-277f-4ddb-af1d-fd5e3c4f1506 admin admin] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:29:52.955927 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57ef3153-277f-4ddb-af1d-fd5e3c4f1506 admin admin] PATH_INFO: `/v3/roles` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:29:52.956583 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-57ef3153-277f-4ddb-af1d-fd5e3c4f1506 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:29:52.958959 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-57ef3153-277f-4ddb-af1d-fd5e3c4f1506 admin admin] RBAC: Authorization granted {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:29:52.971719 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 1274/2579] 10.222.0.27 () {58 vars in 1122 bytes} [Thu Oct 13 04:29:52 2022] GET /identity/v3/roles => generated 1420 bytes in 37 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Oct 13 04:29:52.976338 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e0fa290d-a472-43e8-950c-0c68b56ef31e admin admin] Authenticating user token {{(pid=81409) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:29:52.988035 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e0fa290d-a472-43e8-950c-0c68b56ef31e admin admin] Validating token access rules against request {{(pid=81409) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:29:52.993841 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-674d012e-a28e-4d83-a218-342244ad0da8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8ba823f313144da3b683bacef8fd62e0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81409) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:29:52.994833 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-674d012e-a28e-4d83-a218-342244ad0da8 admin admin] REQUEST_METHOD: `PUT` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:29:52.995324 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-674d012e-a28e-4d83-a218-342244ad0da8 admin admin] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:29:52.995831 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-674d012e-a28e-4d83-a218-342244ad0da8 admin admin] PATH_INFO: `/v3/projects/97397324a0194216bf0a330fa631b84e/users/c94d7ebb1fdf4e36b5e7425f00179a1f/roles/cc56c4ae13a7424ea771c15da4b153c1` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:29:53.018028 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-674d012e-a28e-4d83-a218-342244ad0da8 admin admin] RBAC: Authorizing `identity:create_grant(project_id=97397324a0194216bf0a330fa631b84e, user_id=c94d7ebb1fdf4e36b5e7425f00179a1f, role_id=cc56c4ae13a7424ea771c15da4b153c1)` {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:29:53.021131 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-674d012e-a28e-4d83-a218-342244ad0da8 admin admin] RBAC: Authorization granted {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:29:53.041782 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 1306/2580] 10.222.0.27 () {60 vars in 1483 bytes} [Thu Oct 13 04:29:52 2022] PUT /identity/v3/projects/97397324a0194216bf0a330fa631b84e/users/c94d7ebb1fdf4e36b5e7425f00179a1f/roles/cc56c4ae13a7424ea771c15da4b153c1 => generated 0 bytes in 66 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 13 04:29:53.053156 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34962362-99aa-472e-a5e1-c76f781a1f57 None None] REQUEST_METHOD: `POST` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:29:53.053529 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34962362-99aa-472e-a5e1-c76f781a1f57 None None] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:29:53.053876 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34962362-99aa-472e-a5e1-c76f781a1f57 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:29:53.085187 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.auth.core [None req-34962362-99aa-472e-a5e1-c76f781a1f57 None None] MFA Rules not processed for user `c94d7ebb1fdf4e36b5e7425f00179a1f`. Rule list: `[]` (Enabled: `True`). {{(pid=81408) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 13 04:29:53.086557 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.fernet_utils [None req-34962362-99aa-472e-a5e1-c76f781a1f57 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81408) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:29:53.176533 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 1275/2581] 10.222.0.27 () {58 vars in 958 bytes} [Thu Oct 13 04:29:53 2022] POST /identity/v3/auth/tokens => generated 3266 bytes in 124 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 13 04:29:53.187337 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-674d012e-a28e-4d83-a218-342244ad0da8 admin admin] Authenticating user token {{(pid=81409) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:29:53.189469 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.fernet_utils [None req-674d012e-a28e-4d83-a218-342244ad0da8 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81409) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:29:53.273953 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-674d012e-a28e-4d83-a218-342244ad0da8 admin admin] Validating token access rules against request {{(pid=81409) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:29:53.277872 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b4c3b7e7-24e5-43d6-bdb7-e48b08d5e2b1 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6141b8a6c3d54e179d6879bb019f4d12', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dcec0431fb6849499d6aec35a234daa6', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81409) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:29:53.278564 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4c3b7e7-24e5-43d6-bdb7-e48b08d5e2b1 service cinder] REQUEST_METHOD: `GET` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:29:53.278913 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4c3b7e7-24e5-43d6-bdb7-e48b08d5e2b1 service cinder] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:29:53.279195 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4c3b7e7-24e5-43d6-bdb7-e48b08d5e2b1 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:29:53.289924 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b4c3b7e7-24e5-43d6-bdb7-e48b08d5e2b1 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:29:53.291392 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b4c3b7e7-24e5-43d6-bdb7-e48b08d5e2b1 service cinder] RBAC: Authorization granted {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:29:53.317445 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 1307/2582] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Oct 13 04:29:53 2022] GET /identity/v3/auth/tokens => generated 3266 bytes in 131 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 13 04:29:57.138100 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b6c812a-0dda-4b96-a7bb-6508cb8786be None None] REQUEST_METHOD: `POST` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:29:57.138689 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b6c812a-0dda-4b96-a7bb-6508cb8786be None None] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:29:57.138689 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b6c812a-0dda-4b96-a7bb-6508cb8786be None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:29:57.174294 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.auth.core [None req-0b6c812a-0dda-4b96-a7bb-6508cb8786be None None] MFA Rules not processed for user `e37c37daa5dc4119b0e36508856719d2`. Rule list: `[]` (Enabled: `True`). {{(pid=81408) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 13 04:29:57.176515 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.fernet_utils [None req-0b6c812a-0dda-4b96-a7bb-6508cb8786be None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81408) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:29:57.251598 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG dbcounter [-] [81408] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81408) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:29:57.255763 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 1276/2583] 10.222.0.27 () {58 vars in 958 bytes} [Thu Oct 13 04:29:57 2022] POST /identity/v3/auth/tokens => generated 3321 bytes in 119 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 13 04:29:57.280158 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b4c3b7e7-24e5-43d6-bdb7-e48b08d5e2b1 service cinder] Authenticating user token {{(pid=81409) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:29:57.281664 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.fernet_utils [None req-b4c3b7e7-24e5-43d6-bdb7-e48b08d5e2b1 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81409) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:29:57.311661 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG dbcounter [-] [81409] Writing DB stats keystone:SELECT=90,keystone:DELETE=7,keystone:INSERT=3 {{(pid=81409) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:29:57.392504 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b4c3b7e7-24e5-43d6-bdb7-e48b08d5e2b1 service cinder] Validating token access rules against request {{(pid=81409) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:29:57.398166 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a0560bfe-1c9d-4c39-951d-2c2fbde134b9 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cb740ed7f7684b079e45f887be83f795', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dcec0431fb6849499d6aec35a234daa6', 'project_domain_id': 'default', 'roles': ['ResellerAdmin', 'admin', 'reader', 'service', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81409) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:29:57.398800 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0560bfe-1c9d-4c39-951d-2c2fbde134b9 service nova] REQUEST_METHOD: `GET` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:29:57.399074 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0560bfe-1c9d-4c39-951d-2c2fbde134b9 service nova] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:29:57.399346 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0560bfe-1c9d-4c39-951d-2c2fbde134b9 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:29:57.401117 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.fernet_utils [None req-a0560bfe-1c9d-4c39-951d-2c2fbde134b9 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81409) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:29:57.465267 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a0560bfe-1c9d-4c39-951d-2c2fbde134b9 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:29:57.466900 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a0560bfe-1c9d-4c39-951d-2c2fbde134b9 service nova] RBAC: Authorization granted {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:29:57.476495 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 1308/2584] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Oct 13 04:29:57 2022] GET /identity/v3/auth/tokens => generated 3256 bytes in 197 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 13 04:30:07.472330 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG dbcounter [-] [81409] Writing DB stats keystone:SELECT=57 {{(pid=81409) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:30:07.728830 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0b6c812a-0dda-4b96-a7bb-6508cb8786be None None] Authenticating user token {{(pid=81408) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:30:07.731241 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.fernet_utils [None req-0b6c812a-0dda-4b96-a7bb-6508cb8786be None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81408) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:30:07.836928 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0b6c812a-0dda-4b96-a7bb-6508cb8786be None None] Validating token access rules against request {{(pid=81408) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:30:07.844571 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-138d6069-ed24-4229-80fb-0c1c18412565 service placement] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ba1b6e5868a549a99760ebe6068fd6b9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dcec0431fb6849499d6aec35a234daa6', 'project_domain_id': 'default', 'roles': ['service', 'member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81408) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:30:07.844972 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-138d6069-ed24-4229-80fb-0c1c18412565 service placement] REQUEST_METHOD: `GET` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:30:07.845249 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-138d6069-ed24-4229-80fb-0c1c18412565 service placement] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:30:07.845527 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-138d6069-ed24-4229-80fb-0c1c18412565 service placement] PATH_INFO: `/v3/auth/tokens` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:30:07.848763 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.fernet_utils [None req-138d6069-ed24-4229-80fb-0c1c18412565 service placement] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81408) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:30:07.861784 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-138d6069-ed24-4229-80fb-0c1c18412565 service placement] RBAC: Authorizing `identity:validate_token()` {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:30:07.863429 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-138d6069-ed24-4229-80fb-0c1c18412565 service placement] RBAC: Authorization granted {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:30:07.875884 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 1277/2585] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Oct 13 04:30:07 2022] GET /identity/v3/auth/tokens => generated 3308 bytes in 148 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Oct 13 04:30:16.184545 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a0560bfe-1c9d-4c39-951d-2c2fbde134b9 service nova] Authenticating user token {{(pid=81409) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:30:16.267785 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a0560bfe-1c9d-4c39-951d-2c2fbde134b9 service nova] Validating token access rules against request {{(pid=81409) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:30:16.272682 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8487cdbe-e0a6-43ad-8132-e32ccd023d5a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8ba823f313144da3b683bacef8fd62e0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81409) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:30:16.273335 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8487cdbe-e0a6-43ad-8132-e32ccd023d5a admin admin] REQUEST_METHOD: `DELETE` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:30:16.273598 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8487cdbe-e0a6-43ad-8132-e32ccd023d5a admin admin] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:30:16.278342 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8487cdbe-e0a6-43ad-8132-e32ccd023d5a admin admin] PATH_INFO: `/v3/users/c94d7ebb1fdf4e36b5e7425f00179a1f` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:30:16.280080 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8487cdbe-e0a6-43ad-8132-e32ccd023d5a admin admin] RBAC: Authorizing `identity:delete_user(user_id=c94d7ebb1fdf4e36b5e7425f00179a1f)` {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:30:16.282082 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8487cdbe-e0a6-43ad-8132-e32ccd023d5a admin admin] RBAC: Authorization granted {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:30:16.335166 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.notifications [None req-8487cdbe-e0a6-43ad-8132-e32ccd023d5a admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'c94d7ebb1fdf4e36b5e7425f00179a1f'} {{(pid=81409) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 13 04:30:16.335990 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.notifications [None req-8487cdbe-e0a6-43ad-8132-e32ccd023d5a admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'c94d7ebb1fdf4e36b5e7425f00179a1f'} {{(pid=81409) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 13 04:30:16.346934 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.notifications [None req-8487cdbe-e0a6-43ad-8132-e32ccd023d5a admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'c94d7ebb1fdf4e36b5e7425f00179a1f'} {{(pid=81409) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 13 04:30:16.347768 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.notifications [None req-8487cdbe-e0a6-43ad-8132-e32ccd023d5a admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'c94d7ebb1fdf4e36b5e7425f00179a1f'} {{(pid=81409) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 13 04:30:16.356291 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.notifications [None req-8487cdbe-e0a6-43ad-8132-e32ccd023d5a admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'c94d7ebb1fdf4e36b5e7425f00179a1f'} {{(pid=81409) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 13 04:30:16.385384 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 1309/2586] 10.222.0.27 () {58 vars in 1224 bytes} [Thu Oct 13 04:30:16 2022] DELETE /identity/v3/users/c94d7ebb1fdf4e36b5e7425f00179a1f => generated 0 bytes in 201 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 13 04:30:16.390932 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8569cdb-ed52-42fa-93d7-4a5e03e78a70 None None] REQUEST_METHOD: `POST` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:30:16.391433 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8569cdb-ed52-42fa-93d7-4a5e03e78a70 None None] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:30:16.391639 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8569cdb-ed52-42fa-93d7-4a5e03e78a70 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:30:16.412001 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.auth.core [None req-a8569cdb-ed52-42fa-93d7-4a5e03e78a70 None None] MFA Rules not processed for user `0e2d979563d6421995451a0e86bf3022`. Rule list: `[]` (Enabled: `True`). {{(pid=81408) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 13 04:30:16.413208 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.fernet_utils [None req-a8569cdb-ed52-42fa-93d7-4a5e03e78a70 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81408) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:30:16.480139 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 1278/2587] 10.222.0.27 () {58 vars in 958 bytes} [Thu Oct 13 04:30:16 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 90 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 13 04:30:16.491001 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8487cdbe-e0a6-43ad-8132-e32ccd023d5a admin admin] Authenticating user token {{(pid=81409) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:30:16.492368 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.fernet_utils [None req-8487cdbe-e0a6-43ad-8132-e32ccd023d5a admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81409) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:30:16.553512 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8487cdbe-e0a6-43ad-8132-e32ccd023d5a admin admin] Validating token access rules against request {{(pid=81409) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:30:16.557297 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7f3686c2-801b-4e5b-b5ea-ee6d58a1faad service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd8812c3974f84871bb2d761cf0016257', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dcec0431fb6849499d6aec35a234daa6', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81409) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:30:16.557901 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f3686c2-801b-4e5b-b5ea-ee6d58a1faad service neutron] REQUEST_METHOD: `GET` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:30:16.558170 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f3686c2-801b-4e5b-b5ea-ee6d58a1faad service neutron] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:30:16.558488 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f3686c2-801b-4e5b-b5ea-ee6d58a1faad service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:30:16.568362 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7f3686c2-801b-4e5b-b5ea-ee6d58a1faad service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:30:16.569764 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7f3686c2-801b-4e5b-b5ea-ee6d58a1faad service neutron] RBAC: Authorization granted {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:30:16.580272 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 1310/2588] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Oct 13 04:30:16 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 90 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 13 04:30:16.875917 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a8569cdb-ed52-42fa-93d7-4a5e03e78a70 None None] Authenticating user token {{(pid=81408) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:30:16.877802 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.fernet_utils [None req-a8569cdb-ed52-42fa-93d7-4a5e03e78a70 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81408) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:30:16.895912 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a8569cdb-ed52-42fa-93d7-4a5e03e78a70 None None] Validating token access rules against request {{(pid=81408) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:30:16.900945 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-77212fe3-ac5c-4635-a536-f0cf4c78d7f4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8ba823f313144da3b683bacef8fd62e0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81408) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:30:16.902525 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77212fe3-ac5c-4635-a536-f0cf4c78d7f4 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:30:16.902815 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77212fe3-ac5c-4635-a536-f0cf4c78d7f4 admin admin] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:30:16.903088 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77212fe3-ac5c-4635-a536-f0cf4c78d7f4 admin admin] PATH_INFO: `/v3/projects/97397324a0194216bf0a330fa631b84e` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:30:16.904316 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-77212fe3-ac5c-4635-a536-f0cf4c78d7f4 admin admin] RBAC: Authorizing `identity:delete_project(project_id=97397324a0194216bf0a330fa631b84e)` {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:30:16.906508 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-77212fe3-ac5c-4635-a536-f0cf4c78d7f4 admin admin] RBAC: Authorization granted {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:30:16.955841 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.notifications [None req-77212fe3-ac5c-4635-a536-f0cf4c78d7f4 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '97397324a0194216bf0a330fa631b84e', 'request_id': 'req-77212fe3-ac5c-4635-a536-f0cf4c78d7f4'} {{(pid=81408) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 13 04:30:16.962687 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.notifications [None req-77212fe3-ac5c-4635-a536-f0cf4c78d7f4 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '97397324a0194216bf0a330fa631b84e', 'request_id': 'req-77212fe3-ac5c-4635-a536-f0cf4c78d7f4'} {{(pid=81408) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 13 04:30:16.969908 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.notifications [None req-77212fe3-ac5c-4635-a536-f0cf4c78d7f4 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '97397324a0194216bf0a330fa631b84e', 'request_id': 'req-77212fe3-ac5c-4635-a536-f0cf4c78d7f4'} {{(pid=81408) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 13 04:30:16.973750 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG dbcounter [-] [81408] Writing DB stats keystone:SELECT=93,keystone:DELETE=6,keystone:INSERT=1 {{(pid=81408) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:30:16.981752 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.notifications [None req-77212fe3-ac5c-4635-a536-f0cf4c78d7f4 admin admin] The token cache is being invalidate because project 97397324a0194216bf0a330fa631b84e was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81408) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 13 04:30:16.982339 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.notifications [None req-77212fe3-ac5c-4635-a536-f0cf4c78d7f4 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81408) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 13 04:30:16.983266 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.notifications [None req-77212fe3-ac5c-4635-a536-f0cf4c78d7f4 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81408) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 13 04:30:16.984795 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 1279/2589] 10.222.0.27 () {58 vars in 1233 bytes} [Thu Oct 13 04:30:16 2022] DELETE /identity/v3/projects/97397324a0194216bf0a330fa631b84e => generated 0 bytes in 111 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 13 04:30:23.365431 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a869219f-6aa4-4a2e-8066-fa707a6cdb19 None None] REQUEST_METHOD: `POST` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:30:23.365974 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a869219f-6aa4-4a2e-8066-fa707a6cdb19 None None] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:30:23.366108 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a869219f-6aa4-4a2e-8066-fa707a6cdb19 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:30:23.386253 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.auth.core [None req-a869219f-6aa4-4a2e-8066-fa707a6cdb19 None None] MFA Rules not processed for user `4cd203fe04644ace8be512ea2c455774`. Rule list: `[]` (Enabled: `True`). {{(pid=81409) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 13 04:30:23.387469 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.fernet_utils [None req-a869219f-6aa4-4a2e-8066-fa707a6cdb19 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81409) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:30:23.404355 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG dbcounter [-] [81409] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81409) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:30:23.445750 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 1311/2590] 10.222.0.27 () {58 vars in 958 bytes} [Thu Oct 13 04:30:23 2022] POST /identity/v3/auth/tokens => generated 3256 bytes in 81 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 13 04:30:23.765877 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-77212fe3-ac5c-4635-a536-f0cf4c78d7f4 admin admin] Authenticating user token {{(pid=81408) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:30:23.767310 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.fernet_utils [None req-77212fe3-ac5c-4635-a536-f0cf4c78d7f4 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81408) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:30:23.823924 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-77212fe3-ac5c-4635-a536-f0cf4c78d7f4 admin admin] Validating token access rules against request {{(pid=81408) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:30:23.828140 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3fcce4fc-82fb-41bc-b30f-18db9401a414 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd8812c3974f84871bb2d761cf0016257', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dcec0431fb6849499d6aec35a234daa6', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81408) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:30:23.828909 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3fcce4fc-82fb-41bc-b30f-18db9401a414 service neutron] REQUEST_METHOD: `GET` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:30:23.829185 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3fcce4fc-82fb-41bc-b30f-18db9401a414 service neutron] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:30:23.829455 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3fcce4fc-82fb-41bc-b30f-18db9401a414 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:30:23.831173 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.fernet_utils [None req-3fcce4fc-82fb-41bc-b30f-18db9401a414 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81408) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:30:23.840834 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3fcce4fc-82fb-41bc-b30f-18db9401a414 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:30:23.842277 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3fcce4fc-82fb-41bc-b30f-18db9401a414 service neutron] RBAC: Authorization granted {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:30:23.850923 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 1280/2591] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Oct 13 04:30:23 2022] GET /identity/v3/auth/tokens => generated 3121 bytes in 86 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 13 04:30:26.213785 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18264201-0c6c-4731-9122-a5ef8804f911 None None] REQUEST_METHOD: `POST` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:30:26.214195 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18264201-0c6c-4731-9122-a5ef8804f911 None None] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:30:26.214363 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18264201-0c6c-4731-9122-a5ef8804f911 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:30:26.234219 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.auth.core [None req-18264201-0c6c-4731-9122-a5ef8804f911 None None] MFA Rules not processed for user `4efdcf15e0e448e2a8d651555f4de29a`. Rule list: `[]` (Enabled: `True`). {{(pid=81409) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 13 04:30:26.235827 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.fernet_utils [None req-18264201-0c6c-4731-9122-a5ef8804f911 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81409) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:30:26.290600 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 1312/2592] 10.222.0.27 () {58 vars in 958 bytes} [Thu Oct 13 04:30:26 2022] POST /identity/v3/auth/tokens => generated 3260 bytes in 78 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 13 04:30:33.847349 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG dbcounter [-] [81408] Writing DB stats keystone:DELETE=1,keystone:SELECT=28 {{(pid=81408) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:30:36.284179 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG dbcounter [-] [81409] Writing DB stats keystone:SELECT=40 {{(pid=81409) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:30:40.486250 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3fcce4fc-82fb-41bc-b30f-18db9401a414 service neutron] Authenticating user token {{(pid=81408) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:30:40.487726 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.fernet_utils [None req-3fcce4fc-82fb-41bc-b30f-18db9401a414 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81408) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:30:40.497952 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3fcce4fc-82fb-41bc-b30f-18db9401a414 service neutron] Validating token access rules against request {{(pid=81408) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:30:40.502118 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5eafbb35-34fe-40c8-9b4d-a5192f05c2b2 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd8812c3974f84871bb2d761cf0016257', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dcec0431fb6849499d6aec35a234daa6', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81408) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:30:40.502736 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5eafbb35-34fe-40c8-9b4d-a5192f05c2b2 service neutron] REQUEST_METHOD: `GET` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:30:40.503008 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5eafbb35-34fe-40c8-9b4d-a5192f05c2b2 service neutron] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:30:40.503275 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5eafbb35-34fe-40c8-9b4d-a5192f05c2b2 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:30:40.504921 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.fernet_utils [None req-5eafbb35-34fe-40c8-9b4d-a5192f05c2b2 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81408) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:30:40.565573 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5eafbb35-34fe-40c8-9b4d-a5192f05c2b2 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:30:40.567142 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5eafbb35-34fe-40c8-9b4d-a5192f05c2b2 service neutron] RBAC: Authorization granted {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:30:40.576478 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 1281/2593] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Oct 13 04:30:40 2022] GET /identity/v3/auth/tokens => generated 3321 bytes in 91 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 13 04:30:50.572229 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG dbcounter [-] [81408] Writing DB stats keystone:SELECT=30 {{(pid=81408) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:30:56.361243 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-18264201-0c6c-4731-9122-a5ef8804f911 None None] Authenticating user token {{(pid=81409) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:30:56.362671 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.fernet_utils [None req-18264201-0c6c-4731-9122-a5ef8804f911 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81409) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:30:56.419696 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-18264201-0c6c-4731-9122-a5ef8804f911 None None] Validating token access rules against request {{(pid=81409) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:30:56.423413 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3720f3ca-59aa-4409-99be-334f1afb9dc1 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6141b8a6c3d54e179d6879bb019f4d12', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dcec0431fb6849499d6aec35a234daa6', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81409) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:30:56.424018 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3720f3ca-59aa-4409-99be-334f1afb9dc1 service cinder] REQUEST_METHOD: `GET` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:30:56.424296 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3720f3ca-59aa-4409-99be-334f1afb9dc1 service cinder] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:30:56.424568 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3720f3ca-59aa-4409-99be-334f1afb9dc1 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:30:56.426303 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.fernet_utils [None req-3720f3ca-59aa-4409-99be-334f1afb9dc1 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81409) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:30:56.496394 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3720f3ca-59aa-4409-99be-334f1afb9dc1 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:30:56.497795 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3720f3ca-59aa-4409-99be-334f1afb9dc1 service cinder] RBAC: Authorization granted {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:30:56.509262 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 1313/2594] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Oct 13 04:30:56 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 149 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 13 04:30:59.103347 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5eafbb35-34fe-40c8-9b4d-a5192f05c2b2 service neutron] Authenticating user token {{(pid=81408) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:30:59.112875 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5eafbb35-34fe-40c8-9b4d-a5192f05c2b2 service neutron] Validating token access rules against request {{(pid=81408) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:30:59.116419 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-57bdab4b-18a1-4c4c-ba35-c6bf03c57003 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd8812c3974f84871bb2d761cf0016257', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dcec0431fb6849499d6aec35a234daa6', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81408) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:30:59.117001 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57bdab4b-18a1-4c4c-ba35-c6bf03c57003 service neutron] REQUEST_METHOD: `GET` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:30:59.117274 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57bdab4b-18a1-4c4c-ba35-c6bf03c57003 service neutron] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:30:59.117539 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57bdab4b-18a1-4c4c-ba35-c6bf03c57003 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:30:59.119253 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.fernet_utils [None req-57bdab4b-18a1-4c4c-ba35-c6bf03c57003 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81408) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:30:59.133142 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-57bdab4b-18a1-4c4c-ba35-c6bf03c57003 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:30:59.134580 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-57bdab4b-18a1-4c4c-ba35-c6bf03c57003 service neutron] RBAC: Authorization granted {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:30:59.145998 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 1282/2595] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Oct 13 04:30:59 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 43 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 13 04:31:06.504335 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG dbcounter [-] [81409] Writing DB stats keystone:SELECT=54 {{(pid=81409) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:31:09.141349 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG dbcounter [-] [81408] Writing DB stats keystone:SELECT=6 {{(pid=81408) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:31:23.540467 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3720f3ca-59aa-4409-99be-334f1afb9dc1 service cinder] Authenticating user token {{(pid=81409) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:31:23.541977 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.fernet_utils [None req-3720f3ca-59aa-4409-99be-334f1afb9dc1 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81409) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:31:23.606101 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3720f3ca-59aa-4409-99be-334f1afb9dc1 service cinder] Validating token access rules against request {{(pid=81409) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:31:23.611303 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-898cd2ef-6854-49c3-b066-e899607091d3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8ba823f313144da3b683bacef8fd62e0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81409) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:31:23.611918 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-898cd2ef-6854-49c3-b066-e899607091d3 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:31:23.612192 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-898cd2ef-6854-49c3-b066-e899607091d3 admin admin] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:31:23.612458 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-898cd2ef-6854-49c3-b066-e899607091d3 admin admin] PATH_INFO: `/v3/users/4efdcf15e0e448e2a8d651555f4de29a` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:31:23.613568 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-898cd2ef-6854-49c3-b066-e899607091d3 admin admin] RBAC: Authorizing `identity:delete_user(user_id=4efdcf15e0e448e2a8d651555f4de29a)` {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:31:23.615499 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-898cd2ef-6854-49c3-b066-e899607091d3 admin admin] RBAC: Authorization granted {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:31:23.661092 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.notifications [None req-898cd2ef-6854-49c3-b066-e899607091d3 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '4efdcf15e0e448e2a8d651555f4de29a'} {{(pid=81409) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 13 04:31:23.661683 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.notifications [None req-898cd2ef-6854-49c3-b066-e899607091d3 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '4efdcf15e0e448e2a8d651555f4de29a'} {{(pid=81409) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 13 04:31:23.668574 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.notifications [None req-898cd2ef-6854-49c3-b066-e899607091d3 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '4efdcf15e0e448e2a8d651555f4de29a'} {{(pid=81409) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 13 04:31:23.669178 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.notifications [None req-898cd2ef-6854-49c3-b066-e899607091d3 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '4efdcf15e0e448e2a8d651555f4de29a'} {{(pid=81409) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 13 04:31:23.677048 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.notifications [None req-898cd2ef-6854-49c3-b066-e899607091d3 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '4efdcf15e0e448e2a8d651555f4de29a'} {{(pid=81409) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 13 04:31:23.695111 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 1314/2596] 10.222.0.27 () {58 vars in 1224 bytes} [Thu Oct 13 04:31:23 2022] DELETE /identity/v3/users/4efdcf15e0e448e2a8d651555f4de29a => generated 0 bytes in 155 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 13 04:31:23.864071 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-57bdab4b-18a1-4c4c-ba35-c6bf03c57003 service neutron] Authenticating user token {{(pid=81408) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:31:23.875021 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-57bdab4b-18a1-4c4c-ba35-c6bf03c57003 service neutron] Validating token access rules against request {{(pid=81408) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:31:23.933469 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-428c459c-70b1-48d6-ad5d-8c00a294de2f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8ba823f313144da3b683bacef8fd62e0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81408) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:31:23.934027 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-428c459c-70b1-48d6-ad5d-8c00a294de2f admin admin] REQUEST_METHOD: `DELETE` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:31:23.934345 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-428c459c-70b1-48d6-ad5d-8c00a294de2f admin admin] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:31:23.934618 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-428c459c-70b1-48d6-ad5d-8c00a294de2f admin admin] PATH_INFO: `/v3/projects/7f78c358e5c0423d9ba9520c90f163b5` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:31:23.935733 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-428c459c-70b1-48d6-ad5d-8c00a294de2f admin admin] RBAC: Authorizing `identity:delete_project(project_id=7f78c358e5c0423d9ba9520c90f163b5)` {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:31:23.937826 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-428c459c-70b1-48d6-ad5d-8c00a294de2f admin admin] RBAC: Authorization granted {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:31:23.982508 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.notifications [None req-428c459c-70b1-48d6-ad5d-8c00a294de2f admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '7f78c358e5c0423d9ba9520c90f163b5', 'request_id': 'req-428c459c-70b1-48d6-ad5d-8c00a294de2f'} {{(pid=81408) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 13 04:31:23.990312 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.notifications [None req-428c459c-70b1-48d6-ad5d-8c00a294de2f admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '7f78c358e5c0423d9ba9520c90f163b5', 'request_id': 'req-428c459c-70b1-48d6-ad5d-8c00a294de2f'} {{(pid=81408) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 13 04:31:23.996527 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.notifications [None req-428c459c-70b1-48d6-ad5d-8c00a294de2f admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '7f78c358e5c0423d9ba9520c90f163b5', 'request_id': 'req-428c459c-70b1-48d6-ad5d-8c00a294de2f'} {{(pid=81408) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 13 04:31:24.006827 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.notifications [None req-428c459c-70b1-48d6-ad5d-8c00a294de2f admin admin] The token cache is being invalidate because project 7f78c358e5c0423d9ba9520c90f163b5 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81408) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 13 04:31:24.007112 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.notifications [None req-428c459c-70b1-48d6-ad5d-8c00a294de2f admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81408) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 13 04:31:24.007736 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.notifications [None req-428c459c-70b1-48d6-ad5d-8c00a294de2f admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81408) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 13 04:31:24.009235 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 1283/2597] 10.222.0.27 () {58 vars in 1233 bytes} [Thu Oct 13 04:31:23 2022] DELETE /identity/v3/projects/7f78c358e5c0423d9ba9520c90f163b5 => generated 0 bytes in 146 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 13 04:31:33.691580 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG dbcounter [-] [81409] Writing DB stats keystone:SELECT=53,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81409) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:31:34.003876 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG dbcounter [-] [81408] Writing DB stats keystone:SELECT=51,keystone:DELETE=7,keystone:INSERT=1 {{(pid=81408) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:32:15.647334 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-898cd2ef-6854-49c3-b066-e899607091d3 admin admin] Authenticating user token {{(pid=81409) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:32:15.648831 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.fernet_utils [None req-898cd2ef-6854-49c3-b066-e899607091d3 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81409) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:32:15.706387 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-898cd2ef-6854-49c3-b066-e899607091d3 admin admin] Validating token access rules against request {{(pid=81409) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:32:15.753000 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ae0d090a-561d-4cf4-b740-fc526443267f service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd8812c3974f84871bb2d761cf0016257', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dcec0431fb6849499d6aec35a234daa6', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81409) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:32:15.753595 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae0d090a-561d-4cf4-b740-fc526443267f service neutron] REQUEST_METHOD: `GET` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:32:15.753871 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae0d090a-561d-4cf4-b740-fc526443267f service neutron] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:32:15.754139 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae0d090a-561d-4cf4-b740-fc526443267f service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:32:15.755891 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.fernet_utils [None req-ae0d090a-561d-4cf4-b740-fc526443267f service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81409) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:32:15.765146 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ae0d090a-561d-4cf4-b740-fc526443267f service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:32:15.766587 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ae0d090a-561d-4cf4-b740-fc526443267f service neutron] RBAC: Authorization granted {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:32:15.775446 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 1315/2598] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Oct 13 04:32:15 2022] GET /identity/v3/auth/tokens => generated 3121 bytes in 129 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 13 04:32:25.771594 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG dbcounter [-] [81409] Writing DB stats keystone:SELECT=46 {{(pid=81409) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:32:49.500358 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-428c459c-70b1-48d6-ad5d-8c00a294de2f admin admin] Authenticating user token {{(pid=81408) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:32:49.502271 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.fernet_utils [None req-428c459c-70b1-48d6-ad5d-8c00a294de2f admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81408) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:32:49.589507 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-428c459c-70b1-48d6-ad5d-8c00a294de2f admin admin] Validating token access rules against request {{(pid=81408) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:32:49.597146 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-22b74ad5-3a92-4ac5-8bcd-9aa134270d9d service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cb740ed7f7684b079e45f887be83f795', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dcec0431fb6849499d6aec35a234daa6', 'project_domain_id': 'default', 'roles': ['ResellerAdmin', 'admin', 'reader', 'service', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81408) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:32:49.597861 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22b74ad5-3a92-4ac5-8bcd-9aa134270d9d service nova] REQUEST_METHOD: `GET` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:32:49.598198 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22b74ad5-3a92-4ac5-8bcd-9aa134270d9d service nova] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:32:49.598617 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22b74ad5-3a92-4ac5-8bcd-9aa134270d9d service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:32:49.600633 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.fernet_utils [None req-22b74ad5-3a92-4ac5-8bcd-9aa134270d9d service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81408) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:32:49.616579 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-22b74ad5-3a92-4ac5-8bcd-9aa134270d9d service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:32:49.618294 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-22b74ad5-3a92-4ac5-8bcd-9aa134270d9d service nova] RBAC: Authorization granted {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:32:49.630297 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 1284/2599] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Oct 13 04:32:49 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 131 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 13 04:32:51.517375 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7e0984f-9dd9-4c50-858e-7fb80a4132f4 None None] REQUEST_METHOD: `POST` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:32:51.517857 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7e0984f-9dd9-4c50-858e-7fb80a4132f4 None None] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:32:51.518106 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7e0984f-9dd9-4c50-858e-7fb80a4132f4 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:32:51.538042 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.auth.core [None req-d7e0984f-9dd9-4c50-858e-7fb80a4132f4 None None] MFA Rules not processed for user `4cd203fe04644ace8be512ea2c455774`. Rule list: `[]` (Enabled: `True`). {{(pid=81409) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 13 04:32:51.539263 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.fernet_utils [None req-d7e0984f-9dd9-4c50-858e-7fb80a4132f4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81409) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:32:51.593799 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 1316/2600] 10.222.0.27 () {58 vars in 958 bytes} [Thu Oct 13 04:32:51 2022] POST /identity/v3/auth/tokens => generated 3256 bytes in 77 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 13 04:32:52.359786 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-22b74ad5-3a92-4ac5-8bcd-9aa134270d9d service nova] Authenticating user token {{(pid=81408) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:32:52.363353 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.fernet_utils [None req-22b74ad5-3a92-4ac5-8bcd-9aa134270d9d service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81408) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:32:52.436193 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-22b74ad5-3a92-4ac5-8bcd-9aa134270d9d service nova] Validating token access rules against request {{(pid=81408) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:32:52.442041 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-429ca23c-9884-4fed-bb1b-e1dab6e43408 service placement] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ba1b6e5868a549a99760ebe6068fd6b9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dcec0431fb6849499d6aec35a234daa6', 'project_domain_id': 'default', 'roles': ['service', 'member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81408) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:32:52.442662 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-429ca23c-9884-4fed-bb1b-e1dab6e43408 service placement] REQUEST_METHOD: `GET` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:32:52.442937 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-429ca23c-9884-4fed-bb1b-e1dab6e43408 service placement] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:32:52.443203 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-429ca23c-9884-4fed-bb1b-e1dab6e43408 service placement] PATH_INFO: `/v3/auth/tokens` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:32:52.445196 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.fernet_utils [None req-429ca23c-9884-4fed-bb1b-e1dab6e43408 service placement] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81408) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:32:52.457307 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-429ca23c-9884-4fed-bb1b-e1dab6e43408 service placement] RBAC: Authorizing `identity:validate_token()` {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:32:52.458897 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-429ca23c-9884-4fed-bb1b-e1dab6e43408 service placement] RBAC: Authorization granted {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:32:52.468607 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 1285/2601] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Oct 13 04:32:52 2022] GET /identity/v3/auth/tokens => generated 3308 bytes in 109 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 13 04:33:01.587070 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG dbcounter [-] [81409] Writing DB stats keystone:SELECT=25 {{(pid=81409) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:33:02.464935 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG dbcounter [-] [81408] Writing DB stats keystone:SELECT=66 {{(pid=81408) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:33:06.486851 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d7e0984f-9dd9-4c50-858e-7fb80a4132f4 None None] Authenticating user token {{(pid=81409) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:33:06.488529 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.fernet_utils [None req-d7e0984f-9dd9-4c50-858e-7fb80a4132f4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81409) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:33:06.509670 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d7e0984f-9dd9-4c50-858e-7fb80a4132f4 None None] Validating token access rules against request {{(pid=81409) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:33:06.515976 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2d9ce51b-3ad0-452c-9f91-950c1b3c1f0a service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cb740ed7f7684b079e45f887be83f795', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dcec0431fb6849499d6aec35a234daa6', 'project_domain_id': 'default', 'roles': ['ResellerAdmin', 'admin', 'reader', 'service', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81409) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:33:06.516821 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d9ce51b-3ad0-452c-9f91-950c1b3c1f0a service nova] REQUEST_METHOD: `GET` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:33:06.517199 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d9ce51b-3ad0-452c-9f91-950c1b3c1f0a service nova] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:33:06.517708 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d9ce51b-3ad0-452c-9f91-950c1b3c1f0a service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:33:06.520803 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.fernet_utils [None req-2d9ce51b-3ad0-452c-9f91-950c1b3c1f0a service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81409) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:33:06.535600 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2d9ce51b-3ad0-452c-9f91-950c1b3c1f0a service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:33:06.537599 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2d9ce51b-3ad0-452c-9f91-950c1b3c1f0a service nova] RBAC: Authorization granted {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:33:06.549800 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 1317/2602] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Oct 13 04:33:06 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 63 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 13 04:33:16.544540 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG dbcounter [-] [81409] Writing DB stats keystone:SELECT=6 {{(pid=81409) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:34:43.885527 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba48d63d-6ea2-4867-a4dc-e9b5ab04feca None None] REQUEST_METHOD: `POST` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:34:43.886071 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba48d63d-6ea2-4867-a4dc-e9b5ab04feca None None] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:34:43.886071 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba48d63d-6ea2-4867-a4dc-e9b5ab04feca None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:34:43.906523 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.auth.core [None req-ba48d63d-6ea2-4867-a4dc-e9b5ab04feca None None] MFA Rules not processed for user `4cd203fe04644ace8be512ea2c455774`. Rule list: `[]` (Enabled: `True`). {{(pid=81408) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 13 04:34:43.907728 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.fernet_utils [None req-ba48d63d-6ea2-4867-a4dc-e9b5ab04feca None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81408) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:34:43.914555 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 1286/2603] 10.222.0.27 () {58 vars in 958 bytes} [Thu Oct 13 04:34:43 2022] POST /identity/v3/auth/tokens => generated 3256 bytes in 30 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 13 04:34:53.900387 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG dbcounter [-] [81408] Writing DB stats keystone:SELECT=3 {{(pid=81408) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:34:58.582805 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2d9ce51b-3ad0-452c-9f91-950c1b3c1f0a service nova] Authenticating user token {{(pid=81409) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:34:58.595006 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2d9ce51b-3ad0-452c-9f91-950c1b3c1f0a service nova] Validating token access rules against request {{(pid=81409) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:34:58.600702 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-09f2b0b3-e9d6-4322-9564-fd7b2912b4c8 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cb740ed7f7684b079e45f887be83f795', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dcec0431fb6849499d6aec35a234daa6', 'project_domain_id': 'default', 'roles': ['ResellerAdmin', 'admin', 'reader', 'service', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81409) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:34:58.601238 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09f2b0b3-e9d6-4322-9564-fd7b2912b4c8 service nova] REQUEST_METHOD: `GET` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:34:58.601509 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09f2b0b3-e9d6-4322-9564-fd7b2912b4c8 service nova] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:34:58.601775 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09f2b0b3-e9d6-4322-9564-fd7b2912b4c8 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:34:58.603498 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.fernet_utils [None req-09f2b0b3-e9d6-4322-9564-fd7b2912b4c8 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81409) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:34:58.614067 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-09f2b0b3-e9d6-4322-9564-fd7b2912b4c8 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:34:58.615661 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-09f2b0b3-e9d6-4322-9564-fd7b2912b4c8 service nova] RBAC: Authorization granted {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:34:58.624298 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 1318/2604] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Oct 13 04:34:58 2022] GET /identity/v3/auth/tokens => generated 3256 bytes in 42 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 13 04:35:08.621038 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG dbcounter [-] [81409] Writing DB stats keystone:SELECT=6 {{(pid=81409) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:35:13.012532 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ba48d63d-6ea2-4867-a4dc-e9b5ab04feca None None] Authenticating user token {{(pid=81408) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:35:13.013972 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.fernet_utils [None req-ba48d63d-6ea2-4867-a4dc-e9b5ab04feca None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81408) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:35:13.029617 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ba48d63d-6ea2-4867-a4dc-e9b5ab04feca None None] Validating token access rules against request {{(pid=81408) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:35:13.034622 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2837e06b-8d41-47ea-974b-2c3d7fa336a5 service placement] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ba1b6e5868a549a99760ebe6068fd6b9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dcec0431fb6849499d6aec35a234daa6', 'project_domain_id': 'default', 'roles': ['service', 'member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81408) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:35:13.035218 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2837e06b-8d41-47ea-974b-2c3d7fa336a5 service placement] REQUEST_METHOD: `GET` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:35:13.035495 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2837e06b-8d41-47ea-974b-2c3d7fa336a5 service placement] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:35:13.035770 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2837e06b-8d41-47ea-974b-2c3d7fa336a5 service placement] PATH_INFO: `/v3/auth/tokens` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:35:13.037441 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.fernet_utils [None req-2837e06b-8d41-47ea-974b-2c3d7fa336a5 service placement] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81408) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:35:13.049064 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2837e06b-8d41-47ea-974b-2c3d7fa336a5 service placement] RBAC: Authorizing `identity:validate_token()` {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:35:13.050670 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2837e06b-8d41-47ea-974b-2c3d7fa336a5 service placement] RBAC: Authorization granted {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:35:13.061293 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 1287/2605] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Oct 13 04:35:13 2022] GET /identity/v3/auth/tokens => generated 3308 bytes in 49 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 13 04:35:23.056371 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG dbcounter [-] [81408] Writing DB stats keystone:SELECT=6 {{(pid=81408) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:36:42.640995 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-09f2b0b3-e9d6-4322-9564-fd7b2912b4c8 service nova] Authenticating user token {{(pid=81409) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:36:42.642332 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.fernet_utils [None req-09f2b0b3-e9d6-4322-9564-fd7b2912b4c8 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81409) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:36:42.654678 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-09f2b0b3-e9d6-4322-9564-fd7b2912b4c8 service nova] Validating token access rules against request {{(pid=81409) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:36:42.658512 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c0141c3a-f5a7-49d8-81cd-283aeade9783 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd8812c3974f84871bb2d761cf0016257', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dcec0431fb6849499d6aec35a234daa6', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81409) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:36:42.659204 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0141c3a-f5a7-49d8-81cd-283aeade9783 service neutron] REQUEST_METHOD: `GET` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:36:42.659688 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0141c3a-f5a7-49d8-81cd-283aeade9783 service neutron] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:36:42.660035 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0141c3a-f5a7-49d8-81cd-283aeade9783 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:36:42.661919 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.fernet_utils [None req-c0141c3a-f5a7-49d8-81cd-283aeade9783 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81409) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:36:42.677249 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c0141c3a-f5a7-49d8-81cd-283aeade9783 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:36:42.679085 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c0141c3a-f5a7-49d8-81cd-283aeade9783 service neutron] RBAC: Authorization granted {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:36:42.691355 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 1319/2606] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Oct 13 04:36:42 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 51 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 13 04:36:43.879322 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2837e06b-8d41-47ea-974b-2c3d7fa336a5 service placement] Authenticating user token {{(pid=81408) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:36:43.880924 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.fernet_utils [None req-2837e06b-8d41-47ea-974b-2c3d7fa336a5 service placement] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81408) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:36:43.893051 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2837e06b-8d41-47ea-974b-2c3d7fa336a5 service placement] Validating token access rules against request {{(pid=81408) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:36:43.896748 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f2852038-33e6-47f5-a854-42733a92f329 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd8812c3974f84871bb2d761cf0016257', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dcec0431fb6849499d6aec35a234daa6', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81408) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:36:43.897356 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2852038-33e6-47f5-a854-42733a92f329 service neutron] REQUEST_METHOD: `GET` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:36:43.897629 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2852038-33e6-47f5-a854-42733a92f329 service neutron] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:36:43.897895 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2852038-33e6-47f5-a854-42733a92f329 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:36:43.899838 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.fernet_utils [None req-f2852038-33e6-47f5-a854-42733a92f329 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81408) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:36:43.914732 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f2852038-33e6-47f5-a854-42733a92f329 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:36:43.916153 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f2852038-33e6-47f5-a854-42733a92f329 service neutron] RBAC: Authorization granted {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:36:43.928014 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 1288/2607] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Oct 13 04:36:43 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 49 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Oct 13 04:36:52.686245 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG dbcounter [-] [81409] Writing DB stats keystone:SELECT=6 {{(pid=81409) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:36:53.923096 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG dbcounter [-] [81408] Writing DB stats keystone:SELECT=6 {{(pid=81408) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:36:54.811787 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c0141c3a-f5a7-49d8-81cd-283aeade9783 service neutron] Authenticating user token {{(pid=81409) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:36:54.820453 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c0141c3a-f5a7-49d8-81cd-283aeade9783 service neutron] Validating token access rules against request {{(pid=81409) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:36:54.824917 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-acc614a9-3986-4640-8a93-9d630cc1d04a service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd8812c3974f84871bb2d761cf0016257', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dcec0431fb6849499d6aec35a234daa6', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81409) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:36:54.825507 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-acc614a9-3986-4640-8a93-9d630cc1d04a service neutron] REQUEST_METHOD: `GET` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:36:54.825779 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-acc614a9-3986-4640-8a93-9d630cc1d04a service neutron] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:36:54.826044 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-acc614a9-3986-4640-8a93-9d630cc1d04a service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:36:54.827771 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.fernet_utils [None req-acc614a9-3986-4640-8a93-9d630cc1d04a service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81409) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:36:54.895009 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-acc614a9-3986-4640-8a93-9d630cc1d04a service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:36:54.896660 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-acc614a9-3986-4640-8a93-9d630cc1d04a service neutron] RBAC: Authorization granted {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:36:54.909700 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 1320/2608] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Oct 13 04:36:54 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 98 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 13 04:36:58.006293 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f2852038-33e6-47f5-a854-42733a92f329 service neutron] Authenticating user token {{(pid=81408) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:36:58.030240 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f2852038-33e6-47f5-a854-42733a92f329 service neutron] Validating token access rules against request {{(pid=81408) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:36:58.036675 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a38a2615-0a76-4e5b-9cb3-01016646d943 service placement] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ba1b6e5868a549a99760ebe6068fd6b9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dcec0431fb6849499d6aec35a234daa6', 'project_domain_id': 'default', 'roles': ['service', 'member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81408) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:36:58.037488 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a38a2615-0a76-4e5b-9cb3-01016646d943 service placement] REQUEST_METHOD: `GET` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:36:58.037703 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a38a2615-0a76-4e5b-9cb3-01016646d943 service placement] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:36:58.037915 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a38a2615-0a76-4e5b-9cb3-01016646d943 service placement] PATH_INFO: `/v3/auth/tokens` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:36:58.039657 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.fernet_utils [None req-a38a2615-0a76-4e5b-9cb3-01016646d943 service placement] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81408) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:36:58.057885 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a38a2615-0a76-4e5b-9cb3-01016646d943 service placement] RBAC: Authorizing `identity:validate_token()` {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:36:58.060384 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a38a2615-0a76-4e5b-9cb3-01016646d943 service placement] RBAC: Authorization granted {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:36:58.072430 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 1289/2609] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Oct 13 04:36:58 2022] GET /identity/v3/auth/tokens => generated 3308 bytes in 67 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 13 04:37:01.233880 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-acc614a9-3986-4640-8a93-9d630cc1d04a service neutron] Authenticating user token {{(pid=81409) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:37:01.235702 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.fernet_utils [None req-acc614a9-3986-4640-8a93-9d630cc1d04a service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81409) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:37:01.247804 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-acc614a9-3986-4640-8a93-9d630cc1d04a service neutron] Validating token access rules against request {{(pid=81409) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:37:01.252398 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aec7970f-5b66-49c8-97ac-d8d3959fe4b4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8ba823f313144da3b683bacef8fd62e0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81409) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:37:01.253018 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aec7970f-5b66-49c8-97ac-d8d3959fe4b4 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:37:01.253294 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aec7970f-5b66-49c8-97ac-d8d3959fe4b4 admin admin] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:37:01.253559 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aec7970f-5b66-49c8-97ac-d8d3959fe4b4 admin admin] PATH_INFO: `/v3/users/4cd203fe04644ace8be512ea2c455774` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:37:01.254679 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aec7970f-5b66-49c8-97ac-d8d3959fe4b4 admin admin] RBAC: Authorizing `identity:delete_user(user_id=4cd203fe04644ace8be512ea2c455774)` {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:37:01.256576 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aec7970f-5b66-49c8-97ac-d8d3959fe4b4 admin admin] RBAC: Authorization granted {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:37:01.303017 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.notifications [None req-aec7970f-5b66-49c8-97ac-d8d3959fe4b4 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '4cd203fe04644ace8be512ea2c455774'} {{(pid=81409) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 13 04:37:01.303658 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.notifications [None req-aec7970f-5b66-49c8-97ac-d8d3959fe4b4 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '4cd203fe04644ace8be512ea2c455774'} {{(pid=81409) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 13 04:37:01.310558 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.notifications [None req-aec7970f-5b66-49c8-97ac-d8d3959fe4b4 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '4cd203fe04644ace8be512ea2c455774'} {{(pid=81409) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 13 04:37:01.311122 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.notifications [None req-aec7970f-5b66-49c8-97ac-d8d3959fe4b4 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '4cd203fe04644ace8be512ea2c455774'} {{(pid=81409) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 13 04:37:01.318513 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.notifications [None req-aec7970f-5b66-49c8-97ac-d8d3959fe4b4 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '4cd203fe04644ace8be512ea2c455774'} {{(pid=81409) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 13 04:37:01.337024 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 1321/2610] 10.222.0.27 () {58 vars in 1224 bytes} [Thu Oct 13 04:37:01 2022] DELETE /identity/v3/users/4cd203fe04644ace8be512ea2c455774 => generated 0 bytes in 104 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 13 04:37:01.514400 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a38a2615-0a76-4e5b-9cb3-01016646d943 service placement] Authenticating user token {{(pid=81408) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:37:01.528948 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a38a2615-0a76-4e5b-9cb3-01016646d943 service placement] Validating token access rules against request {{(pid=81408) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:37:01.586160 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7b114ef2-969d-4d71-a857-717fd3818874 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8ba823f313144da3b683bacef8fd62e0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81408) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:37:01.586764 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b114ef2-969d-4d71-a857-717fd3818874 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:37:01.587039 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b114ef2-969d-4d71-a857-717fd3818874 admin admin] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:37:01.587309 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b114ef2-969d-4d71-a857-717fd3818874 admin admin] PATH_INFO: `/v3/projects/eb360c69b8c240408cccc0cfbbb709f1` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:37:01.588470 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7b114ef2-969d-4d71-a857-717fd3818874 admin admin] RBAC: Authorizing `identity:delete_project(project_id=eb360c69b8c240408cccc0cfbbb709f1)` {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:37:01.591126 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7b114ef2-969d-4d71-a857-717fd3818874 admin admin] RBAC: Authorization granted {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:37:01.636818 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.notifications [None req-7b114ef2-969d-4d71-a857-717fd3818874 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'eb360c69b8c240408cccc0cfbbb709f1', 'request_id': 'req-7b114ef2-969d-4d71-a857-717fd3818874'} {{(pid=81408) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 13 04:37:01.643867 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.notifications [None req-7b114ef2-969d-4d71-a857-717fd3818874 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'eb360c69b8c240408cccc0cfbbb709f1', 'request_id': 'req-7b114ef2-969d-4d71-a857-717fd3818874'} {{(pid=81408) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 13 04:37:01.649993 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.notifications [None req-7b114ef2-969d-4d71-a857-717fd3818874 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'eb360c69b8c240408cccc0cfbbb709f1', 'request_id': 'req-7b114ef2-969d-4d71-a857-717fd3818874'} {{(pid=81408) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 13 04:37:01.660789 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.notifications [None req-7b114ef2-969d-4d71-a857-717fd3818874 admin admin] The token cache is being invalidate because project eb360c69b8c240408cccc0cfbbb709f1 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81408) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 13 04:37:01.661172 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.notifications [None req-7b114ef2-969d-4d71-a857-717fd3818874 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81408) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 13 04:37:01.661936 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.notifications [None req-7b114ef2-969d-4d71-a857-717fd3818874 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81408) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 13 04:37:01.663628 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 1290/2611] 10.222.0.27 () {58 vars in 1233 bytes} [Thu Oct 13 04:37:01 2022] DELETE /identity/v3/projects/eb360c69b8c240408cccc0cfbbb709f1 => generated 0 bytes in 150 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 13 04:37:01.667916 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aec7970f-5b66-49c8-97ac-d8d3959fe4b4 admin admin] Authenticating user token {{(pid=81409) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:37:01.669305 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.fernet_utils [None req-aec7970f-5b66-49c8-97ac-d8d3959fe4b4 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81409) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:37:01.682107 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aec7970f-5b66-49c8-97ac-d8d3959fe4b4 admin admin] Validating token access rules against request {{(pid=81409) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:37:01.738308 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a93d6ae9-7909-491f-b6ec-292da9c05fbc admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8ba823f313144da3b683bacef8fd62e0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81409) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:37:01.738926 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a93d6ae9-7909-491f-b6ec-292da9c05fbc admin admin] REQUEST_METHOD: `DELETE` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:37:01.739200 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a93d6ae9-7909-491f-b6ec-292da9c05fbc admin admin] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:37:01.739465 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a93d6ae9-7909-491f-b6ec-292da9c05fbc admin admin] PATH_INFO: `/v3/users/e37c37daa5dc4119b0e36508856719d2` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:37:01.740572 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a93d6ae9-7909-491f-b6ec-292da9c05fbc admin admin] RBAC: Authorizing `identity:delete_user(user_id=e37c37daa5dc4119b0e36508856719d2)` {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:37:01.742496 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a93d6ae9-7909-491f-b6ec-292da9c05fbc admin admin] RBAC: Authorization granted {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:37:01.748350 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG dbcounter [-] [81409] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81409) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:37:01.789507 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.notifications [None req-a93d6ae9-7909-491f-b6ec-292da9c05fbc admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'e37c37daa5dc4119b0e36508856719d2'} {{(pid=81409) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 13 04:37:01.790160 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.notifications [None req-a93d6ae9-7909-491f-b6ec-292da9c05fbc admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'e37c37daa5dc4119b0e36508856719d2'} {{(pid=81409) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 13 04:37:01.798169 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.notifications [None req-a93d6ae9-7909-491f-b6ec-292da9c05fbc admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'e37c37daa5dc4119b0e36508856719d2'} {{(pid=81409) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 13 04:37:01.798820 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.notifications [None req-a93d6ae9-7909-491f-b6ec-292da9c05fbc admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'e37c37daa5dc4119b0e36508856719d2'} {{(pid=81409) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 13 04:37:01.806032 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.notifications [None req-a93d6ae9-7909-491f-b6ec-292da9c05fbc admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'e37c37daa5dc4119b0e36508856719d2'} {{(pid=81409) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 13 04:37:01.825909 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 1322/2612] 10.222.0.27 () {58 vars in 1224 bytes} [Thu Oct 13 04:37:01 2022] DELETE /identity/v3/users/e37c37daa5dc4119b0e36508856719d2 => generated 0 bytes in 159 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 13 04:37:01.998838 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7b114ef2-969d-4d71-a857-717fd3818874 admin admin] Authenticating user token {{(pid=81408) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:37:02.000384 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.fernet_utils [None req-7b114ef2-969d-4d71-a857-717fd3818874 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81408) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:37:02.013289 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7b114ef2-969d-4d71-a857-717fd3818874 admin admin] Validating token access rules against request {{(pid=81408) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:37:02.069810 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5056bc12-e932-4fd6-ba56-22cf14dfd41b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e2d979563d6421995451a0e86bf3022', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8ba823f313144da3b683bacef8fd62e0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81408) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:37:02.070468 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5056bc12-e932-4fd6-ba56-22cf14dfd41b admin admin] REQUEST_METHOD: `DELETE` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:37:02.070740 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5056bc12-e932-4fd6-ba56-22cf14dfd41b admin admin] SCRIPT_NAME: `/identity` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:37:02.071009 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5056bc12-e932-4fd6-ba56-22cf14dfd41b admin admin] PATH_INFO: `/v3/projects/372da12f356748f28a1261ee8b643e8d` {{(pid=81408) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:37:02.072101 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5056bc12-e932-4fd6-ba56-22cf14dfd41b admin admin] RBAC: Authorizing `identity:delete_project(project_id=372da12f356748f28a1261ee8b643e8d)` {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:37:02.074113 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5056bc12-e932-4fd6-ba56-22cf14dfd41b admin admin] RBAC: Authorization granted {{(pid=81408) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:37:02.096282 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG dbcounter [-] [81408] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=81408) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:37:02.123301 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.notifications [None req-5056bc12-e932-4fd6-ba56-22cf14dfd41b admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '372da12f356748f28a1261ee8b643e8d', 'request_id': 'req-5056bc12-e932-4fd6-ba56-22cf14dfd41b'} {{(pid=81408) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 13 04:37:02.130062 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.notifications [None req-5056bc12-e932-4fd6-ba56-22cf14dfd41b admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '372da12f356748f28a1261ee8b643e8d', 'request_id': 'req-5056bc12-e932-4fd6-ba56-22cf14dfd41b'} {{(pid=81408) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 13 04:37:02.136392 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.notifications [None req-5056bc12-e932-4fd6-ba56-22cf14dfd41b admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '372da12f356748f28a1261ee8b643e8d', 'request_id': 'req-5056bc12-e932-4fd6-ba56-22cf14dfd41b'} {{(pid=81408) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 13 04:37:02.150792 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.notifications [None req-5056bc12-e932-4fd6-ba56-22cf14dfd41b admin admin] The token cache is being invalidate because project 372da12f356748f28a1261ee8b643e8d was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81408) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 13 04:37:02.151083 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.notifications [None req-5056bc12-e932-4fd6-ba56-22cf14dfd41b admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81408) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 13 04:37:02.151755 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG keystone.notifications [None req-5056bc12-e932-4fd6-ba56-22cf14dfd41b admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81408) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 13 04:37:02.153848 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: [pid: 81408|app: 0|req: 1291/2613] 10.222.0.27 () {58 vars in 1233 bytes} [Thu Oct 13 04:37:01 2022] DELETE /identity/v3/projects/372da12f356748f28a1261ee8b643e8d => generated 0 bytes in 156 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 13 04:37:11.821332 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG dbcounter [-] [81409] Writing DB stats keystone:SELECT=23,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81409) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:37:12.144691 dv-cinder-860997-1-iscsi devstack@keystone.service[81408]: DEBUG dbcounter [-] [81408] Writing DB stats keystone:SELECT=16,keystone:DELETE=7,keystone:INSERT=1 {{(pid=81408) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 13 04:40:18.986972 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a93d6ae9-7909-491f-b6ec-292da9c05fbc admin admin] Authenticating user token {{(pid=81409) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 13 04:40:18.988408 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.fernet_utils [None req-a93d6ae9-7909-491f-b6ec-292da9c05fbc admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81409) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:40:19.125312 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a93d6ae9-7909-491f-b6ec-292da9c05fbc admin admin] Validating token access rules against request {{(pid=81409) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 13 04:40:19.183338 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b3f964b3-373b-4225-9521-64a2a8b5cbd8 service placement] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ba1b6e5868a549a99760ebe6068fd6b9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dcec0431fb6849499d6aec35a234daa6', 'project_domain_id': 'default', 'roles': ['service', 'member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81409) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 13 04:40:19.183936 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3f964b3-373b-4225-9521-64a2a8b5cbd8 service placement] REQUEST_METHOD: `GET` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 13 04:40:19.184207 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3f964b3-373b-4225-9521-64a2a8b5cbd8 service placement] SCRIPT_NAME: `/identity` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 13 04:40:19.184476 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3f964b3-373b-4225-9521-64a2a8b5cbd8 service placement] PATH_INFO: `/v3/auth/tokens` {{(pid=81409) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 13 04:40:19.186228 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.fernet_utils [None req-b3f964b3-373b-4225-9521-64a2a8b5cbd8 service placement] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81409) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 13 04:40:19.198452 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b3f964b3-373b-4225-9521-64a2a8b5cbd8 service placement] RBAC: Authorizing `identity:validate_token()` {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 13 04:40:19.199999 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b3f964b3-373b-4225-9521-64a2a8b5cbd8 service placement] RBAC: Authorization granted {{(pid=81409) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 13 04:40:19.210518 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: [pid: 81409|app: 0|req: 1323/2614] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Oct 13 04:40:18 2022] GET /identity/v3/auth/tokens => generated 3308 bytes in 224 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 13 04:40:29.206859 dv-cinder-860997-1-iscsi devstack@keystone.service[81409]: DEBUG dbcounter [-] [81409] Writing DB stats keystone:SELECT=89 {{(pid=81409) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}}